This is the story of a major Plone upgrade project for KCRW, a top-tier public radio station located in Santa Monica, CA, USA

1. Ancient To Modern
Upgrading nearly a decade of Plone in Public Radio

2. !Image Credit: Fall Path by john22s CC-BY
(https://www.flickr.com/photos/21926145@N04/15242844379)
this is the story of a journey

3. Of how a site born to Plone nearly a decade ago
!
Has grown with Plone ever since.

4. Image Credit: Amy Marbach “puppy snuggles are the best” CC-BY-NC-ND
https://www.flickr.com/photos/amyofbadgroove/14327508940
Of how a site born to Plone nearly a decade ago
!
Has grown with Plone ever since.

5. Image Credit: Amy Marbach “puppy snuggles are the best” CC-BY-NC-ND
https://www.flickr.com/photos/amyofbadgroove/14327508940
Image Credit: Richard Miles “Who Walks Who?…” CC-BY_NC_ND
https://www.flickr.com/photos/_f1guy68_/2231052733
Of how a site born to Plone nearly a decade ago
!
Has grown with Plone ever since.

6. Image Credit: Tym “Great Strength” CC-BY-NC-ND
https://www.flickr.com/photos/tym/269105455
It’s a success story for Plone, which plays to its greatest strengths

7. Image Credit: David Ellis CC- B Y -N C - N
D
https://www.flickr.com/photos/97578613@N08/10771455296
Also a morality play about how incautious customizations can make upgrading a challenge.

8. The Beginning…

9. Plone 2.0
It started back in 2005, with Plone 2.0

10. The site had a nice, modern mid-aughts look and feel, with a clean color scheme and simple navigation.

11. It also featured drop-down menus for the many top-level navigation targets, for that easy-to-discover feel.

12. especially in areas where there were a large number of sub-items, like music shows in the “music” area.

13. Image Credit: Louise Docker CC - B Y
-NC-ND
https://www.flickr.com/photos/aussiegall/6417891273
But like a lot of old things, it worked fine for its purpose

14. A new theme doesn’t
justify an upgrade

15. There’s more!
But the look-and-feel were not the only issues

16. This method is called each time a “show” is viewed.
!
The highlighted line is preparing to run a query against an external RDBMS
!
This system was built to serve certain information from the Plone content to legacy PERL scripts, which were later abandoned.

17. This method is called each time a “show” is viewed.
!
The highlighted line is preparing to run a query against an external RDBMS
!
This system was built to serve certain information from the Plone content to legacy PERL scripts, which were later abandoned.

18. ZSQL scripts like this ran through much of the site functionality

19. * SQLWindowStorage to store some AT field data in an external database
* Inefficient to retrieve this data field-by-field
* Leads to “optimization” code that retrieves values all at once and stores them persistently in object attributes
* Since this happens on __view__ of an object, there are lots of conflict errors and retries, meaning less efficient operations and more potential for real
errors.

20. * SQLWindowStorage to store some AT field data in an external database
* Inefficient to retrieve this data field-by-field
* Leads to “optimization” code that retrieves values all at once and stores them persistently in object attributes
* Since this happens on __view__ of an object, there are lots of conflict errors and retries, meaning less efficient operations and more potential for real
errors.

21. * SQLWindowStorage to store some AT field data in an external database
* Inefficient to retrieve this data field-by-field
* Leads to “optimization” code that retrieves values all at once and stores them persistently in object attributes
* Since this happens on __view__ of an object, there are lots of conflict errors and retries, meaning less efficient operations and more potential for real
errors.

22. * SQLWindowStorage to store some AT field data in an external database
* Inefficient to retrieve this data field-by-field
* Leads to “optimization” code that retrieves values all at once and stores them persistently in object attributes
* Since this happens on __view__ of an object, there are lots of conflict errors and retries, meaning less efficient operations and more potential for real
errors.

23. Image Credit: http://failblog.cheezburger.com/thereifixedit
http://failblog.cheezburger.com/thereifixedit
Clearly a safe and stable system for production.
!

24. Image Credit: Todd Jordan “Wednesday’s Angry Boss” CC-BY-NC-ND
https://www.flickr.com/photos/tojosan/14374817787
This raised the level of anger enough to really justify a large scale upgrade project.

25. Time for a Change

26. Plone?
But first, is Plone the right tool going forward?

27. Reasons To Stay
• Heterogenous content in a deep tree
• Broad editorial staff with different access
rights
• In-Place management model
• Staff familiarity
• Transmogrifier
There were definitely reasons to stay

28. Upgrade or Migrate?
• Want to use dexterity types for new site
• Heavy customizations
• Lots of not-best-practice code
• Better just to “start over”
Once the decision to stay was made, the next decision is to run an in-place upgrade, or a more extensive migration.
!I
n this case, starting from scratch was really desirable, though the content needed to be preserved

29. Migration Out
• Read content out from old Plone 2.5 site
• quintagroup.transmogrifier
• Marshall as json
• collective.jsonify
• Write site structure to disk
• modified q.transmogrifier writer
So a two-phase migration was planned, allowing us to transport the site content, stretching back into the late 1990s, into the new site.

30. A simple export pipeline moved the content to the filesystem as a series of folders and .json files

31. A simple export pipeline moved the content to the filesystem as a series of folders and .json files

32. A simple export pipeline moved the content to the filesystem as a series of folders and .json files

33. New Content Types
• Dexterity Based
• Custom shared behaviors
• IAirings (for things that go on air)
• IScheduled (for things that occur on a
schedule)
• IContentImages (for things that have a
standard set of images associated)
• …
Next, time to set up the new content types. Using dexterity allowed us to build shared behaviors that would be used by different types.

34. Migration In
• Read in json and map old types to new
• Split pipeline for individual type handling
• Remap locationsCreate new content
objects
• Or, identify existing and update
(repeatable)
• Reconnect related objects
With content types in place, the second phase involved migrating the content back into the new Plone site.
!
This process featured a number of the nice properties of the transmogrifier system.

35. The “splitter” pipeline section allows you to set up individual pipelines associated with some subset of your migrating content.
!
You can predicate entry to a particular pipeline on any number of tests, but in this case we used the portal type to determine the path to take.
!
A second feature is the ability to annotate the transmogrifier object itself with values like a mapping of user ids in the old system to UIDs for user objects
in the new system.
!
This mapping can then be shared from other pipeline sections that need to re-connect radio shows with their hosts and producers, or episodes with guests
and artists.

36. The “splitter” pipeline section allows you to set up individual pipelines associated with some subset of your migrating content.
!
You can predicate entry to a particular pipeline on any number of tests, but in this case we used the portal type to determine the path to take.
!
A second feature is the ability to annotate the transmogrifier object itself with values like a mapping of user ids in the old system to UIDs for user objects
in the new system.
!
This mapping can then be shared from other pipeline sections that need to re-connect radio shows with their hosts and producers, or episodes with guests
and artists.

37. The “splitter” pipeline section allows you to set up individual pipelines associated with some subset of your migrating content.
!
You can predicate entry to a particular pipeline on any number of tests, but in this case we used the portal type to determine the path to take.
!
A second feature is the ability to annotate the transmogrifier object itself with values like a mapping of user ids in the old system to UIDs for user objects
in the new system.
!
This mapping can then be shared from other pipeline sections that need to re-connect radio shows with their hosts and producers, or episodes with guests
and artists.

38. The “splitter” pipeline section allows you to set up individual pipelines associated with some subset of your migrating content.
!
You can predicate entry to a particular pipeline on any number of tests, but in this case we used the portal type to determine the path to take.
!
A second feature is the ability to annotate the transmogrifier object itself with values like a mapping of user ids in the old system to UIDs for user objects
in the new system.
!
This mapping can then be shared from other pipeline sections that need to re-connect radio shows with their hosts and producers, or episodes with guests
and artists.

39. The “splitter” pipeline section allows you to set up individual pipelines associated with some subset of your migrating content.
!
You can predicate entry to a particular pipeline on any number of tests, but in this case we used the portal type to determine the path to take.
!
A second feature is the ability to annotate the transmogrifier object itself with values like a mapping of user ids in the old system to UIDs for user objects
in the new system.
!
This mapping can then be shared from other pipeline sections that need to re-connect radio shows with their hosts and producers, or episodes with guests
and artists.

40. The “splitter” pipeline section allows you to set up individual pipelines associated with some subset of your migrating content.
!
You can predicate entry to a particular pipeline on any number of tests, but in this case we used the portal type to determine the path to take.
!
A second feature is the ability to annotate the transmogrifier object itself with values like a mapping of user ids in the old system to UIDs for user objects
in the new system.
!
This mapping can then be shared from other pipeline sections that need to re-connect radio shows with their hosts and producers, or episodes with guests
and artists.

41. 36 hours later
Finally, we were able to run the migration.
!
An advantage of the generator-based approach of transmogrifier is that although the migration took 36 hours, and moved well over 300,000 objects,
including images fetched over HTTP, the memory usage was constant and the server had little problem completing the job.

42. Now we have content!
And now we have plone content, but it isn’t all that great looking.

43. Make it pretty!
Image Credit: Kevin O’Mara “I Feel Pretty”
https://www.flickr.com/photos/kevinomara/13394685405

44. The client engaged a well-known design firm in NYC, Hard Candy Shell

45. ! fully responsive, changing UI elements size and positioning to fit the device.
stylish, modern design

46. ! fully responsive, changing UI elements size and positioning to fit the device.
stylish, modern design

47. ! fully responsive, changing UI elements size and positioning to fit the device.
stylish, modern design

48. The mockups featured a lot of “block” content
!
With the same types of items appearing in different ways depending on location and purpose
!
HTML consists of a lot of repeating fragments showing up in various places

49. The mockups featured a lot of “block” content
!
With the same types of items appearing in different ways depending on location and purpose
!
HTML consists of a lot of repeating fragments showing up in various places

50. The mockups featured a lot of “block” content
!
With the same types of items appearing in different ways depending on location and purpose
!
HTML consists of a lot of repeating fragments showing up in various places

51. Other Requirements
• Some pages had to be “composable”
• collective.cover
• Some pages would be pre-built, but
feature dynamic content selection
• custom browserviews & templates
• We want to keep our code DRY

52. zope.contentprovider
• Much like a browserview, but for a fragment
• Multi-adapters of context, request and view
• Allows you to render some object differently
due to the view in which it is being seen.
• Add named adapters, and you have four
possible axes for choosing a rendering
These requirements led us to investigate a little-known zope package that powers the Plone portlet system.

53. Content Providers
A content provider is and adapter, initializing it works much like its cousin the browser view. Then, when called, it is updated via an “update” method and
then rendered to an HTML fragment via the template.
!
Once a generic version is set up, making a custom version can be as simple as subclassing, giving it a different name, and providing a different template.

54. Content Providers
A content provider is and adapter, initializing it works much like its cousin the browser view. Then, when called, it is updated via an “update” method and
then rendered to an HTML fragment via the template.
!
Once a generic version is set up, making a custom version can be as simple as subclassing, giving it a different name, and providing a different template.

55. In Cover Tiles
In cover tiles, we created a “content provider” tile, which was essentially identical to the basic persistent tile from collective.cover.
!
The big difference was an added “content_provider” method, which resolved a name and passed that and the other required arguments into a
“content_provider” module-level function.
!
This method was then called from the template for all “tiles” in the project.
!
The pass-through to the module-level function simply ran the standard content provider rendering chain, but allowed for a name-based resolution of
which provider to render.

56. In Cover Tiles
In cover tiles, we created a “content provider” tile, which was essentially identical to the basic persistent tile from collective.cover.
!
The big difference was an added “content_provider” method, which resolved a name and passed that and the other required arguments into a
“content_provider” module-level function.
!
This method was then called from the template for all “tiles” in the project.
!
The pass-through to the module-level function simply ran the standard content provider rendering chain, but allowed for a name-based resolution of
which provider to render.

57. In Cover Tiles
In cover tiles, we created a “content provider” tile, which was essentially identical to the basic persistent tile from collective.cover.
!
The big difference was an added “content_provider” method, which resolved a name and passed that and the other required arguments into a
“content_provider” module-level function.
!
This method was then called from the template for all “tiles” in the project.
!
The pass-through to the module-level function simply ran the standard content provider rendering chain, but allowed for a name-based resolution of
which provider to render.

58. Picture of KCRW homepage
This let us build features like this row on the live homepage, which contains two shows and one episode, managed by KCRW staff.

59. In Browserviews
We can use the same basic trick in a browser view. First we add a “content_provider” method to the view code, then we call that code from the view
template, passing in any arguments we need to customize each call. Finally, that same method renders a contextually appropriate provider, and returns an
HTML fragment for the page to use.

60. In Browserviews
We can use the same basic trick in a browser view. First we add a “content_provider” method to the view code, then we call that code from the view
template, passing in any arguments we need to customize each call. Finally, that same method renders a contextually appropriate provider, and returns an
HTML fragment for the page to use.

61. In Browserviews
We can use the same basic trick in a browser view. First we add a “content_provider” method to the view code, then we call that code from the view
template, passing in any arguments we need to customize each call. Finally, that same method renders a contextually appropriate provider, and returns an
HTML fragment for the page to use.

62. In Browserviews
We can use the same basic trick in a browser view. First we add a “content_provider” method to the view code, then we call that code from the view
template, passing in any arguments we need to customize each call. Finally, that same method renders a contextually appropriate provider, and returns an
HTML fragment for the page to use.

63. Picture of HR show homepage
This style allowed us to create sections like these “featured content” blocks on the music area and individual show landing pages. Here, individual items
gathered by a view method are individually rendered with contextually appropriate HTML, creating a dynamic and flexible page.

64. Picture of HR show homepage
This style allowed us to create sections like these “featured content” blocks on the music area and individual show landing pages. Here, individual items
gathered by a view method are individually rendered with contextually appropriate HTML, creating a dynamic and flexible page.

65. Outcomes
• Using the same content provider let us
write templates once, use anywhere
• Having default names let us establish a
standard display and then custom ones
• Adapting catalog brains as context let us
avoid expensive .getObject() calls
In general, the outcomes of this approach were positive. Early investments in flexibility paid off as later additions “just worked”, with minimal updates.

66. Quirks
• HCS design used a custom grid system
• All rows shared the same markup
• But cell markup differed depending on
how many in a row
• Not convenient for dynamic content
• Required a custom grid engine for
collective.cover
• Would have been nice to have input in
that decision
But there were issues. For example, in service of the responsive design the custom grid provided by HCS put the responsibility for awareness of layout on
cells within a row. From a programming perspective, this inversion of responsibility caused a bit of a headache. In retrospect I would recommend having
a member of the development team on board in the theme planning process to allow for better coordination of approach.

67. Happily, collective.cover allows for customized grid layout engines, and we built one with a special case for every possible configuration of the layout.
!

68. Ugly, but it works
But It Works
No, it wasn’t pretty, but it worked.

69. Other Features
• Adapters provide consistent API for
shared behavior
• Views expose these APIs
• Client-side .js plugins consume and
display the data
• Mobile app also consumes the same
apis
shared behavior apis:
get latest airing
get airings between x and y
get upcoming things
get playable media for this thing
get data about media to be played

70. For example, the IPlayable adapter. The interface defined a number of properties and methods. An abstract implementation provided methods that could
be coded the same for all content types. Then individual adapters for various content types, including catalog brains, allowed for customized approaches
for various API methods.
!
The approach allows us to use a single API on various content types to provide information like download links for playable media files. These can be used
in javascript plugins to power the download flyout menus that appear throughout the site.

71. For example, the IPlayable adapter. The interface defined a number of properties and methods. An abstract implementation provided methods that could
be coded the same for all content types. Then individual adapters for various content types, including catalog brains, allowed for customized approaches
for various API methods.
!
The approach allows us to use a single API on various content types to provide information like download links for playable media files. These can be used
in javascript plugins to power the download flyout menus that appear throughout the site.

72. For example, the IPlayable adapter. The interface defined a number of properties and methods. An abstract implementation provided methods that could
be coded the same for all content types. Then individual adapters for various content types, including catalog brains, allowed for customized approaches
for various API methods.
!
The approach allows us to use a single API on various content types to provide information like download links for playable media files. These can be used
in javascript plugins to power the download flyout menus that appear throughout the site.

73. For example, the IPlayable adapter. The interface defined a number of properties and methods. An abstract implementation provided methods that could
be coded the same for all content types. Then individual adapters for various content types, including catalog brains, allowed for customized approaches
for various API methods.
!
The approach allows us to use a single API on various content types to provide information like download links for playable media files. These can be used
in javascript plugins to power the download flyout menus that appear throughout the site.

74. For example, the IPlayable adapter. The interface defined a number of properties and methods. An abstract implementation provided methods that could
be coded the same for all content types. Then individual adapters for various content types, including catalog brains, allowed for customized approaches
for various API methods.
!
The approach allows us to use a single API on various content types to provide information like download links for playable media files. These can be used
in javascript plugins to power the download flyout menus that appear throughout the site.

75. For example, the IPlayable adapter. The interface defined a number of properties and methods. An abstract implementation provided methods that could
be coded the same for all content types. Then individual adapters for various content types, including catalog brains, allowed for customized approaches
for various API methods.
!
The approach allows us to use a single API on various content types to provide information like download links for playable media files. These can be used
in javascript plugins to power the download flyout menus that appear throughout the site.

76. Other Features
• Javascript-based player for live or
recorded audio
• Player can be broken out into a stand-alone
window
• Viewed on mobile, player can play when
browser app is closed
• Player is persistent across page loads
The HCS designs called for an in-browser media player. One challenge was allowing playback in this player to persist across page loads as the user
browsed the site.

77. By extending the powerful history.js plugin, we were able to ajaxify links, forms and other page-loading elements in the design. We were also able to
include calls to google analytics, and reload all dynamic front-end features so that from the user’s perspective the site experience is smooth and
unsurprising.

78. By extending the powerful history.js plugin, we were able to ajaxify links, forms and other page-loading elements in the design. We were also able to
include calls to google analytics, and reload all dynamic front-end features so that from the user’s perspective the site experience is smooth and
unsurprising.

79. By extending the powerful history.js plugin, we were able to ajaxify links, forms and other page-loading elements in the design. We were also able to
include calls to google analytics, and reload all dynamic front-end features so that from the user’s perspective the site experience is smooth and
unsurprising.

80. Other Features
• Solr integration provides great search
results (alm.solrindex)
• Content from external WP blogs is
regularly indexed and searchable
• Content_provider approach helps theme
external content distinctively
A final feature was integration of the solr text search engine. We are now wrapping up a new feature that will index external blog content from KCRWs
extensive WordPress blogiverse into their Plone site. This allows in-site search to surface external content alongside the content managed inside the site.

81. A celery-managed task runs daily to collect the blogs named in the site.
!
This fires off individual celery tasks to read and index the content from each blog.
!
The tasks result in the temporary creation of objects for each posts that live just long enough to be indexed into the catalog.
!
Then, our content providers allow us to represent these posts in a distinct way within search results.

82. A celery-managed task runs daily to collect the blogs named in the site.
!
This fires off individual celery tasks to read and index the content from each blog.
!
The tasks result in the temporary creation of objects for each posts that live just long enough to be indexed into the catalog.
!
Then, our content providers allow us to represent these posts in a distinct way within search results.

83. A celery-managed task runs daily to collect the blogs named in the site.
!
This fires off individual celery tasks to read and index the content from each blog.
!
The tasks result in the temporary creation of objects for each posts that live just long enough to be indexed into the catalog.
!
Then, our content providers allow us to represent these posts in a distinct way within search results.

84. A celery-managed task runs daily to collect the blogs named in the site.
!
This fires off individual celery tasks to read and index the content from each blog.
!
The tasks result in the temporary creation of objects for each posts that live just long enough to be indexed into the catalog.
!
Then, our content providers allow us to represent these posts in a distinct way within search results.

85. Many Thanks To
• KCRW, who gave us the job
• Alec Mitchell, who led the project
• HCS, for the wonderful and feature-rich
design
• The creators and maintainers of the add-ons
we used
• PloneConf for the chance to show it all off
And that’s all we have time for. There’s much more to cover, but at this point I’ll have to stop.
!
Before I do, thanks to all the folks involved in this project. It was enormously fun to build and satisfying to see in action.

86. And You!
Any questions?

87. And You!
questions?
Any questions?