Using JsonMigrator & Transmogrifier to upgrade Plone sites. Presentation given at Plone Conference 2014 in Bristol

1. Journeys with Transmogrifier &
friends
or
How not to get stuck in
the Plone dark ages
Daniel Jowett
Jowett Enterprises Ltd

2. About me
● Software Professional since 1997
● Background in C, Java, SQL, etc …. even P*P
● Using Plone since 2008 on starting self-employment
● Once upon a time...
● … I came to a Plone Conference in Bristol...
● … in 2010
● And everyone seemed to be cleverer than me
● And now 4 years later? ...
● I know most of them are cleverer than me
● But I have done 4 or 5 projects with Transmogrifer / Jsonmigrator

3. Foundational Principles
● Calvin & Hobbes
● Calvin is an over-imaginative 6 year old
● Hobbes is his toy tiger
● A cardboard box can be anything
● But in this case it's a “Transmogrifier”

4. Transmogrifier: ...what is it?
● Transmogrifier – turns
something into anything else
like Calvin into a toad, or tiger
● Plone's Transmogrifier:
a way to migrate content
– typically between Plone versions
– also from plain html sites & other CMS databases
– Aims for and achieves a decent amount of code reuse
● JsonMigrator: Builds on top of Transmogrifier so you don't have to
install it in the source Plone site

5. Transmogrifier Variants
● Plain Transmogrifier – 2 stage process to export/import from Plone
site via xml files (or csv files)
● Funnelweb - Crawls and parses static sites for import, which are then
'pushed' into Plone
● Jsonmigrator – 1 stage process that crawls Json views of an old site
● Transmogrifier blueprints also exist that can read from SQL Dbs, so
for example Drupal data
We will be focussing on Jsonmigrator

6. When to use Jsonmigrator
● From old Plone versions, like 2.5, but even as far back as 1.0 in theory
(but requires Python >= 2.2)
● Even from old Zope/CMF sites
● particularly when you have no buildout
● When changing content type framework from archetypes to dexterity
● To clean cruft out of an error prone old site
● When scared to upgrade because you don't know what might bite you
later
Note: This isn't the whole story - you still have to Upgrade Your Plone
products & custom code

7. When NOT to use Transmogrifier
● From Plone 4.x
● Probably not from 3.3.6 with buildout
● Without considering the alternatives:
– running Plone upgrade steps (plone.app.upgrade)
– plone.app.contenttypes migration for AT to DX based types

8. Technology Stack
● Transmogrifier:
– plone.app.transmogrifier
– collective.transmogrifier
● collective.jsonify
● collective.jsonmigrator
● Products containing extra pipeline sections, such as:
– transmogrify.dexterity
– quintagroup.transmogrifier
– transmogrify.print

9. Setting Up Jsonmigrator
1) * Duplicate old Plone in your staging environment
2) install collective.jsonify to this duplicate
3) Add “external methods”
4) * Create a brand new Plone 4.x instance (try 4.3.3 for starters)
5) * Install the updated or replacement products you need to the new
instance
6) install collective.jsonmigrator & any special pipeline sections to new
instance
* These steps are outside the scope of this presentation – hopefully
you've got an idea of what to do here (though step 5 may well not be
trivial)

10. Install collective.jsonify
This is the really old way to install stuff & generally bad practise, but if
you are using a virtualenv (or dedicated python) it's passable. It's much
easier if you're using buildout, but I'll assume that you're not!
1. Download the egg (my version in this case) to your Plone directory:
wget https://github.com/djowett/collective.jsonify/archive/0.1.8b.zip
2. Unzip it
unzip 0.1.8b.zip
3. Add it to your python (it will also pull in simplejson)
cd collective.jsonify-0.1.8b/
../bin/zopepy setup.py install
…
Adding collective.jsonify 0.1.8b to easy-install.pth file
…
… simplejson...

11. Add “External Methods”
● Add an Extensions folder
● Add json_methods.py
from collective.jsonify import get_item
from collective.jsonify import get_children
from collective.jsonify import get_catalog_results
● Then run your Plone instance, go to the Zope root and create the three
equivalently named External Methods, ie: “get_item”, “get_children”
& “get_catalog_results”, eg:

12. Install collective.jsonmigrator
● Add the highlighted lines to your buildout's: develop.cfg
[sources]
...
collective.jsonmigrator = git https://github.com/djowett/collective.jsonmigrator.git
[buildout]
eggs +=
...
collective.jsonmigrator
● You might also want to add transmogrify pipelines like
transmogrify.dexterity or quintagroup.transmogrifier to your eggs
section
● Then run buildout & run up your instance
Note: We check out the collective.jsonmigrator egg for development so
that we can tweak the pipeline configuration when we need to

13. Looking closer
● Installing Collective.jsonify & the external methods gives a (fairly)
complete json view of every piece of content on your old site
● For example...
● Collective.jsonmigrator can pick up these json data dictionarys, and
using transmogrifier 'massage' them and create the corresponding
objects on your new site

14. Have a go!
● Go to: http://<your-plone>:<port>/<instance>/@@jsonmigrator
● Select one of the pre-loaded sample configurations
● Select username, password & a small folder for your first run

15. Demo

16. So what happened there? (I)
● This was our “pipeline”
[transmogrifier]
pipeline =
remotesource
removeid
strip-leading-path
correct-zope-file-types
fix_index_html_in_id
fix_index_html_in_path
print_seq_in_desc
description_remove_seq
tag_general_if_ccsnews_in_desc
tag_service_if_ccsnews_not_in_desc
description_remove_ccsnews
find_path_depth
repl_relative_urls_in_text
get_hardcoded_urls_in_text
print_hardcoded_url_post_replace
get_siteImage_urls_in_text
print_siteImage_urls
fix_index_html_refs_in_text
find_h1_title_in_text
remove_h1_title_from_text
constructor
schema
uid
datafields
fix_defaultPage
# print_defaultpage
browser-default
# workflow-history
get-last-workflow-state
calc-workflow-transition
publish_all_folders
workflowupdater
properties
# permissions
owner
local_roles
mimetype
commit
IMPORTING

17. So what happened there? (II)
● Here are some pipeline section definitions:
[constructor]
blueprint = collective.transmogrifier.sections.constructor
# Zope Images & Files get incorrectly reported as Folders
[correct-zope-file-types]
blueprint = collective.transmogrifier.sections.manipulator
keys = _classname
destination = python:"_type"
condition = python:(item['_type'] == 'Folder' and item.has_key('_classname')
and item['_classname'] in ['Image', 'File'])
[strip-leading-path]
blueprint = collective.transmogrifier.sections.inserter
key = string:_path
value = python: item['_path'].replace('ccs/', '', 1)
[tag_general_if_ccsnews_in_desc]
blueprint = collective.transmogrifier.sections.inserter
key = string:subject
value = python: item['subject'] + ['general']
condition = python: item['_type'] == "News Item" and 'ccsnews' in item['description']

18. Real life isn't Blue Peter*
So, even if it did work just now, it will probably break for you first time,
so make some adjustments in the pipeline
(<instance>/src/collective.jsonmigrator/collective/jsonmigrator/
config_plone3.cfg)
● Exclude objects from the import (particularly plone tools)
● Use print a lot (from transmogrify.print)
● Delete the import results (if you had any) & run again
* Blue Peter was the staple TV diet for 1980's Kids in the UK, featuring
the oft used phrase “here's one I made earlier” when things didn't work

19. Benefits compared to Legacy Plone
● A fresh start
● A chance to use new Plone technologies like Diazo, Dexterity, new
Collections, plone.app.contenttypes. etc. etc.
● Moving from retired products to cutting edge equivalents
● The speed of Plone 4 & blob storage
● Using buildout to control your site

20. Caveats
● This doesn't export/import users (though see
collective.blueprint.usersandgroups)
● And it doesn't do portlets
(portlets can be exported by generic setup, but that doesn't include
“deep” content portlets – you might be stuck with a manual portlet
setup)
● Collective.jsonify ignores Plone's security model and is a security
loophole – add this to a publicly accessible site at your own risk
● The docs can be a little misleading at times
● Many folk using this are probably working off checked out code, so
egg releases are few & far between – you are going to have to get your
hands dirty
● “Debugging is backwards” in transmogrifier

21. Credits
● Transmogrifier – Martjin Pieters, Jarn
● Jsonmigrator - Rok Garbas
● RCS – letting me loose on this
● Calvin & Hobbes – Bill Watterson – creating the original
Transmogrifier
Links
● http://www.jowettenterprises.com/blog/jsonmigrator
● http://collectivejsonify.readthedocs.org/
● http://collectivejsonmigrator.readthedocs.org/

22. Fin
Daniel Jowett
www.jowettenterprises.com
daniel@jowettenterprises.com
IRC nick: danima1
Github user: djowett