Migrating from drupal to plone with transmogrifier

Migrating From Drupal
to Plone with
Transmogriﬁer
Clayton Parker, Senior Developer
PLONE SYMPOSIUM EAST 2011

Who Am I? PLONE SYMPOSIUM EAST 2011

• claytron
• Python dev since 2003
• Plone Core Committer
• Foundation Member

What will we learn? PLONE SYMPOSIUM EAST 2011

• Transmogriﬁer basics
• Migration Planning Process
• Creating a pipeline

Migrations PLONE SYMPOSIUM EAST 2011

• One off scripts
• In multiple places
• Little to no re-usability

Transmogriﬁer PLONE SYMPOSIUM EAST 2011

• A framework for migrations
• Re-usable parts

Basics PLONE SYMPOSIUM EAST 2011

• Pipeline
• Blueprints
• Sources
• Section

Sources PLONE SYMPOSIUM EAST 2011

• A blueprint
• First item in your pipeline

Example Pipeline PLONE SYMPOSIUM EAST 2011

[transmogrifier]
pipeline =
csv_file
constructor
schemaupdater
<html>
<body>
[csv_file]
<h3>Code Sample</h3>
blueprint = collective.transmogrifier.sections.csvsource
<p>Replace this text!</p>
filename = my.migration.import:my_items.csv
</body>
</html>
[constructor]
blueprint = collective.transmogrifier.sections.constructor

[schemaupdater]
blueprint = plone.app.transmogrifier.atschemaupdater

my_items.csv PLONE SYMPOSIUM EAST 2011

_path , _type , title , description
/folder1 , Folder , First Folder , This is folder One
/folder2 , Folder , Second Folder , This is folder Two
/folder1/foo , Document , One Foo , A document named foo
/folder2/foo , Document , Two Foo , Another doc named foo

The Result PLONE SYMPOSIUM EAST 2011

Items PLONE SYMPOSIUM EAST 2011

• Each item is a mapping
• Keys are ﬁelds
• Keys with a leading underscore are controllers

Example Item PLONE SYMPOSIUM EAST 2011

{'_id': 'a-stronger-connection-to-out-customers',
'_path': 'content/stronger-connection-out-customers',
'_status': 1L,
'_text_mimetype': 'text/html',
'_transitions': 'publish',
'_type': 'Document',
'allowDiscussion': False,
'creation_date': '2011/05/14 9:20:50',
'effectiveDate': '2011/05/14 9:20:50',
'modification_date': '2011/05/18 9:22:22',
'subject': 'betterninterestingnstronger',
'text': '<p>this is some text</p>rn',
'title': 'A stronger connection to out customers'}

Migration Strategy PLONE SYMPOSIUM EAST 2011

• Investigate the source
• Prepare the destination
• Find Transmogriﬁer blueprints
• Write the pipeline

Write your own PLONE SYMPOSIUM EAST 2011

• Missing blueprint
• Write one
• Contribute it back

GenericSetup PLONE SYMPOSIUM EAST 2011

• Make migration part of your release
• Ability to package migrations

My Migration PLONE SYMPOSIUM EAST 2011

• Drupal backed by MySQL
• transmogrify.sqlalchemy
• Plone 4.0.5
• collective.blog.star
• plone.app.discussion

Package Layout PLONE SYMPOSIUM EAST 2011

my.migration
!"" my
#   !"" __init__.py
#   %"" migration
#   !"" __init__.py
#   !"" config
#   #   !"" articles.cfg
#   #   !"" base.cfg
#   #   !"" blogs.cfg
#   #   !"" comments.cfg
#   #   %"" pages.cfg
#   !"" configure.zcml
#   %"" profiles
#      %"" default
#      !"" metadata.xml
#      %"" transmogrifier.txt
!"" setup.cfg
%"" setup.py

Registering Conﬁgs PLONE SYMPOSIUM EAST 2011

<transmogrifier:registerConfig
name="my.migration.base"
title="My migration base config"
description="Base settings for all transmogrifier imports"
configuration="config/base.cfg"
/>


name="my.migration.pages"
title="Drupal pages"
description="Import pages from Drupal into Plone"
configuration="config/pages.cfg"
/>


name="my.migration.articles"
title="Drupal articles"
description="Import articles from Drupal into Plone"
configuration="config/articles.cfg"
/>


name="my.migration.blogs"
title="Drupal blog entries"
description="Import blog entries from Drupal into Plone"
configuration="config/blogs.cfg"
/>


name="my.migration.comments"
title="Drupal comments"
description="Import comments from Drupal into Plone"
configuration="config/comments.cfg"
/>

transmogriﬁer.txt PLONE SYMPOSIUM EAST 2011

my.migration.pages
my.migration.articles
my.migration.blogs
my.migration.comments

base.cfg pipeline PLONE SYMPOSIUM EAST 2011

[transmogrifier]
pipeline =
drupal
portal_type
url_normalizer
path
publication_state
text_mimetype
mimetype_encapsulator
folders
constructor
commenting
comments
schema_update
workflow
reindex_object

[settings]
# Path to use if there isn’t one given
base_path = other-content
# Have to escape python string formatting for when
# this gets passed into sqlalchemy
date_format = %%Y/%%m/%%d %%k:%%i:%%s

drupal section PLONE SYMPOSIUM EAST 2011

[drupal]
blueprint = transmogrify.sqlalchemy
dsn = mysql://user:password@localhost/drupal-transmog

articles.cfg PLONE SYMPOSIUM EAST 2011

[transmogrifier]
include = my.migration.base

[drupal]
query =
SELECT
node.title,
node.status AS status,
GROUP_CONCAT(tag_data.name SEPARATOR 'n') AS subject,
FROM_UNIXTIME(node.created, "${settings:date_format}") AS creation_date,
FROM_UNIXTIME(node.created, "${settings:date_format}") AS effectiveDate,
FROM_UNIXTIME(node.changed, "${settings:date_format}") AS modification_date,
body_data.body_value AS text
url_alias.alias AS _path
FROM node
INNER JOIN field_data_field_tags AS tag_field ON tag_field.entity_id = node.nid
INNER JOIN taxonomy_term_data AS tag_data ON tag_data.tid = tag_field.field_tags_tid
INNER JOIN field_data_body AS body_data ON body_data.entity_id = node.nid
INNER JOIN url_alias ON url_alias.source = CONCAT("node/", node.nid)
GROUP BY
node.title,
node.status,
node.created,
node.changed,
body_data.body_value,
url_alias.alias;

[portal_type]
value = string:Document

pages.cfg PLONE SYMPOSIUM EAST 2011

[transmogrifier]

[drupal]
query =
my.migration.base
SELECT
node.title,
node.status AS _status,
FROM_UNIXTIME(node.changed, "${settings:date_format}") AS modification_date,
body_data.body_value AS text,
FROM node
WHERE node.type = "page"
GROUP BY
node.title,
node.status,
node.created,
node.changed,
url_alias.alias;

[portal_type]
value = string:Document

blogs.cfg PLONE SYMPOSIUM EAST 2011

[transmogrifier]

[drupal]
query =
SELECT
node.title,
node.satus AS _status,
FROM_UNIXTIME(node.created, "${settings:date_format}") AS modification_date,
body_data.body_value AS text,
FROM node
WHERE node.type = "blog"
GROUP BY
node.title,
node.status,
node.created,
node.changed,
url_alias.alias;

[portal_type]
value = string:BlogEntry

[commenting]
value = python:True

comments.cfg PLONE SYMPOSIUM EAST 2011

[transmogrifier]

[drupal]
query =
my.migration.base
SELECT
comment.subject AS title,
FROM_UNIXTIME(comment.created, "${settings:date_format}") AS published,
FROM_UNIXTIME(comment.changed, "${settings:date_format}") AS updated,
comment.name AS author_name,
body_data.comment_body_value AS text,
url_alias.alias AS _parent_path
FROM comment
INNER JOIN field_data_comment_body AS body_data ON body_data.entity_id =
comment.cid
INNER JOIN node ON node.nid = comment.nid
GROUP BY
comment.subject,
comment.created,
comment.changed,
comment.name,
body_data.comment_body_value,
url_alias.alias;

[portal_type]
# Override the portal type to use the "comment_type"
key = string:_comment_type
value = string:plone.app.discussion

base.cfg overrides PLONE SYMPOSIUM EAST 2011

[path]
blueprint = collective.transmogrifier.sections.inserter
# only add a path if one does not exist
condition = python:'_path' not in item and not '_parent_path' in item
key = string:_path
# Add the value in the extended configuration
value = string:${settings:base_path}/${item/_id}

[portal_type]
key = string:_type
# We will add the value in the extended config, but we need a
# default set here
value = string:

[commenting]
key = string:allowDiscussion
# default to false
value = python:False

Content Creation PLONE SYMPOSIUM EAST 2011

[comments]
blueprint = transmogrify.comments

[folders]
blueprint = collective.transmogrifier.sections.folders

[constructor]
blueprint = collective.transmogrifier.sections.constructor

[schema_update]
blueprint = plone.app.transmogrifier.atschemaupdater

[workflow]
blueprint = plone.app.transmogrifier.workflowupdater

[reindex_object]
blueprint = plone.app.transmogrifier.reindexobject

Item Modiﬁcation PLONE SYMPOSIUM EAST 2011

[publication_state]
condition = python:'_status' in item and item['_status'] == 1
key = string:_transitions
value = string:publish

[text_mimetype]
# This could probably be taken from the database as well
key = string:_text_mimetype
value = string:text/html

[mimetype_encapsulator]
blueprint = plone.app.transmogrifier.mimeencapsulator
key = text
mimetype = python:item.get('_%s_mimetype', key)
field = key
condition = mimetype

PLONE SYMPOSIUM EAST 2011

DEMO

Links PLONE SYMPOSIUM EAST 2011

• collective.transmogrifier
http://pypi.python.org/pypi/collective.transmogrifier/

• plone.app.transmogrifier
http://pypi.python.org/pypi/plone.app.transmogrifier/

Useful Sources and PLONE SYMPOSIUM EAST 2011

Blueprints
• plone.app.transmogrifier
• transmogrify.filesystem
• transmogrify.sqlalchemy
• transmogrify.webcrawler
• quintagroup.transmogrifier
• transmogrify.comments

Check out
.co m/d emos
s ixfeetup

Migrating from drupal to plone with transmogrifier

Empfohlen

Empfohlen

Weitere ähnliche Inhalte

Was ist angesagt?

Was ist angesagt? (20)

Ähnlich wie Migrating from drupal to plone with transmogrifier

Ähnlich wie Migrating from drupal to plone with transmogrifier (20)

Mehr von Clayton Parker

Mehr von Clayton Parker (20)

Kürzlich hochgeladen

Kürzlich hochgeladen (20)

Migrating from drupal to plone with transmogrifier