This document discusses migrating data to Drupal 8. It covers automating migrations from Drupal 5-7 and WordPress using modules like Migrate Upgrade. Custom migrations allow more control but require defining source, destination, and field mappings. Large data volumes can be handled by optimizing hardware, running migrations in chunks, and parallelizing. Tips include prioritizing important data, automating and measuring to identify issues, and testing key pages.

1. MIGRATING DATA TO DRUPAL 8
Jonathan Araña Cruz
#DrupalCampES2017.drupalcamp.es

2. about:me
Jonathan Araña Cruz / jonhattan
● +10 years Drupal
● +10 years sysadmin
● Drush maintainer. Other contribs
● Infrastructure as code
● Indentation, syntax highlight addict

3. Topics
● Migrate & Drupal 8
● Automagic migrations
○ Limitations
● Custom migrations
○ Anatomy of migrations
○ Development workflow
○ Solving hard problems
● Migrating large volumes of data
● Tips & Tricks

4. Well known solution in contrib (D6, D7)
Drastically changed although
philosophically the same
Migrate contents and configuration
settings
New upgrade path for major version
upgrades (D6,7->D8)
Migrate & Drupal 8

5. Automagic migrations
● Drupal-to-Drupal
● Wordpress-to-Drupal
● Other sources

6. Drupal 5, 6 or 7 to Drupal 8
● Install a fresh Drupal 8
● Download and enable Drupal 8 version of your modules
● Enable core modules: Migrate Drupal UI, Migrate Drupal, Migrate
● Visit /upgrade
(Drupal 5 requires Drupal 5 migration (migrate_drupal_d5) module --only supports terms, users and nodes)
Automagic migrations »

7. Via web
Automagic migrations » Drupal 5, 6 or 7 to Drupal 8

8. With Drush
drush dl migrate_upgrade
drush en migrate, migrate_drupal, migrate_upgrade
drush migrate-upgrade
--legacy-db-url=mysql://user:pass@12.34.56.78/d6db
--legacy-root=http://myd6site.com
● Provided by Migrate Upgrade (migrate_upgrade)
Automagic migrations » Drupal 5, 6 or 7 to Drupal 8

9. Limitations
● Preserves auto-incremental ids
● Fresh install. One shot.
○ Overwrites existing config
○ No rollbacks
○ No incremental migrations
● 1-1 Migration (attempt)
○ Source structure is literally replicated
● Many modules not covered (imce , ubercart/commerce, ...)
Automagic migrations » Drupal 5, 6 or 7 to Drupal 8

10. Wordpress to Drupal 8
Two modules available:
● Wordpress Migrate (wordpress_migrate)
○ Migrates WordPress blog exports (WXR format)
○ Posts, pages, comments, attachments, tags and categories
● WP Migrate (wp_migrate)
○ SQL: comments, posts, terms, users, vocabularies
○ Half baked
Automagic migrations »

11. From other sources to Drupal 8
● SQL, json, xml, csv, excel, …
● No automation
● Helper module Migrate UI (migrate_ui) - outdated with Drupal 8.3.x
Automagic migrations »

12. Custom migrations

13. Why?
● Limitations of automagic migrations:
○ One-shot
○ 1-1 migration
○ Unsupported modules
● Incremental migrations, updates, rollbacks
● Opportunities:
○ New site structure
○ Transformation of contents
○ Migrate from diverse sources
○ ...
Custom migrations »

14. How to
● Migrations are config
● Quickstart: pick & tweak automagic migrations
● Migrate Plus (migrate_plus) provides config entities, hooks, events...
● Put your migrations in a module:
○ mymodule/config/install/migrate_plus.migration.articles.yml
Custom migrations »
drush migrate-upgrade
--legacy-db-url=mysql://user:pass@12.34.56.78/d6db
--legacy-root=http://myd6site.com
--configure-only

15. Anatomy of migrations (I)
● A migration config consists of:
○ ID, label,...
○ Migration source
○ Migration destination
○ Fields mapping / process
■ Describes, property-by-property, how the destination is to be
constructed from the source data
Custom migrations »

16. Anatomy of migrations (II)
id: enredate_teachers__user
label: 'enredate.org teachers - user entity'
migration_group: enredate
migration_dependencies: { }
source:
plugin: csv
path: 'private://migrate/sources/crm-teachers.csv'
header_row_count: 1
delimiter: ';'
column_names:
- {Nombre: 'Nombre'}
- {Email: 'Email'}
- {CodWebEnredate: CodWebEnredate}
- {UserNameEnredate: UserNameEnredate}
- {PassEnredate: PassEnredate}
keys: ['CodWebEnredate']
Custom migrations »

17. Anatomy of migrations (II)
destination:
plugin: 'entity:user'
md5_passwords: true
process:
name: UserNameEnredate
pass: PassEnredate
mail: Email
status:
plugin: default_value
default_value: 1
roles: roles
Custom migrations »

18. Source plugins (I)
● Core modules provide source plugins for Drupal 6 and 7 elements
○ d6_node, d7_node, user, file, menu, block, ...
● Contrib:
○ Migrate Plus (migrate_plus) - url source and parsers for xml, json, soap
○ Migrate Source CSV (migrate_source_csv)
○ Migrate Google Sheets (migrate_google_sheets)
○ Commerce Migrate (commerce_migrate) - ubercart and commerce sources
Custom migrations » Anatomy of migrations »

19. Source plugins (II)
● @MigrateSource annotation
● Plugin/migrate/source namespace
● Some core interfaces and base classes:
○ MigrateSourceInterface
○ Node < FieldableEntity < DrupalSqlBase < SqlBase < SourcePluginBase
Custom migrations » Anatomy of migrations »

20. Destination plugins (I)
Custom migrations » Anatomy of migrations »
● Core modules provide destination plugins for Drupal 8
○ entity:user, entity:node, url_alias…
● Contrib modules provide their destinations as needed
○ field_group, entity_reference_revisions(paragraphs), externalauth, redirect...

21. ● @MigrateDestination annotation
● Plugin/migrate/destination namespace
● Some core interfaces and base classes:
○ MigrateDestinationInterface
○ EntityUser < Entity < DestinationBase
Destination plugins (II)
Custom migrations » Anatomy of migrations »

22. Process plugins
● Core provides a bunch of plugins:
○ get, callback, concat, default_value, iterator, machine_name,
migration_lookup, skip_on_empty, skip_row_if_not_set, static_map,
substr,...
● Contrib modules provide their process plugins as needed: geofield,
migrate_process_map...
Custom migrations » Anatomy of migrations »

23. Examples (I)
get default_value static_map
langcode:
plugin: static_map
map:
_esp: es
_cat: ca
_eus: eu
_gal: gl
source: language
mail: Email
mail:
plugin: get
source: Email
init_mail: ‘@mail’
status:
plugin: default_value
default_value: 1
Custom migrations » Anatomy of migrations » Process plugins

24. Examples (II)
callback migration_lookup skip_on_empty
field_related/target_id:
- plugin: skip_on_empty
source: relacionados
- plugin: migration
migration:
- abo__news
- abo__videos
name:
plugin: callback
callable: trim
source: UserNameEnredate
field_related/target_id:
plugin: migration
migration:
- abo__news
- abo__videos
source: relacionados
Custom migrations » Anatomy of migrations » Process plugins

25. Pipeline
Custom migrations » Anatomy of migrations » Process plugins
process:
foo:
- plugin: concat
delimiter: '_'
source:
- source1
- source2
- plugin: callback
callable: striptags
- plugin: callback
callable: trim
- plugin: skip_on_empty
- plugin: machine_name

26. Development workflow
● Code - Execute (import, update) - Debug - Rollback. Cleanup
● You want a runner to execute migrations:
○ Migrate Tools (migrate_tools) - Drush commands & UI
○ Migrate Run (migrate_run) - lightweight fork of Migrate Tools
○ Migrate manifest (migrate_manifest)
Custom migrations »

27. Debugging
● Migrate Devel (migrate_devel) - extends Migrate Tools
● Drush Entity (drush_entity) - Inspect entities
● Migrate messages and maps
● var_dump() Breakpoints:
○ DrupalmigrateMigrateExecutable::processRow
○ DrupalmigratePluginmigratedestinationEntityContent::import
● Dummy fields
Custom migrations » Lifecycle
dummy:
plugin: callback
callable: drush_print_r
source: source_field

28. Solving hard problems
● Auxiliary and compound migrations
● Massaging the body
○ Cleanup
○ Convert tokens
○ Get rid of <img>
● Preparation of files and videos
● Redirections in HAProxy
Custom migrations »

29. Migrating largevolumes of data

30. What we mean by «large»
● 150k files
● 20k videos
● 60k nodes
● 8k terms
● 30k users
● 1M comments
Large migrations can take hours to run. Days to review.
~9 hours in our best effort
Migrating large volumes of data »

31. Migrating on steroids
● PHP 7.x & Redis
● MySQL datadir and tmpdir on ramdisk
● MySQL indexes
● Run large migrations in chunks
● Parallelized migrations
● Lightweight database. Drush Shrinkdb (drush_shrinkdb)
● Previous download
Migrating large volumes of data »

32. Tips & Tricks
First one:
“before [...] you need to stop and
install your brain”
@hernanibf
“My site is slow”
Drupalcamp Spain ‘12

33. Tips & Tricks
● Ponder. Not everything deserves being migrated automatically
● Prioritize. Not all data have the same value
● Automate & measure. Identify regressions and opportunities to improve
● Use KPIs to measure the quality of the migration
● Test the most important urls

34. Tips & Tricks
● Order matters
● Ensure consistent source database state
● Use a MySQL read only user to connect the sources
● Don’t do it in production. Prevent performance impact or data loss/corruption
● Test run in a production-like environment. Avoid surprises the M-day