Concurrency in Drupal is the description and demonstration of making migrate faster. The slides cover the basics of why concurrency makes programs faster and how to wrapper drush and Migrate to make it execute in parallel.

1. C O N C U R R E N C Y
A N I N T R O D U C T I O N T O
J O H N E N N E W, S O L U T I O N S A R C H I T E C T, D E E S O N O N L I N E
C E N G O N D . O .
@ D E E S O N _ L A B S
H T T P : / / W W W. D E E S O N - O N L I N E . C O . U K / L A B S
… O R H O W I M A D E M I G R AT E FA S T E R

2. W H AT I S
C O N C U R R E N C Y ?

4. Database
Filestore
Web service
Application

6. Database
Filestore
Web service
Application

7. B R E A K I N G T H E
P R O B L E M D O W N …

8. – E D WA R D A . L E E
“… a folk definition of insanity is to do the same
thing over and over again and to expect the results
to be different. By this definition, we in fact require
that programmers of multithreaded systems be
insane. Were they sane, they could not understand
their programs …”
T H E P R O B L E M W I T H T H R E A D S , 2 0 0 6
T H E H T T P : / / W W W. E E C S . B E R K E L E Y. E D U / P U B S / T E C H R P T S / 2 0 0 6 /
E E C S - 2 0 0 6 - 1 . H T M L W I T H T H R E A D S , 2 0 0 6

9. T H E D I N I N G P H I L O S O P H E R S
H T T P : / / D E E S O N - O N L I N E . C O . U K / L A B S / M U LT I - P R O C E S S I N G - PA R T- 1 -
H O W - M A K E - D R U S H - R U S H

10. • lock_acquire()
• drush_invoke_process()
• DB transactions (innodb)
D R U PA L 7 T O O L S

11. M A K I N G M I G R AT E
M O V E …

13. W H Y S P E E D U P M I G R A T I O N S ?
• Reduce the time a migration takes
• Discover and fix problems quicker
• Remigrate faster / regular migrations

14. T H E P R O C E S S …
• https://github.com/johnennewdeeson/drush-multi-
processing
• Download mt.drush.inc and place in sites/all/drush
• Update to migrate 7.x-2.6-rc1
• Configure your migration class for batched operation
• Create a custom drush import command (or modify
mtm.drush.inc also at github URL above)
• Full instructions online at: http://deeson-online.co.uk/labs/
multi-processing-part-2-how-make-migrate-move

15. Configuring your migration class

16. Configuring your migration class

17. Creating a Drush command

18. Starting the process thread manager

19. Process setup …

20. Process teardown …

21. L I V E D E M O …

22. F I N E T U N I N G …
• Ramp up slowly - monitor server load and db load
• /etc/my.cnf
innodb_locks_unsafe_for_binlog = 1
innodb_flush_log_at_trx_commit=2
• Use alternative cache mechanism to ease your database
load (e.g. memcache)
• $conf['lock_inc'] = 'sites/all/modules/memcache/
memcache-lock.inc';
• $conf['memcache_stampede_protection'] = TRUE;

23. T H A N K S !
… A N Y Q U E S T I O N S ?
J O H N E N N E W, S O L U T I O N S A R C H I T E C T, D E E S O N O N L I N E
C E N G O N D . O .
@ D E E S O N _ L A B S
H T T P : / / W W W. D E E S O N - O N L I N E . C O . U K / L A B S