Michel van de Ven @ European Drupal Days 2015

1. © Ibuildings 2014/2015 - All rights reserved
#DrupalDaysEU
Deploying an Open Source DAM in SAAS Mode
Michel van de Ven

2. #DrupalDaysEU
© Ibuildings 2014/2015 - All rights reserved
Gold Sponsors

3. #DrupalDaysEU
© Ibuildings 2014/2015 - All rights reserved
Media Sponsors
Silver Sponsors

4. Deploying an Open Source DAM
in SAAS Mode
Michel van de Ven
Inuits
European Drupal Days 2015 Milano

5. MediaMosa
● Drupal-based Digital Asset Management system
● Commissioned by SURFnet and Kennisnet in 2007
● Open Source (GPLv2), Open Standards
● Webservice oriented: REST
● Store & retrieve assets
● Manage metadata using open standards
– Dublin Core, Qualified DC, IEEE/LOM, W3C Ontology
– OAI-PMH, CQL (SRU/SRW), BagIt
● Transcode video / audio (FFmpeg), images, PDF &c, stream content
● Storage on disk, SAN/NAS, S3
● Drupal modules for connection mgmt, media and views (by Acquia)
● Users: Kennisnet, NIBG, UGent, Uni Leuven, UvA, TiU, RUG, Digipolis Antwerp,
UOslo, Avans, Politie Academie, Acquia (NBC Sports), Berkeley, Cineca &c
● http://mediamosa.org & MediaMosa Foundation

7. =
MediaMosa as a Service

8. MediaSalsa infrastructure (simplified)
● For each DTAP environment
– Backends: Core MediaMosa services
– Frontends: Optional, Drupal, Java, Python, apps, ...
– Web servers
– Database servers
– Solr servers
– Transcoding servers
● Instance @ RUG DC for NL .edu
● Instances @ Commercial DCs for non-edu

9. Culture
Automation
Measurement
Sharing

10. Inuits
● Inuits is an Open Source company
– We contribute back
● 50+ people in 4 locations (BE, NL, UA, CZ)
● One language: English
● We offer
– Consulting
– Development
– System Administration
– and a nice SaaS Platform

11. Ideal world vs budget and reality
→ pragmatic approach

12. Distributed team
Communication = HELL
How to fix it?

13. Distributed team
● Daily virtual stand-up over XMPP / Hangout
● Redmine project management
– Issues, tasks, sprints
– Repositories
– Documentation (wiki)
– Used by customers as well
● Internal mailing lists
● Notifications by mail and XMPP
● Internal training, all the time

14. Dev and Ops
● Dev
● Architecture
● Develop new features
● Write unit tests (!)
● Ops
● Infrastructure (Puppet)
● CD and CI (Jenkins)
● Monitoring
→ Ops teach dev about monitoring and
distributed services, scale, redundancy, failover,
high availability, systems engineering, ...
→ Dev teach ops about required libraries and
testing, dependencies, daemon tuning,
software engineering & architecture, ...

15. Culture
Automation
Measurement
Sharing

16. Puppet
Puppet automates all the things
mcollective orchestrates all the things

17. Continuous D.*
● Unit Tests + Continuous Integration →
● Continuous Delivery vs Continuous Deployment
– “Continuous Delivery doesn't mean every change is deployed to production ASAP.
It means every change is proven to be deployable at any time” (@ccaum)
● Puppet code
– Deployed to dev environment
– Same puppet code for each environment
– User-triggered deployments to UAT & production
– Feature flags in Puppet code per environment (switchable architecture)
● Application code
– Continuous integration in dev
– User-triggered deployments to UAT
– Deployment to production is a business decision

18. Testing
● Developers test a lot, but
– The tests don’t work
– It works on my machine™
– Wrong platform
– Wrong PHP version
Fixed now, thanks to Jenkins!
and Vagrant / $container_tech of the week

19. Version Control
● Git
● Code is under revision control
– Prefer small commits
– Local feature branches
● Infrastructure as code → git / hiera
next: Software Defined Infrastructure

20. Use the dist native packaging system
● Consistency, security, dependencies
● Uniquely identify where files are coming from
● Source repo may not be reacheable
● Little overhead when you automate
● Configuration does not belong in a package

21. Pipelines
● A collection of jobs
● Run in sequence
● Start on checkout, end at deployment
● From the developers’ side:
→ Git push
← Mail with changes + link to deploy / promote

22. Our current pipelines
● Puppetized
● Puppet code and Application code (backend and
frontends)
● 1 pipeline per project phase / sprint

23. Pipelines steps
● Checkout
● Syntax: php -l
● Style: Drupal Coder
● Package: FPM
● Deploy to dev environment: mcollective
● Tests in dev environment: drush run-tests
● Publish package and promote: mcollective

24. Promotion
● At the end of the pipeline
● Send a simple email
● A link to the promotion page
● The changelog
● Promotion page contains one button per environment
● Must promote to UAT before Production

25. Tools used with Jenkins
● Pulp to manage RPM repositories
– Collect, process, serve / push out
● Mcollective to update packages, run drush
– Control and update marionettes

26. Shared responsibility, pain, and joy
●Same Pipelines, Tools, Patterns
are used by both devs and ops

27. Culture
Automation
Measurement
Sharing

28. Logstash
● Collect all the logs
– Drupal logs
– Apache logs
– Deployment logs
– System logs
● Interpret, filter and correlate them
● Logstash, ElasticSearch, Kibana, statsd, Graphite

29. Icinga
● Monitor everything
– vhosts
– databases
– cronjobs
– unit test suites

30. Graphite + gdash
● Collectd
● Monitor platform usage
● FFmpeg usage
● Number of accounts
● Pipelined !

31. Culture
Automation
Measurement
Sharing

32. Open Source
● Mediamosa is Fully Open Source
● Lots of the PuppetCode to deploy it is Open Source
● Our passwords etc aren't

33. Experiences
We have just shared our experience...
But also at the NLUUG Fall Conference, DrupalDays Kyiv, DrupalCon Prague, FlossUK, ElastiCon SF, ...

34. ContactContact
Kris Buytaert <Kris Buytaert <Kris.Buytaert@inuits.beKris.Buytaert@inuits.be>>
Inuits Chief Trolling OcerInuits Chief Trolling Ocer
Further ReadingFurther Reading
@krisbuytaert@krisbuytaert
http://www.krisbuytaert.be/blog/http://www.krisbuytaert.be/blog/
http://www.inuits.be/http://www.inuits.be/
Inuits HQInuits HQ
Antwerp, BelgiumAntwerp, Belgium
+32 475 961221+32 475 961221
Branches:Branches:
Rotterdam NLRotterdam NL
Kyiv UAKyiv UA
Prague CZPrague CZ