DevOps has successfully conquered many IT departments and brought a new era of collaboration. With DevOps developers gain more control over the production environment and can do more on their own. Admins learn to code their infrastruture and to build delivery chains. But what about the risk of allowing more people to bring changes into production? Advanced automation can also quickly propagate human error to all the platform. How to trust everybody with production level access? Common wisdom has it that test efforts should be related to the risk of a change. However, the reality is different: Developers build elaborate automated test chains to test every single commit of their application. Admins regularly “test” changes on the live platform in production. But which change carries a higher risk of taking the live platform down? What about the software that runs at the “lower levels” of your platform, e.g. systems automation, provisioning, proxy configuration, mail server configuration, database systems etc. An outage of any of those systems can have a financial impact that is as severe as a bug in the “main” software! This talk gives a quick introduction into DevOps and continues to explore how test automation in operations and in infrastructure development helps to mitigate the risks associated with open collaboration between development and operations. Based on recent developments at ImmobilienScout24 valuable lessons can be learned for every IT environment: How to to combine the goals of stable operations and quickly bringing changes into production through test automation and delivery chains. Detailed examples can be found in this German Linux Magazin article: http://www.linux-magazin.de/Ausgaben/2014/09/Testgetrieben

1. www.immobilienscout24.de
Dresden | 19./20.03.2015 | Schlomo Schapiro
Systems Architect & Open Source Evangelist
http://creativecommons.org/licenses/by-nd/4.0
DevOps Risk Mitigation:
Test Driven Infrastructure
@schlomoschapiro

2. www.ImmobilienScout24.de
◉ >2 billion PI per month
◉ 2 data center with ~1900 VM
◉ ~2.5 million outgoing email/day
◉ total of ~550 employees
◉ ~30 crossfunctional IT teams
◉ ~150 in IT
◉ 16 years in business

3. Agenda
◉ DevOps
◉ Test Driven Development
◉ Infrastructure Development
◉ Risk Mitigation

4. DevOps

5. DEV OPS
Once Upon A Time...

6. DEV OPS

7. DEV OPS

8. DEV
Agile Methods
in Software
Development

9. DEV OPS

10. DEV

11. DevOps
Agile that
works for all IT
departments OPS

12. birgitta hohenester / pixelio.de
Steve Jurvetson / flickr

13. DEV OPS
...and they lived happily ever after.

14. DEV OPSPROD
Product Delivery Pipeline
Agile DevOps

15. DevOps
◉ Culture
◉ Automation
◉ Measuement
◉ Sharing

16. DEV OPS
Operational Concerns
Start Stop Dependencies
Ressource Management
Monitoring and Alarming
...
Incremental Improvement
Infrastructure as Code
Delivery Automation
Code Abstraction
Defined Interfaces
Test Automation
Sharing

17. Untested
=
Broken

18. Test Types
Unit Test:
Test the smallest possible components in
an artificial environment.
System Test:
Test the entire application in a real(istic)
environment together with other
applications.

19. Infrastructure
Development

20. Software
Development
◉ Java Webapp on Tomcat
◉ REST API integration
◉ Frameworks & Libraries
◉ ...
◉ BIOS patching
◉ OS Provisioning
◉ Master-Slave Database
Replication
◉ Mail Transport Servers
◉ Load Balancers
◉ Hardware Failures
◉ Backup & Restore
◉ Disaster Recovery
◉ Data Center Failover
Infrastructure
Development

21. Risk
Mitigation

22. DEV OPS
Trust
BOSS

23. DEV OPS
Unit Tests
System Tests

24. Trust the Code
and
Run the Tests
http://go.schapiro.org/slides
http://www.schapiro.org/schlomo/publications