The (perceived) risk of the DevOps is that too many people get the right to "break" the platform. Test Driven Infrastructure is about adapting proven ideas from our developer colleagues to the development and operations of Infrastructure services like virtualization, OS provisioning, postfix configuration, httpd configuration, ssh tuning, SAN LUN mounting and others. This talk shows how ImmobilienScout24 utilizes more and more test driven development in IT operations to increase quality and to mitigate the risk of opening up the infrastructure developmen to all developers. See https://ep2014.europython.eu/en/schedule/sessions/44/ for conference archive.

1. DevOps Risk Mitigation
www.immobilienscout24.de
Berlin | 23.07.2014 | Schlomo Schapiro
Systems Architect, Open Source Evangelist
License: http://creativecommons.org/licenses/by-nc-nd/3.0/
Test Driven Infrastructure
@schlomoschapiro

2. Expensive
Fix
Cheap
Fix
Costs Of Finding
Bugs In Production

3. Plan
Design
Budget
Develop Test
RUN
DEV
Plan
Design
Budget
Develop Test RUNOPS
TIME

4. Plan
Design
Budget
Develop Test
Plan
Design
Budget
Develop Test RUN
Proxy
config
Database
borked
„Buy Now“
button broken
MTA drop
all mail
Load Balancer
Configuration
Everything costs
only 0 €
Login possible
without password
NFS not
available
DB Replication
stopped
No ads
shown
Broken
CSS / JS
Tomcat won't
start/stop
Service user
not defined
sudoers
invalid

5. DevOps: Respect & Learning
DEV learn
from OPS to
think about:
●
Resources (CPU, RAM, Disk)
●
Services (Start, Stop, Status)
●
Dependencies (Start DB before App)
●
Logfiles (Rotate, Remove)
●
Disk Space
●
Monitoring and Alarming
●
...
OPS learn
from DEV to
think about:
●
Incremental Improvement
●
Infrastructure as Code
●
Version Control System
●
Coding (OO, Functions, Libraries …)
●
Code Quality
●
Unit & Integration Tests
●
Test Automation
●
...

6. Untested
=
Broken

7. Unit Tests
Test the smallest possible components in an
artifical environment.
System Tests
Test the entire application in a real(istic)
environment together with other
applications.

8. Part of build process
Syntax checks
 Scripts
 Config Files
 Data Files
Unit tests for functions/libs
Run program with test data
 Check result
 Check program behaviour
with wrong/broken test data
Also run on Developer desktop
Quick feedback (~ seconds)
Install on test server
Run tests from outside
 HTTP calls
 Send emails
 Try to login
Run tests from inside
 Remote Exec (rsh, ssh …)
http://go.schapiro.org/rshpitfall
 Service Start, Stop & Status
 Modify server to create
good & bad test scenarios
 Reboot
Unit Tests System Tests

9. Unit
Tests

10. %prep
%setup -q
%install
install … %{buildroot}/…
install … %{buildroot}/…
%files
%defattr(-,root,root,-)
/...

11. BuildRequires: sudo
%build
set -e
visudo -c -f sudoers
%install
install -m 0440 sudoers -D
%{buildroot}/etc/suoders.d/%{name}
%files%defattr(-,root,root,-)
/etc/suoders.d/%{name}

12. BuildRequires: PyYAML, pylint
%build
set -e
# syntax checks
bash -n my_script.sh
# Should be valid python code
pylint -E yum-repo-propagate
# should be valid YAML file
python -c "↩
import yaml↩
yaml.safe_load(open('config.yaml'))↩
"

13. System
Tests
http://impreza-gt-club.ch/V2.0/Tests/WRX08/Koch2.jpg

14. Build
Auto-
mation
Source
Code
(SVN)
Monitor
Changes
Build Server
Run
Build
Job
●
Check out source
●
Run Unit Tests
●
Create RPM
●
Source
Code DEV
YUM
Repo
Upload
PRO
YUM
Repo
Propagate
RPM
Test Server
Deploy
and
Run
TestJob
yum
Prod Server
Deploy
to PROD
yum

15. Build
Auto-
mation
Source
Code
(SVN)
Monitor
Changes
Build Server
Run
Build
Job
●
Check out source
●
Run Unit Tests
●
Create RPM
●
Source
Code DEV
YUM
Repo
Upload
PRO
YUM
Repo
Propagate
RPM
Test Server
Deploy
and
Run
TestJob
yum
Prod Server
Deploy
to PROD
yum

16. SAN mount service
Test via rsh
Mock SAN devices
with losetup
service start, stop
mounts/umounts
Error handling

17. HTTP Proxy Configuration
X-Forwarded-For header spoofs source
Check result for ERR_ACCESS_DENIED
Run tests for all function groups!
Test Server
GET http://external.com/
X-Forwarded-For: 10.11.12.01
502 Bad Gateway✔
GET http://external.com/
X-Forwarded-For: 10.34.56.01
403 Forbidden
ERR_ACCESS_DENIED by proxy server
✘
✘

18. VM Provisioning & Kickstart Installation
Test via HTTP API
Create broken VMs and check error reports
Create valid VM and install Linux OS
Scrape VM screen via OCR
http://github.com/Immobilienscout24/lab-manager-light

19. 34 35 36
2 3
53
87 88 89 90 91 92 93 94 95 96 97
TIME
Continous Live Deployment
Deploy every application when it is ready.
Automate the delivery chain from source till production.

20. Low Risk – Lots of Fun
http://go.schapiro.org/slides

21. Kontakt:
Immobilien Scout GmbH
Andreasstraße 10
10243 Berlin
Fon: +49 30 243 01-1229
Email: schlomo.schapiro@immobilienscout24.de
URL: www.immobilienscout24.de
Thank you very much!
Please contact me for further
questions and discussions.