Presented at Area41.io (Defcon Switzerland Conference) 10.6.2016

1. SecDevOps
securing DevOps
Area41
10.6.2016 Zürich
Aarno Aukia
VSHN AG - The DevOps Company

2. 10.6.2016 VSHN AG | http://vshn.ch 2
About me
●
MSc Computer Science ETH
●
Security Operations @ Google
●
Co-Founder & CTO @ Atrila (Security Operations)
●
Co-Founder & CTO @ VSHN (DevOps)
●
Spare Time: Event Networks/WiFi at Area41
●
@aarnoaukia
●
http://about.me/aarno

3. 10.6.2016 VSHN AG | http://vshn.ch 3
Agenda
●
DevOps ?
●
Where is the security ?
●
Customer example 1
●
Customer example 2
●
Discussion

4. 10.6.2016 VSHN AG | http://vshn.ch 4
DevOps?
●
Collaboration: Development (Dev) and Operations (Ops)
●
Bring agile software engineering methods to operations
– Automation: infrastructure as a code, versioning/rollback
– Testing: continuous integration/testing/deployment
●
Bring operations engineering experience to developers
– Scalability: independent microservices
– Production insight: monitoring/logging/metrics
●
Together: make the application's owner happier

5. 10.6.2016 VSHN AG | http://vshn.ch 5
Dev + Ops collaboration
●
Bring together Developers & Operations
●
Practice agile Operations Engineering
●
Counter fear of change with (automated) testing
●
Provide developer and development infrastructure
– Tools for developers, preferably self-served

6. 10.6.2016 VSHN AG | http://vshn.ch 6
Infrastructure as code
●
Change from hand-groomed servers to Operations
Engineering (from pets to cattle)
●
Speed & reliability
●
Versioning & rollback
●
Prerequisite for self-service
– Give each developer a full stack
– No manual changes in production
– As many testing instances as needed

7. 10.6.2016 VSHN AG | http://vshn.ch 7
Infrastructure tools
●
Packaging code & dependencies for atomic
deployment/rollback
– Deb/rpm, Docker
●
Infrastructure state management (configuration mgmt)
– Puppet, Salt, Chef, Ansible
●
Continuous Integration/Testing/Deployment
– Jenkins/TravisCI/GitlabCI/Atlassian Bamboo
●
Self-Service
– Vagrant/Docker or through Continuous Deployment

8. 10.6.2016 VSHN AG | http://vshn.ch 8
Infrastructure testing
●
Bring software engineering best practice to operations
●
Large complex infrastructure (as code) → many moving parts
– Unit testing each module (webserver setup, database
setup, cache setup, etc)
– Functional end-to-end testing of full stack (request to
cache delivers content from database)
●
Basically the same thing as production service monitoring
but for each change

9. 10.6.2016 VSHN AG | http://vshn.ch 9
Infrastructure feedback
●
Collect all logs in ELK (Elasticsearch, Logstash & Kibana)
– Let the developers search for prod error root cause
– No sudo/root access to production needed
– Added value: merged & indexed
●
Collect Server & Application Metrics
– Correlate with deployments & site traffic

10. 10.6.2016 VSHN AG | http://vshn.ch 10
Software Delivery Automation

11. 10.6.2016 VSHN AG | http://vshn.ch 11
Where is the security ?

12. 10.6.2016 VSHN AG | http://vshn.ch 12
Developers
●
Duh!
●
Education, education, education
●
Concept/architecture/code audits
●
Use proven libraries
●
...

13. 10.6.2016 VSHN AG | http://vshn.ch 13
Configuration management
●
Declare target state
●
Enforce state every x minutes, e.g. 15min
●
Establish baseline system security
– Services enabled/disabled
– System (admin) users, groups, keys, hashes, sudoers
– AAA (AD/LDAP) for 'normal users'
– Host firewall (e.g. iptables)
– Installed software

14. 10.6.2016 VSHN AG | http://vshn.ch 14
Logging
●
Audit logging (who changed what when)
●
Application/request log
●
As WORM as feasible for the customer
– Generally read-only for 'normal users'
– Restricted admin access
●
ELK-Stack
– Transport, parsing, ingest: Logstash
– Storage & Indexing: Elasticsearch
– Querying & Dashboard: Kibana

15. 10.6.2016 VSHN AG | http://vshn.ch 15
Service Monitoring
●
Layer 7: HTTP, SMTP, etc
●
Layer 6: SSL, certificates, protocols, ciphers, etc
●
System parameters
●
Updates
●
Backup
●
Tool: Icinga2

16. 10.6.2016 VSHN AG | http://vshn.ch 16
Backup
●
As WORM as feasible
– Restricted admin access, no access for customer staf
– Only new data can be pushed
●
Servers are enrolled automatically by configuration
management
– Enforcing the backup target will not be in the same
location/infrastructure
●
Data encrypted at source server using multiple keys
●
Control connections use SSL/TLS
●
Continuously monitored, regularly restore-tested

17. 10.6.2016 VSHN AG | http://vshn.ch 17
Version Management
●
Everything is in version management = GIT
– Customer code
– Configuration management code & config
●
Changes/commits feed into audit log
●
Shared or dedicated service
– Shared: github.com, bitbucket.com, gitlab.com
– Dedicated: Atlassian Bitbucket, Gitlab
●
AAA through AD/LDAP
●
Since all Devs have offline copies: no credentials in code !

18. 10.6.2016 VSHN AG | http://vshn.ch 18
Continuous integration
●
Trigger Build/Package/Test/Deploy on each commit
– Targets configurable per repository, branch, tag
– Manual 'promote' e.g. of production release
– Feed into audit log
– Store completed build/package artifacts
●
Artifactory
●
(private) docker registry
●
Deb/RPM repository
●
Feed back status to Git-GUI, dashboard, monitoring
●
AAA through AD/LDAP

19. 10.6.2016 VSHN AG | http://vshn.ch 19
Automated testing
●
All code is tested automatically
– Customer code
– Config management code & parameters
●
Testing depth depends on customer...
– Syntax, coding style (lint), static code analysis
– Unit tests
●
'Does this module do what it is supposed to do?'
– Functional tests
●
'Does the application behave correctly end-to-end?'
●
detect changes in nikto/sqlmap output ?

20. 10.6.2016 VSHN AG | http://vshn.ch 20
Databases & Backends
●
Growing list of 'standard software' needed as backends for
customer applications
– MySQL/MariaDB/Galera/MaxScale, PostgreSQL, Redis,
MongoDB, RabbitMQ, Memcached, Solr, Elasticsearch,
NFS/DRBD, Ceph
●
All services automatically deployed by configuration
management
●
Provide each service with sane config, clustering, credential
management, firewall config, backup config, monitoring
config

21. 10.6.2016 VSHN AG | http://vshn.ch 21
Web & Application Servers
●
Growing list of application servers
– PHP, Python, Ruby, Java/Tomcat, Java/Wildfly, Java/Play,
Coldfusion, Docker
●
Provide each service with sane config, firewall config, backup
config, monitoring config
●
Provide backend credentials through environment variables
– http://12factor.net
●
Other standard components:
– Apache, Nginx, Varnish, mod_security, HAproxy,
OpenVPN, iptables, pacemaker, keepalived

22. 10.6.2016 VSHN AG | http://vshn.ch 22
Customer case 1
●
Server stack (Puppet)
– Nginx, Varnish
– PHP versions 5.6 and 7
– MySQL/MariaDB-Galera-Cluster
– Memcached/Redis/Solr/Elasticsearch
●
Application Deployment/update (Ansible/SSH)
●
Bundle know-how (settings, tunings, etc.) in common module,
override if necessary per customer through YAML-File in Git-
Repository
●
Docker image for local testing/developing

23. 10.6.2016 VSHN AG | http://vshn.ch 23
Case1

24. 10.6.2016 VSHN AG | http://vshn.ch 24
Customer case 2
●
OpenShift: PaaS Plattform as a Service
●
Docker, Kubernetes (Google), Openshift (Redhat)
●
100% Opensource, enterprise support available
●
Swiss public PaaS: appuio.ch
●
EU/US public: AWS
●
Dedicated/private available worldwide
– AWS
– Enterprise on-premises

25. 10.6.2016 VSHN AG | http://vshn.ch 25
OpenShift

26. 10.6.2016 VSHN AG | http://vshn.ch 26
About VSHN
●
Swiss DevOps & Ops Company, 17 people in Zürich
●
Building the tools and workflows for self-service
●
Managing web applications in any cloud
– We are cloud-agnostic: we run on AWS, MSA, GCE, DO,
Hetzner, OVH, SafeSwissCloud, Cloudscale, Exoscale and
on any on-premises Enterprise private cloud
●
We work for Amazee Labs, Liip, Mercedes Benz
Switzerland,Migros, SaltCinema, SIX Group, Sherpany,
Sobrado, Starticket, Suisa, Taskfleet, zurichopenair.ch, etc.
●
How can we help YOU?

27. 10.6.2016 VSHN AG | http://vshn.ch 28
Thanks
●
Questions ?
●
We're hiring System and Software Engineers @vshn_ch !
●
Get in touch with @aarnoaukia, @tobruzh or @vshnemanuel