SecDevOps
securing DevOps
Area41
10.6.2016 Zürich
Aarno Aukia
VSHN AG - The DevOps Company
10.6.2016 VSHN AG | http://vshn.ch 2
About me
●
MSc Computer Science ETH
●
Security Operations @ Google
●
Co-Founder & CTO...
10.6.2016 VSHN AG | http://vshn.ch 3
Agenda
●
DevOps ?
●
Where is the security ?
●
Customer example 1
●
Customer example 2...
10.6.2016 VSHN AG | http://vshn.ch 4
DevOps?
●
Collaboration: Development (Dev) and Operations (Ops)
●
Bring agile softwar...
10.6.2016 VSHN AG | http://vshn.ch 5
Dev + Ops collaboration
●
Bring together Developers & Operations
●
Practice agile Ope...
10.6.2016 VSHN AG | http://vshn.ch 6
Infrastructure as code
●
Change from hand-groomed servers to Operations
Engineering (...
10.6.2016 VSHN AG | http://vshn.ch 7
Infrastructure tools
●
Packaging code & dependencies for atomic
deployment/rollback
–...
10.6.2016 VSHN AG | http://vshn.ch 8
Infrastructure testing
●
Bring software engineering best practice to operations
●
Lar...
10.6.2016 VSHN AG | http://vshn.ch 9
Infrastructure feedback
●
Collect all logs in ELK (Elasticsearch, Logstash & Kibana)
...
10.6.2016 VSHN AG | http://vshn.ch 10
Software Delivery Automation
10.6.2016 VSHN AG | http://vshn.ch 11
Where is the security ?
10.6.2016 VSHN AG | http://vshn.ch 12
Developers
●
Duh!
●
Education, education, education
●
Concept/architecture/code audi...
10.6.2016 VSHN AG | http://vshn.ch 13
Configuration management
●
Declare target state
●
Enforce state every x minutes, e.g...
10.6.2016 VSHN AG | http://vshn.ch 14
Logging
●
Audit logging (who changed what when)
●
Application/request log
●
As WORM ...
10.6.2016 VSHN AG | http://vshn.ch 15
Service Monitoring
●
Layer 7: HTTP, SMTP, etc
●
Layer 6: SSL, certificates, protocol...
10.6.2016 VSHN AG | http://vshn.ch 16
Backup
●
As WORM as feasible
– Restricted admin access, no access for customer staf
...
10.6.2016 VSHN AG | http://vshn.ch 17
Version Management
●
Everything is in version management = GIT
– Customer code
– Con...
10.6.2016 VSHN AG | http://vshn.ch 18
Continuous integration
●
Trigger Build/Package/Test/Deploy on each commit
– Targets ...
10.6.2016 VSHN AG | http://vshn.ch 19
Automated testing
●
All code is tested automatically
– Customer code
– Config manage...
10.6.2016 VSHN AG | http://vshn.ch 20
Databases & Backends
●
Growing list of 'standard software' needed as backends for
cu...
10.6.2016 VSHN AG | http://vshn.ch 21
Web & Application Servers
●
Growing list of application servers
– PHP, Python, Ruby,...
10.6.2016 VSHN AG | http://vshn.ch 22
Customer case 1
●
Server stack (Puppet)
– Nginx, Varnish
– PHP versions 5.6 and 7
– ...
10.6.2016 VSHN AG | http://vshn.ch 23
Case1
10.6.2016 VSHN AG | http://vshn.ch 24
Customer case 2
●
OpenShift: PaaS Plattform as a Service
●
Docker, Kubernetes (Googl...
10.6.2016 VSHN AG | http://vshn.ch 25
OpenShift
10.6.2016 VSHN AG | http://vshn.ch 26
About VSHN
●
Swiss DevOps & Ops Company, 17 people in Zürich
●
Building the tools an...
10.6.2016 VSHN AG | http://vshn.ch 28
Thanks
●
Questions ?
●
We're hiring System and Software Engineers @vshn_ch !
●
Get i...
Nächste SlideShare
Wird geladen in …5
×

Securing DevOps

358 Aufrufe

Veröffentlicht am

Presented at Area41.io (Defcon Switzerland Conference) 10.6.2016

Veröffentlicht in: Präsentationen & Vorträge
0 Kommentare
1 Gefällt mir
Statistik
Notizen
  • Als Erste(r) kommentieren

Keine Downloads
Aufrufe
Aufrufe insgesamt
358
Auf SlideShare
0
Aus Einbettungen
0
Anzahl an Einbettungen
22
Aktionen
Geteilt
0
Downloads
17
Kommentare
0
Gefällt mir
1
Einbettungen 0
Keine Einbettungen

Keine Notizen für die Folie

Securing DevOps

  1. 1. SecDevOps securing DevOps Area41 10.6.2016 Zürich Aarno Aukia VSHN AG - The DevOps Company
  2. 2. 10.6.2016 VSHN AG | http://vshn.ch 2 About me ● MSc Computer Science ETH ● Security Operations @ Google ● Co-Founder & CTO @ Atrila (Security Operations) ● Co-Founder & CTO @ VSHN (DevOps) ● Spare Time: Event Networks/WiFi at Area41 ● @aarnoaukia ● http://about.me/aarno
  3. 3. 10.6.2016 VSHN AG | http://vshn.ch 3 Agenda ● DevOps ? ● Where is the security ? ● Customer example 1 ● Customer example 2 ● Discussion
  4. 4. 10.6.2016 VSHN AG | http://vshn.ch 4 DevOps? ● Collaboration: Development (Dev) and Operations (Ops) ● Bring agile software engineering methods to operations – Automation: infrastructure as a code, versioning/rollback – Testing: continuous integration/testing/deployment ● Bring operations engineering experience to developers – Scalability: independent microservices – Production insight: monitoring/logging/metrics ● Together: make the application's owner happier
  5. 5. 10.6.2016 VSHN AG | http://vshn.ch 5 Dev + Ops collaboration ● Bring together Developers & Operations ● Practice agile Operations Engineering ● Counter fear of change with (automated) testing ● Provide developer and development infrastructure – Tools for developers, preferably self-served
  6. 6. 10.6.2016 VSHN AG | http://vshn.ch 6 Infrastructure as code ● Change from hand-groomed servers to Operations Engineering (from pets to cattle) ● Speed & reliability ● Versioning & rollback ● Prerequisite for self-service – Give each developer a full stack – No manual changes in production – As many testing instances as needed
  7. 7. 10.6.2016 VSHN AG | http://vshn.ch 7 Infrastructure tools ● Packaging code & dependencies for atomic deployment/rollback – Deb/rpm, Docker ● Infrastructure state management (configuration mgmt) – Puppet, Salt, Chef, Ansible ● Continuous Integration/Testing/Deployment – Jenkins/TravisCI/GitlabCI/Atlassian Bamboo ● Self-Service – Vagrant/Docker or through Continuous Deployment
  8. 8. 10.6.2016 VSHN AG | http://vshn.ch 8 Infrastructure testing ● Bring software engineering best practice to operations ● Large complex infrastructure (as code) → many moving parts – Unit testing each module (webserver setup, database setup, cache setup, etc) – Functional end-to-end testing of full stack (request to cache delivers content from database) ● Basically the same thing as production service monitoring but for each change
  9. 9. 10.6.2016 VSHN AG | http://vshn.ch 9 Infrastructure feedback ● Collect all logs in ELK (Elasticsearch, Logstash & Kibana) – Let the developers search for prod error root cause – No sudo/root access to production needed – Added value: merged & indexed ● Collect Server & Application Metrics – Correlate with deployments & site traffic
  10. 10. 10.6.2016 VSHN AG | http://vshn.ch 10 Software Delivery Automation
  11. 11. 10.6.2016 VSHN AG | http://vshn.ch 11 Where is the security ?
  12. 12. 10.6.2016 VSHN AG | http://vshn.ch 12 Developers ● Duh! ● Education, education, education ● Concept/architecture/code audits ● Use proven libraries ● ...
  13. 13. 10.6.2016 VSHN AG | http://vshn.ch 13 Configuration management ● Declare target state ● Enforce state every x minutes, e.g. 15min ● Establish baseline system security – Services enabled/disabled – System (admin) users, groups, keys, hashes, sudoers – AAA (AD/LDAP) for 'normal users' – Host firewall (e.g. iptables) – Installed software
  14. 14. 10.6.2016 VSHN AG | http://vshn.ch 14 Logging ● Audit logging (who changed what when) ● Application/request log ● As WORM as feasible for the customer – Generally read-only for 'normal users' – Restricted admin access ● ELK-Stack – Transport, parsing, ingest: Logstash – Storage & Indexing: Elasticsearch – Querying & Dashboard: Kibana
  15. 15. 10.6.2016 VSHN AG | http://vshn.ch 15 Service Monitoring ● Layer 7: HTTP, SMTP, etc ● Layer 6: SSL, certificates, protocols, ciphers, etc ● System parameters ● Updates ● Backup ● Tool: Icinga2
  16. 16. 10.6.2016 VSHN AG | http://vshn.ch 16 Backup ● As WORM as feasible – Restricted admin access, no access for customer staf – Only new data can be pushed ● Servers are enrolled automatically by configuration management – Enforcing the backup target will not be in the same location/infrastructure ● Data encrypted at source server using multiple keys ● Control connections use SSL/TLS ● Continuously monitored, regularly restore-tested
  17. 17. 10.6.2016 VSHN AG | http://vshn.ch 17 Version Management ● Everything is in version management = GIT – Customer code – Configuration management code & config ● Changes/commits feed into audit log ● Shared or dedicated service – Shared: github.com, bitbucket.com, gitlab.com – Dedicated: Atlassian Bitbucket, Gitlab ● AAA through AD/LDAP ● Since all Devs have offline copies: no credentials in code !
  18. 18. 10.6.2016 VSHN AG | http://vshn.ch 18 Continuous integration ● Trigger Build/Package/Test/Deploy on each commit – Targets configurable per repository, branch, tag – Manual 'promote' e.g. of production release – Feed into audit log – Store completed build/package artifacts ● Artifactory ● (private) docker registry ● Deb/RPM repository ● Feed back status to Git-GUI, dashboard, monitoring ● AAA through AD/LDAP
  19. 19. 10.6.2016 VSHN AG | http://vshn.ch 19 Automated testing ● All code is tested automatically – Customer code – Config management code & parameters ● Testing depth depends on customer... – Syntax, coding style (lint), static code analysis – Unit tests ● 'Does this module do what it is supposed to do?' – Functional tests ● 'Does the application behave correctly end-to-end?' ● detect changes in nikto/sqlmap output ?
  20. 20. 10.6.2016 VSHN AG | http://vshn.ch 20 Databases & Backends ● Growing list of 'standard software' needed as backends for customer applications – MySQL/MariaDB/Galera/MaxScale, PostgreSQL, Redis, MongoDB, RabbitMQ, Memcached, Solr, Elasticsearch, NFS/DRBD, Ceph ● All services automatically deployed by configuration management ● Provide each service with sane config, clustering, credential management, firewall config, backup config, monitoring config
  21. 21. 10.6.2016 VSHN AG | http://vshn.ch 21 Web & Application Servers ● Growing list of application servers – PHP, Python, Ruby, Java/Tomcat, Java/Wildfly, Java/Play, Coldfusion, Docker ● Provide each service with sane config, firewall config, backup config, monitoring config ● Provide backend credentials through environment variables – http://12factor.net ● Other standard components: – Apache, Nginx, Varnish, mod_security, HAproxy, OpenVPN, iptables, pacemaker, keepalived
  22. 22. 10.6.2016 VSHN AG | http://vshn.ch 22 Customer case 1 ● Server stack (Puppet) – Nginx, Varnish – PHP versions 5.6 and 7 – MySQL/MariaDB-Galera-Cluster – Memcached/Redis/Solr/Elasticsearch ● Application Deployment/update (Ansible/SSH) ● Bundle know-how (settings, tunings, etc.) in common module, override if necessary per customer through YAML-File in Git- Repository ● Docker image for local testing/developing
  23. 23. 10.6.2016 VSHN AG | http://vshn.ch 23 Case1
  24. 24. 10.6.2016 VSHN AG | http://vshn.ch 24 Customer case 2 ● OpenShift: PaaS Plattform as a Service ● Docker, Kubernetes (Google), Openshift (Redhat) ● 100% Opensource, enterprise support available ● Swiss public PaaS: appuio.ch ● EU/US public: AWS ● Dedicated/private available worldwide – AWS – Enterprise on-premises
  25. 25. 10.6.2016 VSHN AG | http://vshn.ch 25 OpenShift
  26. 26. 10.6.2016 VSHN AG | http://vshn.ch 26 About VSHN ● Swiss DevOps & Ops Company, 17 people in Zürich ● Building the tools and workflows for self-service ● Managing web applications in any cloud – We are cloud-agnostic: we run on AWS, MSA, GCE, DO, Hetzner, OVH, SafeSwissCloud, Cloudscale, Exoscale and on any on-premises Enterprise private cloud ● We work for Amazee Labs, Liip, Mercedes Benz Switzerland,Migros, SaltCinema, SIX Group, Sherpany, Sobrado, Starticket, Suisa, Taskfleet, zurichopenair.ch, etc. ● How can we help YOU?
  27. 27. 10.6.2016 VSHN AG | http://vshn.ch 28 Thanks ● Questions ? ● We're hiring System and Software Engineers @vshn_ch ! ● Get in touch with @aarnoaukia, @tobruzh or @vshnemanuel

×