Talk held October 5th 2020 at the WeAreDevelopers.com Live week Recording: https://youtu.be/_4rXBsp6K1I?t=7527

1. VSHN - The DevOps Company
DevSecOps
Security in DevOps
Aarno Aukia, CTO @ VSHN - The DevOps Company
05.10.2020
WeAreDevelopers Live Week

2. VSHN - The DevOps Company
● About Aarno & VSHN.ch
● Why? From Dev to DevOps to DevSecOps
● What? DevSecOps principles
● How? Concrete measures
● Who? Customer Example: Finnova AG Bankware
● Because? Resulting IT Governance & security benefits
22
Agenda

3. VSHN - The DevOps Company
@aarnoaukia a@vshn.ch
ETH → Google → Atrila → VSHN
VSHN - The DevOps Company @vshn_ch
Since 2014, currently 45 VSHNeers in Zürich, Switzerland
Helping developers run online businesses without having to worry about
operations
33
About Aarno & VSHN.ch

4. VSHN - The DevOps Company
Software Project Management
Requirements Design Implementation Validation Maintenance

5. VSHN - The DevOps Company
Software Project Management
Requirements Design Implementation Validation Maintenance

6. VSHN - The DevOps Company
Software Project Management
Requirements Design Implementation Validation Maintenance

7. VSHN - The DevOps Company
Software Project Management
Requirements Design Implementation Testing Release
Biz

8. VSHN - The DevOps Company
Software Project Management: Dev vs. Ops
Requirements Design Implementation Testing Release
Ops
Biz

9. VSHN - The DevOps Company 9
OPS = Firefighting-as-a-Service ?
9

10. VSHN - The DevOps Company
Collaboration between software developers and operations:
● Teamwork
● Continuous improvement
● Efficient and lean
● Agile: being able to react to new requirements
● Automate as much as possible (“Infrastructure as code”)
1212
DevOps: People, Processes & Tools

11. VSHN - The DevOps Company
Software Project Management: DevOps
Requirements Design Implementation Testing
ReleaseDeployOperateMonitor

12. VSHN - The DevOps Company
Software Project Management: DevOps
Requirements Design Implementation Testing
ReleaseDeployOperateMonitor
SECURITY

13. VSHN - The DevOps Company
Software Project Management: DevSecOps
Requirements Design Implementation Testing
ReleaseDeployOperateMonitor
Todo-List Data & Risks
Secure
Practices
Validation
traceabilityauditabilityAnomalies Availability

14. VSHN - The DevOps Company
● Developer education, requirements engineering, design review ->
AppSec/DevSec
● Software Build/Deployment/Operations -> DevSecOps
● Incident detection & management -> SecOps
1616
Areas of security improvement

15. VSHN - The DevOps Company
DevSecOps principles
1717

16. VSHN - The DevOps Company
● static code analysis automatically for each commit (DevSecOps principle 4)
● Dependency Management (DevSecOps principle 5)
● (base) container image scanning (DevSecOps principle 5)
1818
Build

17. VSHN - The DevOps Company
Code analysis: sonarqube
1919

18. VSHN - The DevOps Company 2020
Dependency updates: https://dependabot.com

19. VSHN - The DevOps Company
Container scanning: aquasec
2121

20. VSHN - The DevOps Company
● smoke tests (DevSecOps principle 4)
● As many test environments as necessary (DevSecOps principle 4)
● atomic container deployment (DevSecOps principle 6)
● every deployment (and rollback) is a “normal deployment” (DevSecOps
principle 6)
● deployment automation removes need for (all) devs root prod access and/or
waiting for ops to deploy new dev version (DevSecOps principle 6)
2222
Test & Deployment

21. VSHN - The DevOps Company
● standardization on (minimal, hardened) OS and container orchestrator
● immutable (application) infrastructure using containers
● process/storage/network separation of applications/environments
● detect/prevent configuration drift between dev/test/stage/prod envs
● documentation & automatic backup of all volumes
● documentation & monitoring of routes/loadbalancers/ingresspoints with
enforcing SSL/TLS
● AAI for admin & application
● key & secrets management
● audit logging of control & application planes
2323
Ops (DevSecOps principle 1)

22. VSHN - The DevOps Company
Container isolation
2424
● Kernel namespacing (process & network)
● Control groups (resource quota to prevent DoS)
● SELinux (additional syscall filter)
● prevent running as root inside container, no user-provided privileged
containers (enforce best practice)
● readonly container filesystem (harder to persist exploit at runtime)

23. VSHN - The DevOps Company
● “Docker”
○ Kernel-based process isolation based on lxc/libcontainer/runc (CNCF open standard)
○ Open Source Tools for container image creation and management (“Docker CE”)
○ Company based in San Francisco (“Docker inc”)
○ Enterprise software product (“Docker EE”)
○ Online portal for public docker containers (“Dockerhub” hub.docker.com)
● “Dockerfile”
○ Text file containing all the instructions to build and assemble the application into a container
including application code, appserver, plugins, modules, libraries down to libc
○ Goal: document & automate the build process
○ Usually in the application GIT repository
○ References a base image to incrementally add the application to
3030
Container technology: Docker

24. VSHN - The DevOps Company
● Use declarative formats for setup automation
● Have a clean, portable contract with the underlying operating system
● Are suitable for deployment on modern cloud platforms, obviating the need
for servers and systems administration;
● Minimize divergence between development and production, enabling
continuous deployment for maximum agility;
● And can scale up without significant changes to tooling, architecture, or
development practices.
3131
12 Factor App Patterns: https://12factor.net/

25. VSHN - The DevOps Company
From container
to production?
3232

26. VSHN - The DevOps Company
● How many container instances should be running per service?
● On which IP/port/server are they running on?
● Service discovery
● What happens if a container/server goes away?
● scaling, load balancing, rolling deployments, persistent storage, networking
3333
Container orchestration: Kubernetes

27. VSHN - The DevOps Company
Docker
Kubernetes
3939
Layers of abstraction
Hardware
Operating System
Service discovery & Load
balancing
Application Server
Application
Cloud/Onprem

28. VSHN - The DevOps Company
Zooming in...

29. VSHN - The DevOps Company
● Logging: EFK
● Metrics: Prometheus
● SSL-Certificates: cert-manager (letsencrypt.org)
● Source-to-image builder, Dockerfile builder, Docker-Image-Registry
● Load-balancing: Ingress, NGINX
● horizontal (auto) scaling, rolling deployments
● Scanning: image scanning (repo & runtime)
4141
Kubernetes Ecosystem

30. VSHN - The DevOps Company
AAI: Keycloak
4242
● Identity & Access Management
● Single sign in/out
● Identity brokering:
○ OpenID Connect (OAuth2, FB/Twitter/Github etc.)
○ SAML2.0
○ Kerberos
● User federation: LDAP, AD, etc
● 2FA: TOTP/HOTP
● Managing the Authorization groups

31. VSHN - The DevOps Company
Logs: ELK/EFK/Greylog
4343
● Logging all access and changes through the control plane
● Logging all access to the application and correlate with application logs
● Index, view, filter, aggregate KPI → monitoring
● Store outside of application scope

32. VSHN - The DevOps Company
● Prometheus
○ time series database
○ open source / CNCF-project
○ well-integrated in docker/kubernetes stats
● NewRelic APM
○ application-level profiling
○ performance tracking
○ exception tracking (backend & frontend)
○ available as SaaS
4444
Metrics: Prometheus / NewRelic

33. VSHN - The DevOps Company
● Developer and Operator of Banking Software used by ~100 Banks
● Based in Lenzburg, Switzerland
● Founded 1974
● ~400 Employees
Example: Finnova AG Bankware

34. Hier steht der Präsentationstitel I 47

35. VSHN - The DevOps Company 48
Deployment Process & Security
DEV
(Repository)
Files (Pods)
Docker
Images
Container
(OpenShift)
Betrieb AM
Seewen
(PRD)
GitHub
Code Image-Scan (Security & Compliance Policies)
» Code Analyse
» Image Scanning
» Container Hardening
Runtime
» Network Security
» Monitoring
» Logging & Reporting
» Code Security
» Access
» Security & Auditing
SecurityDeployment

36. VSHN - The DevOps Company 49
Architecture and Security

37. VSHN - The DevOps Company
● “Full Stack Audit”
● Review design document
● Every layer was custom built
○ physical hardware
○ handcrafted servers
○ manual application deployment
● Review each layer
● Review each layer again next year...
5050
Traditional IT governance

38. VSHN - The DevOps Company
● Standardized components
○ already audited, some even externally certified
○ re-used, economies of scale, CMMI level 5
○ tech controls (AAI, RBAC, logs/SIEM) implemented once
○ financial controls implemented once
● Infrastructure: private/public cloud
● Ops: Container orchestration platform
● Review design document & platform
configuration
5151
Cloud native IT governance

39. VSHN - The DevOps Company
● prevent configuration drift
○ immutable (application) infrastructure using containers
○ deploy dev/test/stage/prod envs from CI/CD
● prevent manual errors
○ validate configuration in CI/CD before deployment
○ standardization on (minimal, hardened) OS and container orchestrator
○ deployment automation removes need for (most) root prod access
● security by default
○ image scanning, dependency vulnerability management
○ process/storage/network separation of applications/environments
○ volumes & ingresspoints best practice (documentation, monitoring, backup, SSL/TLS/WAF)
○ AAI for admin & application, audit trail logging of CI/CD, control & application planes
○ key & secrets management
● 5252
IT governance controls in container platforms

40. VSHN - The DevOps Company
● compute resources billable by project
● self-service-onboarding possible
● autoscaling, scale-down dev envs outside office hours
● vendor procurement/due diligence/certification management
● SLA, 24x7, service process, escalation management clearly defined
5353
IT governance financial/compliance controlling

41. VSHN - The DevOps Company
DevSecOps principles
5454

42. VSHN - The DevOps Company
● Please get in touch with feedback
● Twitter: @aarnoaukia
● Linkedin: https://www.linkedin.com/in/aukia/
● Email: aarno.aukia@vshn.ch
5555
Thank you

43. Come visit us for a coffee!
VSHN AG - Neugasse 10 - CH-8005 Zürich - +41 44 545 53 00 - https://vshn.ch/ - info@vshn.ch
https://vshn.ch/kontakt/
Follow us on Twitter!
@vshn_ch
56