In the past 5 years Continuous Delivery has gained much attention. Its benefits of rapid, iterative change are well understood, all the way up to board level. However, CD often encounters an adversary; Security. Protection of data and computer systems seems to stand on concepts like infrequent change, segregation of duties and bureaucratic heavyweight process. But are CD and Security really at odds? We don’t think so. Whilst we’ll show you the dangers of unfettered CD pipelines and the risk of letting security spread fear. We will also share ways in which we’ve managed to balance speed and security in our pipelines–considering both the technical and organisational aspects. In fact we hope you’ll see that not only is there a way, but it’s a far better way.

1. To m D u c k e r i n g & P a t D o w n e y
SECURING THE PIPELINE
Ideas, practices and food for thought to improve the
security surrounding regular delivery of software to
production.

2. WHO ARE WE AND WHAT DO WE KNOW?
2
Dev OpsTom
& Pat

3. WHO ARE WE AND WHAT DO WE KNOW?
3
Dev Ops
Sec
Tom
& Pat

4. WHAT HAVE WE SEEN?
4
Insecure
& Fast
“Over secure”
& Slow

5. WHAT HAPPENS IN HERE?
5
User accounts
Secure coding
Algorithm choice
Penetration
testing
What about
the pipeline!?

6. YOUR BUILD SYSTEM
IS PRODUCTION!
6

7. SECURING THE
PIPELINE
From head to tail
7

8. PIPELINE
8
Workstation
Code
Repo
CI Server
Build
Agent
Deploy
Agent
Pkg
Repo
Local
Cache
Prod.
Staging
QA
3rd party
code
Devs

9. PIPELINE
9
Workstation
Code
Repo
CI Server
Build
Agent
Deploy
Agent
Pkg
Repo
Local
Cache
Prod.
Staging
QA
3rd party
code
Devs

10. SHARED ACCOUNTS
10
Pa$$w0rd2015

11. SECURE WORKSTATIONS
11

12. WHO COMMITED?
12
commit 4698b247268f053299230843dd1ae68e4d15a7e3
Author: You can put anything here <mickey.mouse@foo.com>
Date: Mon Jul 6 16:23:06 2015 +0100
#837: Send logs via syslog
Lorem ipsum dolor sit amet, consetetur sadipscing elitr,
sed diam nonumy eirmod tempor invidunt ut labore et dolore
magna aliquyam erat, sed diam voluptua. At vero eos et
accusam et justo duo dolores et ea rebum. Stet clita kasd
gubergren, no sea takimata sanctus est.

13. KEYS OR PASSWORDS?
13

14. USE HTTPS OR SSH
14
There’s simply no good reason not to.

15. CENTRALISED CONTROL
15
Code Repo User Directory

16. PIPELINE
16
Workstation
Code
Repo
CI Server
Build
Agent
Deploy
Agent
Pkg
Repo
Local
Cache
Prod.
Staging
QA
3rd party
code
Devs

17. HOW MUCH IS *YOUR* CODE?
17

18. WHERE TO START?
18
Use modelling and threat analysis to prioritise the susceptible
Discover what you depend on
Assess the origin of that code for maturity of security practices

19. PIPELINE
19
Workstation
Code
Repo
CI Server
Build
Agent
Deploy
Agent
Pkg
Repo
Local
Cache
Prod.
Staging
QA
3rd party
code
Devs

20. CI SERVER & IT’S AGENTS
20
It’s a remote execution problem
Separate agents to avoid compromises
Isolate builds using chroots and containers

21. PIPELINE
21
Workstation
Code
Repo
CI Server
Build
Agent
Deploy
Agent
Pkg
Repo
Local
Cache
Prod.
Staging
QA
3rd party
code
Devs

22. PACKAGING
22
Use package system facilities to verify and sign code
But lots of them need “root” :(
Containers and unikernels offer a possible approach
But they’re immature in other ways :(

23. PIPELINE
23
Workstation
Code
Repo
CI Server
Build
Agent
Deploy
Agent
Pkg
Repo
Local
Cache
Prod.
Staging
QA
3rd party
code
Devs

24. DEPLOYMENT EXECUTION
24
Deploy
Agent
Web
Server
Service A
Data
Store
Service B
Service C
Push deployments with:
automated key based ssh!
and rights to install as root!
to all machines!
Limit the commands (e.g. via
sudo and ssh)
Consider a notification and pull
based approach

25. KEEPING SECRETS
25

26. KEY, CERT & SECRET MANAGEMENT
26
Secrets required for credentials
Try to use PKI where you can
If it has to be a password then
encrypt them per environment.
Try not to move private keys
Plan for rotation
There’s a chaining problem. It’s
hard.

27. CONTROL VS. AUDIT
27

28. CONTROL VS. AUDIT
28
Stop bad thing from being possible Know when a bad thing happened
Impact of the threat is
greater than impact on
productivity
Productivity impacted
too much to stop it
completely
Need to know
immediately
Acceptable to know
afterwards

29. THE “NSA” WAY
29
Log all the things
Alert on bad things
Look for patterns
Tell everyone that you’re doing it (unlike the NSA)

30. COMPLIANCE
30

31. SEGREGATION OF DUTIES
31
Not always explicitly mandated so RTFM
Good principle: “no single person…”
Bring it forward in the pipeline with pairing, PRs and code reviews

32. HOW TO GET THERE?
32

33. HOW TO GET THERE?
33
Dev Sec Ops
Collaborative
Goal

34. HOW TO GET THERE?
34
Structured & Objective

35. HOW TO GET THERE?
35
No Silver Bullet.
Hard things still hard

36. QUESTIONS?
36

37. Tom Duckering
tduckeri@thoughtworks.com
@tomduckering
Pat Downey
pdowney@thoughtworks.com
@pat_downey
THANK YOU