When we build an CI / CD systems sometimes forget protect it and hackers can take advantage of that

1. 12 tricks to
avoid hackers
breaks your
CI / CD

2. WHO WE ARE
Security research. Hacking
tools developer, DevSecOps.
Python developer.
Daniel García (cr0hn)
Can’t define myself.
I go where my curiosity drives to.
Most of the time goes bad.
I process TeraBytes for breakfast.
César Gallego
@ggdaniel
https://bit.do/cr0hn
@CesarGallegoR
https://bit.do/cesar-gallego

3. Disclaimer!
Any opinions expressed are personal
opinions and don’t represent our
employer’s view in any way

4. https://www.99cs.io
We’re working on free online book this controls of
this presentation.
Leave us your email and we will notify you when it is
published
We’ll not SPAM you, we promise :)

5. Shared vocabulary
Core Concepts

6. Legacy Systems

7. No CD
OpsDev

8. No CI/CD
OpsDev

9. Hell
Dev

10. STEPS IN BUILDING SOFTWARE CONSTRUCTION
User Code Building step Deployment step Production

11. STEPS IN BUILDING SOFTWARE CONSTRUCTION
User Code Building step Deployment step Production

12. Follow us down the rabbit hole
Starting the journey

13. In source code

14. IN THE SOURCE CODE
User Code Building step Deployment step Production

15. No all StackOverflow
people are good
persons (or even
humans)
In STACK OVERFLOW Works
Great!
https://trojan-killer.net/the-most-copied-piece-of-java-code-on-stackoverflow-contains-an-error/

16. ● Are your developers using safe libraries?
● Are you check the libraries they use?
● Even more… they ask you for advice when
choice a new library?
All Libraries
Allowed!
https://securityintelligence.com/news/popular-javascript-library-for-node-js-infected-with-malware-to-empty-bitcoin-
wallets/
You trust all libraries? so you know that all
libraries are malware / vulnerabilidades free?

17. ● Passwords
● API keys
● Private keys
● ….
SECRETS & LEAKS

18. In the
building step

19. IN THE BUILDING STEP
User Code Building step Deployment step Production

20. ● What if an user can execute anything in a
Pipeline?
● What if the C.I. has not limited the output
traffic?
A reverse Shell
in the Pipeline
https://alionder.net/jenkins-script-console-code-exec-reverse-shell-java-deserialization/
Limit user permissions and output
destinations

21. https://www.youtube.com/watch?v=QDGGPoK4gbk

22. ● Do you control what can download a developer
when they runs in a pipeline?
● Do you control which command can launch a
developer in a C.I. / C.D. configuration file?
(Jenkinsfile, gitlab.yaml…)
● Is your C.I / C.D. in different network? Are you sure?
The EVIL AGENT (1 / 3)

24. The EVIL AGENT (3 / 3)
➔ Limit internet access in the
pipeline.
➔ Fix the execution permissions

25. ● Is your company using free tier
services?
● Has your company GitHub Business
account?
The Greedy
Service consumer!
Keep in mind that free tier has limits by IP. Like GitHub,
Google Maps… If your deploy rely on this services may
be stuck if someone exceed the IP quota.

26. ● Is your company using free tier
services?
● Has your company GitHub Business
account?
The Greedy
Service consumer!
Keep in mind that free tier has limits by IP. Like GitHub,
Google Maps… If your deploy rely on this services may
be stuck if someone exceed the IP quota.

27. A git Bomb cannot be cloned. Only a problem with
old git versions. Be aware in your older systems.
The Git BOMB!
● Are your commits PGP signed?
● You know who can access rights?
● Are you using third party repositories?

28. A very fat container can spend all free space and avoid
new docker builds. A fat container make deploy a slow and
error prone process.
The Fat DOCKER!
● Do you inspect your Dockerfiles?
● Do you have Docker builds correctly
configured?
● Do you control where layers are built?

29. In the
deployment step

30. IN THE DEPLOYMENT STEP
User Code Building step Deployment step Production

31. ● ZIP Bomb is an old attack.
● The attack is very simple but very
useful
● Some of system has basic routines to
detect these kinds of attacks.
The ZIP BOMB (1 / 4)

32. ● Major of packaged software is packed as a ZIP
file: .jar, .war, .docx, .xlsx….
● Some Application Servers auto deploy them when
put files in specific path
● What if we put a ZIP bomb renamed as a valid
packed Application for a Tomcat?
The ZIP BOMB (2 / 4)

34. Perform a correct hardening of host and set
conservative limits of files, CPU and memory
that a processes can get
The ZIP BOMB (4/ 4)

35. ● Memory bomb is type of attack that aims to
fill all system memory.
● Not only RAM also SWAP is affected.
● If you don’t have limits in your host it can
consume all of your HD space as a SWAP
space.
Memory BOMB (1 / 5)

36. ● What if you can run a memory bomb in
a C.I. / C.D. system?
● What if the C.I. is deployed as multi-
agent?
Memory BOMB (2 / 5)

37. Jenkins agent 1 Jenkins agent 1 Jenkins agent 1
Jenkins behavior:
1 - You put a memory bomb in your Jenkinsfile
Memory BOMB (3 / 5)
2 - The Jenkins Master send to the job to an Jenkins
agent and it runs the pipeline and the memory bomb. So
the Jenkins agent host break down
Jenkins master
3 - Jenkins Master detect that the jobs was not finished.
So the send the same job to another Jenkins Agent
4 - Jenkins agent runs memory bomb and… break down
5 - Go to step 2

39. ➔ Less known but more effective in Docker.
➔ Today powerful computers can die very fast with no
clue who pipeline is responsible.
➔ You can lost all your agents before you find where the
problem is.
➔ Limit jobs run retries
➔ Perform a correct Operation System Disk Partition
Memory BOMB (5 / 5)

40. ● Fork bomb is type of attack that aims exhaust a
system by creating new processes recursively
● It very difficult to detect if you don’t have a very good
log system configured
● Run in a Pipeline is so easy
● In multi-agent system the results are the same that
with Memory Bomb
Fork BOMB! (1 / 3)

42. ➡ Perform a correct hardening of O.S.
➡ Limit tasks by user and process
➡ Improve monitoring
Fork BOMB! (3 / 3)

43. In production

44. The API contract must be fulfilled. No less, No more. The
more is more problematic.
Is your API
Honest!?
● Do you use thread model on you APIs?
● How do you know all the endpoints that you
have deployed?
● Are debug url opened in production?

45. The API contract must be fulfilled. No less, No more. The
more is more problematic.
Is your API
Honest!?
● Do you use thread model on you APIs?
● How do you know all the endpoints that you
have deployed?
● Are debug url opened in production?

46. In the
infrastructure

47. IN THE DEPLOYMENT STEP
User Code Building step Deployment step Production

48. ● Old hack attack but useful
● Alias commands could be the best trojan in
a system.
● There are very complicated to detect
The Evil Alias!
Perform a well hardening of your host systems & be
careful with the bot users

50. Keep this in mind
Conclusions

51. ➔ Assume that you have a lot of potential
insiders attackers.
➔ Protect your C.I. as your production
systems.
➔ Monitoring. Always monitoring. Not only in
the building step.
QUIS CUSTODIET IPSOS
CUSTODES?