Containers and Kubernetes have revolutionized the way applications are deployed at scale. This new approach, along with the use of CI/CD for deployment automation, brings new challenges, in particular when it comes to security, as containers are static artifacts that require rebuilding and redeployment in order to perform updates. This talk will demonstrate how to set up an automated CI/CD pipeline to deploy applications on Kubernetes using OpenShift and GitLab, so that updates of public base images trigger rebuilds and deployments of derivative containers. It will also show how static image analysis can be plugged into the pipeline to increase application security.

1. Container Security:
A toolchain for automatic image rebuilds
Raphaël Pinson

2. www.camptocamp.com
Who am I?
■ Raphaël Pinson
■ Infrastructure Developer
■ Automation
(Puppet, Augeas,
Docker, Kubernetes)
■ Lausanne, CH

3. www.camptocamp.com
Camptocamp Group

4. www.camptocamp.com
Technologies timeline

5. www.camptocamp.com
Infrastructure Technology Stack

6. www.camptocamp.com
Partnerships

7. www.camptocamp.com
Containers are Static Artifacts
■ Dynamic libraries => shared security updates
■ Containers do not share libraries
○ Static artifacts
○ Might as well build static binaries
○ Need rebuilds

8. www.camptocamp.com
Conditions for Immutability
■ Containers are made for immutability
■ Requires separation of context
○ Code: immutable artifact, promoted
○ Configuration: injected at instantiation
○ Data: separate

9. www.camptocamp.com
Artifacts Promotion
■ Artifacts are built once (CI)
■ Pushed to dev (CD)
■ Promoted without rebuilds (CD)

10. www.camptocamp.com
CI/CD = CI + CD
■ Continuous Integration to build images
○ Diff is used to validate image diff
■ Continuous Deployment to deploy containers
○ Diff is used to validate promotion diff

11. www.camptocamp.com
Requirements
■ Only deploy images stored in the internal registry
(not public images)
■ Deploy automatically/promote semi-automatically
■ Mechanism to import/update images automatically
■ Keep Git history for upstream image changes

12. www.camptocamp.com
Infrastructure as Code (IaC)
Definition
Infrastructure as Code (IaC) is a method to provision and manage
IT infrastructure through the use of source code, rather than
through standard operating procedures and manual processes.
You’re basically treating your servers, databases, networks, and
other infrastructure like software. This code can help you
configure and deploy these infrastructure components quickly
and consistently.
IaC helps you automate the infrastructure deployment process in
a repeatable, consistent manner, which has many benefits.

13. www.camptocamp.com
IaC best practices
■ Code everything
■ Use version control
■ Define Code Review Processes
■ Continuously test, integrate and deploy
■ Document as little as needed

14. www.camptocamp.com
Release mgmt & deployments
GitOps - Operation by merge requests
■ The entire system state is under version control
■ A single Git repository describes one or multiple
namespaces. This is related to access permissions.
■ Operational changes are made by merge request
■ Rollback and audit logs are provided via Git
■ When disaster strikes, the whole infrastructure can be
quickly restored from Git

16. 16/
Container Patching Challenges
■ Monitor upstream images on
security patches (or other
changes)
■ Rebuild deployment image with
patched upstream image
■ Keep history of image references
for each release
■ Deploy patched deployment
image in development and/or
integration environment
■ Promotion to production

17. 17/
ImageStreams & BuildConfigs
In OpenShift:
■ ImageStreams watch distant
Docker images, sync them
locally. Can trigger BuildConfigs.
■ BuildConfigs trigger OpenShift
builds (any action). Can trigger
DeploymentConfigs.

18. 18/
Container Patching Using Openshift
ImageStream and Custom Build Strategy
■ ImageStream is set for listening
on image changes
■ Custom Build is Triggered to
update upstream Image
references in source repo
■ As the source code changed,
default build & deploy pipeline
are executed
■ Finally, just review the diffs and
accept the generated merge
request

19. 19/
Container Patching Demo Goal
As soon as an image used in the openshift cluster is updated (including for
security patches), you’ll find a brand new merge request in the release
management repository asking you if you want to deploy it.

20. 20/
■ Helm charts & Helmfile for release management
■ Gopass for encrypted secret management
■ Gitlab for:
○ Source version control
○ Private Docker registry
○ Continuous integration
○ Continuous deployment
Container Patching Demo Tools

21. 21/
Container Patching Demo Setup

22. 22/
Container Patching Demo Tools

23. 23/
Container Patching Demo Tools

24. 24/
Container Patching Demo Tools

25. 25/
Container Patching Demo Tools

26. 26/
Container Patching Demo Tools

27. 27/
Container Patching Demo Tools

28. 28/
Container Patching Demo Tools

29. 29/
Container Patching Demo Tools

30. 30/
Container Patching Demo Tools

31. 31/
Container Patching Demo Tools

32. 32/
Container Patching Demo Setup

34. 34
Questions?