The presented article will target inspection of security of (un)official Docker formatted container images approaching the resulting safety of the image from two PoVs: examining image content for presence of known security flaws (vulnerability assessment), and validation if the internal software and service(s) encapsulated within the image are configured according to commonly-accepted recommendations as defined in security baselines (security hardening). Starting with reasoning why Docker images security matters, we will proceed to outline architectural concepts Docker images are based on. Subsequently compare these concepts with building blocks used by design of today's virtual machines, and point out areas (main differences) to take care of when container image security is primary concern. We will use the observations from this comparison to emphasize the need to inspect both content of container images themselves, but also the security configuration of the hosting computer in order to reach truly secure infrastructure. We will introduce the section of inspecting security of container images with providing overview of recent effort to implement image signing and verification. Afterwards we will demonstrate inspection of a concrete container image against currently known security flaws, and explain how this approach can be automated and generalized. Thereafter we will focus on examination if software and service(s) included in the container image meet commonly-known requirements for secure configuration. An illustration example how to detect e.g. an unauthorized executable in the container content will be provided. In the part dedicated to securing of the hosting computer we will show it is possible to fully automate this task too. We will conclude with sketching, where development in this area might be heading in the future (features that might be available to strengthen the security of container images even more).

1. Inspecting Security of Docker
formatted Container Images
To find Peace of Mind
Ján Lieskovský
jlieskov@redhat.com
iankko on #openscap

2. Docker Concepts - Introductory Notes

3. Docker Formatted Container Images
Interesting Application Platform

4. Docker Formatted Container Images
Interesting Application Platform
For developers
● Focus on content (rather on build process)
● Data aggregation via Docker Image Specification
● Simplified release management
● Easy customization

5. Docker Formatted Container Images
Interesting Application Platform
For users
● Abundance of applications available in official hub
● Simple application deployment
● Continuous application lifecycle management
● Easy customization

6. Basic Docker Terms
Docker image
● Ordered collection of root filesystem changes
● Coupled with corresponding execution parameters
● Doesn’t have a state
● Read-only (never changes)
● Set of layers stacked on top of each other

7. Basic Docker Terms
Docker image
● Each image is derived from base image
● Transformed to final image through set of steps (instructions)
○ Run a command
○ Add a file or directory
○ Create an environment variable
○ What process to run when launching a container from this image

8. Docker image vs Docker container
Docker container
● Any (running / stopped) instance of Docker image
● Consists of:
○ Docker image
○ Execution environment
○ Standard set of instructions
● It’s possible to have many running containers of the same image

9. Docker architecture

10. Virtual Machines vs Containers

11. Container security. And why it matters

12. Container Security Matters
Basic security bricks of Docker daemon / containers
● Kernel security (updates, support for namespaces, cgroups)

13. Container Security Matters
Basic security bricks of Docker daemon / containers
● Kernel security (updates, support for namespaces, cgroups)
● Security of Docker daemon

14. Container Security Matters
Basic security bricks of Docker daemon / containers
● Kernel security (updates, support for namespaces, cgroups)
● Security of Docker daemon
● Security of specific Dockerfile

15. Container Security Matters
Kernel namespaces, cgroups
● Form of isolation
● Own network stack per container
● Resource mngmt via cgroups

16. Container Security Matters
Other kernel features applied in Docker security
● Linux kernel capabilities
● GRSEC, PAX
● SELinux, AppArmor

17. Container Security Matters
Basic security bricks of Docker daemon / containers
So far looks reasonable, right?

18. Container Security Matters
But, what if we overlooked something?

19. Container Security Matters
But, what if we overlooked something?
22 April 2014 Daniel J Walsh (Red Hat)
Containers do not contain
Bottom line:
● Running a container not every major kernel subsystem is namespaced
○ SELinux
○ Cgroups
○ File system under /sys
○ /proc/sys, /proc/irq, /proc/bus
○ Devices and kernel modules are not namespaced

20. Container Security Matters
But, what if we overlooked something?

21. Container Security Matters
But, what if we overlooked something? Meet Shocker!

22. Container Security Matters
But, what if we overlooked something? Meet Shocker!

23. Virtual Machines vs Containers

24. Container Security Matters
What the wise men have got to say?
22 Jul 2014 Jérôme Petazzoni (Docker Inc.)
Is it Safe to Run Applications in Linux Containers?
Bottom line:
● Don't run things as root ● Use seccomp-bpf
● Drop capabilities ● Get a GRSEC kernel
● Enable user namespaces ● Update kernels often
● Get rid of shady SUID binaries ● Mount everything read-only
● Enable SELinux (or AppArmor) ● Ultimately, fence things in VMs

25. Container Security Matters
What the wise men have got to say?
03 Sep 2014 Daniel J Walsh (Red Hat)
Bringing new security features to Docker
Bottom line:
● Only run applications from a trusted source
● Run applications on a enterprise quality host
● Install updates regularly
● Drop privileges as quickly as possible
● Run as non-root whenever possible
● Watch your logs
● setenforce 1

26. Container Security Matters
Plenty of applications from Official Repositories

27. Container Security Matters
(Another) Plenty of applications from Custom Repositories

28. Container Security Matters
Who built these images?

29. 12 Aug 2015 Introduced in Docker v1.8 using The Update Framework
Docker Content Trust Workflow
● Image producer - pushing an image to remote repository, Docker engine signs the
content using publisher’s private key
● Image consumer - when pulling an image, Docker engine verifies the content of the
image using publisher’s public key. If image tampering is detected, pull fails
Container Security Matters
Docker Image Signing and Verification

30. Two types of keys known by Docker Content Trust
○ Tagging Key
■ One such key is created per each new repository the publisher owns
■ Intended to be shared with any person / system requiring the ability to
sign content for this repository
○ Offline key
■ Can be shared across repositories
■ Required to create a new repository or to rotate existing tagging keys
Container Security Matters
Docker Image Signing and Verification

31. Provides
○ Protection against image tampering
○ Protection against image replay attacks
○ Protection against tagging key compromise
Container Security Matters
Docker Image Signing and Verification

32. Container Security Matters
Who built these images?

33. Container Security Matters
We know the publisher. But how were all these images built?

34. Docker image (quick recap)
● Each image is derived from base image
● Transformed to final image through set of steps (instructions)
Container Security Matters
We know the publisher. But how were all these images built?

35. Docker image (quick recap)
● Each image is derived from base image
● Transformed to final image through set of steps (instructions)
Creating new images
● Update the container (running an image)
Commit the changes to image
● Build a new image from Dockerfile
Container Security Matters
How were all these images built?

36. Container Security Matters
Who can build / update the containers?

37. Container Security Matters
Who can build / update the containers?
Anyone!!!

38. Container Security Matters
What we trust into when pulling images?

39. ● The base image the pulled image is derived from was
secure
Container Security Matters
What we trust into when pulling images?

40. ● The base image the pulled image is derived from was secure
● The newly introduced changes were performed in secure
way
Container Security Matters
What we trust into when pulling images?

41. ● The base image the pulled image is derived from was secure
● The newly introduced changes were performed in secure way
● When a security flaws was found in base image or the
changes, the image available in repository has been
already updated
Container Security Matters
What we trust into when pulling images?

42. Ultimate goal:
● Secure container infrastructure
Container Security Matters
Docker daemon / container security - Lessons Learned

43. Ultimate goal:
● Secure container infrastructure
Trust the design:
● We can trust Docker design to be secure
Container Security Matters
Docker daemon / container security - Lessons Learned

44. Ultimate goal:
● Secure container infrastructure
Trust the design:
● We can trust Docker design to be secure
But act responsibly:
● Verify that all of the host, daemon and containers truly are secure
Container Security Matters
Docker daemon / container security - Lessons Learned

45. How to verify (inspect) security of containers / images?

46. Inspecting Security of Containers /
Images
Two separate tasks:
● Inspect presence of security flaws (vulnerability
assessment)
● Verify the configuration is secure (security
compliance)

47. Vulnerability Assessment of
Containers / Images

48. Vulnerability Assessment of
Containers / Images

49. Vulnerability Assessment of
Containers / Images
● HTML advisories are easy consumable by humans
● But not suitable for machine processing

50. Vulnerability Assessment of
Containers / Images
● We need a standard
● Security errata information available in the form of
that standard
● Scanner able to perform automated scan

51. Vulnerability Assessment of
Containers / Images
● We need a standard to
○ represent configuration information of systems
○ analyze the system for presence of specified
machine state (vulnerability, configuration, …)
○ report the results of the assessment back

52. Vulnerability Assessment of
Containers / Images
Open Vulnerability and Assessment
Language

53. Vulnerability Assessment of
Containers / Images
● We need a standard
● Security errata information available in the form of
that standard
● Scanner able to perform automated scan

54. Vulnerability Assessment of
Containers / Images
● We need a standard
● Security errata information available in the form of
that standard
○ Red Hat OVAL security data
○ Ubuntu OVAL security data
○ …

55. Vulnerability Assessment of
Containers / Images
● We need a standard
● Security errata information available in the form of
that standard
● Scanner able to perform automated scan

56. Vulnerability Assessment of
Containers / Images

57. Vulnerability Assessment
of Containers
# dnf -y install openscap-containers
# docker pull richxsl/rhel7
# wget http://www.redhat.com/security/data/oval/Red_Hat_Enterprise_Linux_7.
xml
# docker run -i -t richxsl/rhel7 /bin/bash

58. Vulnerability Assessment
of Containers
# oscap-docker container-cve richxsl/rhel7
Definition oval:com.redhat.rhsa:def:20160695: false
Definition oval:com.redhat.rhsa:def:20160685: true
Definition oval:com.redhat.rhsa:def:20160676: false
Definition oval:com.redhat.rhsa:def:20160650: false
Definition oval:com.redhat.rhsa:def:20160612: false
Definition oval:com.redhat.rhsa:def:20160594: false
Definition oval:com.redhat.rhsa:def:20160534: false
Definition oval:com.redhat.rhsa:def:20160532: true
...

59. Vulnerability Assessment
of Containers
# oscap-docker container-cve richxsl/rhel7
Definition oval:com.redhat.rhsa:def:20160695: false
Definition oval:com.redhat.rhsa:def:20160685: true
Definition oval:com.redhat.rhsa:def:20160676: false
Definition oval:com.redhat.rhsa:def:20160650: false
Definition oval:com.redhat.rhsa:def:20160612: false
Definition oval:com.redhat.rhsa:def:20160594: false
Definition oval:com.redhat.rhsa:def:20160534: false
Definition oval:com.redhat.rhsa:def:20160532: true
...

60. Vulnerability Assessment
of Images
Running untrusted containers might
be dangerous!

61. Vulnerability Assessment
of Images
Running untrusted containers might
be dangerous!
Let’s scan the images instead !!!

62. Vulnerability Assessment
of Containers
# oscap-docker image-cve richxsl/rhel7
Definition oval:com.redhat.rhsa:def:20160695: false
Definition oval:com.redhat.rhsa:def:20160685: true
Definition oval:com.redhat.rhsa:def:20160676: false
Definition oval:com.redhat.rhsa:def:20160650: false
Definition oval:com.redhat.rhsa:def:20160612: false
Definition oval:com.redhat.rhsa:def:20160594: false
Definition oval:com.redhat.rhsa:def:20160534: false
Definition oval:com.redhat.rhsa:def:20160532: true
...

63. Vulnerability Assessment
of Containers
# oscap-docker image-cve richxsl/rhel7
Definition oval:com.redhat.rhsa:def:20160695: false
Definition oval:com.redhat.rhsa:def:20160685: true
Definition oval:com.redhat.rhsa:def:20160676: false
Definition oval:com.redhat.rhsa:def:20160650: false
Definition oval:com.redhat.rhsa:def:20160612: false
Definition oval:com.redhat.rhsa:def:20160594: false
Definition oval:com.redhat.rhsa:def:20160534: false
Definition oval:com.redhat.rhsa:def:20160532: true
...

64. Vulnerability Assessment
of Containers
# oscap-docker image-cve richxsl/rhel7
Definition oval:com.redhat.rhsa:def:20160695: false
Definition oval:com.redhat.rhsa:def:20160685: true
Definition oval:com.redhat.rhsa:def:20160676: false
Definition oval:com.redhat.rhsa:def:20160650: false
Definition oval:com.redhat.rhsa:def:20160612: false
Definition oval:com.redhat.rhsa:def:20160594: false
Definition oval:com.redhat.rhsa:def:20160534: false
Definition oval:com.redhat.rhsa:def:20160532: true
...

65. Inspecting Security of Containers /
Images
Two separate tasks:
● Inspect presence of security flaws (vulnerability
assessment)
● Verify the configuration is secure (security
compliance)

66. Verification if Configuration of
Containers / Images is Secure?

67. Verification if Configuration of
Containers / Images is Secure?
# dnf -y install scap-security-guide

68. Verification if Configuration of
Containers / Images is Secure?
# oscap-docker image richxsl/rhel7 xccdf eval
--profile standard --report /tmp/report.html
/usr/share/xml/scap/ssg/content/ssg-rhel7-xccdf.xml

69. Verification if Configuration of
Containers / Images is Secure?
# oscap-docker image richxsl/rhel7 xccdf eval
--profile standard --report /tmp/report.html
/usr/share/xml/scap/ssg/content/ssg-rhel7-xccdf.xml

70. Verification if Configuration of
Containers / Images is Secure?

71. Questions?
http://www.open-scap.org/tools/
https://github.com/OpenSCAP
(We are hiring)
Ján Lieskovský
jlieskov@redhat.com
iankko on #openscap

73. Slightly Advanced Topics
Customizing security policy for particular use case
Example use case:
● Detect unauthorised SUID binaries present in the
container

74. Slightly Advanced Topics
Example use case:
● Detect unauthorised SUID binaries present in the
container
Modify standard SCAP Security Guide profile to contain
just:
"file_permissions_unauthorized_suid"
rule

75. Slightly Advanced Topics
● Modify standard SCAP Security Guide profile to
contain just:
"file_permissions_unauthorized_suid"
rule
● Rescan the image