Microcontainers and Tools for Hardcore Container Debugging

1. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Microcontainers and Tools for
Hardcore Container Debugging
Vish Ishaya Abrams, TJ Fontaine
September 7th, 2017
Container Native Meetup

2. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Who Are We?
• Former Maintainer of Node.js
• Authors of Oracle Open Source Container Utilities
• Founder of OpenStack at NASA

3. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Agenda
What is a Container?
Intro to Microcontainers
Building Microcontainers
Container Debugging
Questions and Answers

4. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
WHAT IS A CONTAINER?

5. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
A container is:
1. An application bundled with its dependencies (also called a container
image).
2. A running copy of that application that has been isolated from other
system resources via a container runtime.

6. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Container Workflow
Container Image
Repository
Application Container
Image
Build
Upload Download
Container
Image
Container
Run
Container
Host
Build
Host
Container OrchestratorContinuous Delivery System

7. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Container Images
• Like a VM image
• Immutable Bundle
• Application code plus dependencies
• Portable across different hosts

8. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Running Container
• Like a virtual machine
• Protected from host and other applications via:
– Isolation of Owned Resources – Namespaces
– Restriction of Actions – Capabilities
– Limits on Shared Resources – Cgroups

9. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Isolation – Namespaces + Access Control
• Linux Namespaces
– User
– Ipc
– Uts
– Network
– Mount
– Process
• Access Control: SELinux/AppArmor
– Prevents access to other resources in the event of a namespace escape

10. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Restriction – Capabilities + Setuid + Seccomp
• Drop linux capabilities
• Setuid Setgid to limited user and group
• Restrict access to syscalls via seccomp

11. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Limits – Cgroups + Resource Limits
• Cgroups are primarily used to control
– Memory
– CPU
– Block IO
• Other Resource Limits
– Prevent filling up disk (quotas)
– Prevent overloading host daemons (rate limiting)
– Prevent network saturation (qos)

12. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Container Orchestrator
• Primarily designed for stateless microservices
• Schedules work across a fleet of machines
• Keeps multiple copies of an application running
• Allows for dynamic scaling of application
• Defines how applications can communicate

13. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Containerization Operational Changes
• Rebuild instead of security patch
• Different monitoring tools
• Deployment as immutable artifacts
• Build tooling vs configuration management

14. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Worst Practices
• Developers without an operations mentality
• Multiple applications per container
• Lack of CI/CD automation
• Container Bloat
• Handling of security vulnerabilities
Confidential – Oracle Internal/Restricted/Highly Restricted 14

15. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Oracle – Open Source Contributions
railcar
Alternative Docker runtime
implemented in Rust following
the Open Container Initiative OCI-
Runtime Spec
smith
A simple command line utility for
building microcontainers from rpm
packages or Open Container
Initiative images.
crashcart
A simple command line utility
that lets you side load an image
with linux binaries into an existing
container. Facilitates easier
debugging

16. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
INTRO TO MICROCONTAINERS

17. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | 17
Microcontainers

18. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
What’s wrong with big containers?
• Large images
– Often > 1GB in size
• Bloat
– I wanted a banana. I got the banana and a jungle and an 800lb Gorilla 
• Privilege escalation
– whole Linux user space  bigger attack surface
• Vulnerability management
– What files do I really need to patch?

19. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
A Microcontainer
• Contains only
– Single executable
– Dependencies (of the executable)
• Runs with a read only root filesystem
• Files are all owned and read by a single user

20. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Result
• Small image
– Eliminate layers  reduce complexity
• Fast, easy distribution
• Smaller attack surface
• Certainty over vulnerabilities

21. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
BUILDING MICROCONTAINERS
Demo

22. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Links
• https://blogs.oracle.com/developers/the-microcontainer-manifesto
• https://github.com/oracle/smith
• https://hackernoon.com/how-to-build-a-tiny-httpd-container-
ae622c37db39

23. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
CONTAINER DEBUGGING
Demo

24. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Links
• https://blogs.oracle.com/developers/hardcore-container-debugging
• https://github.com/oracle/crashcart
• http://man7.org/linux/man-pages/man1/nsenter.1.html

25. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Safe Harbor Statement
The preceding is intended to outline our general product direction. It is intended for
information purposes only, and may not be incorporated into any contract. It is not a
commitment to deliver any material, code, or functionality, and should not be relied upon
in making purchasing decisions. The development, release, and timing of any features or
functionality described for Oracle’s products remains at the sole discretion of Oracle.

26. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
QUESTIONS & ANSWERS

27. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
More Information
• Contact Us:
vish.ishaya@oracle.com
tj.fontaine@oracle.com