This document provides an overview of containers and container networking. It begins with defining containers and their advantages over virtual machines. It then discusses the container ecosystem including key projects like Docker, CoreOS, and the Open Container Initiative. The document reviews container orchestration systems like Kubernetes, Docker Swarm, and Mesos. It concludes with a demo of OpenStack and containers and a discussion of containers on Cisco platforms.

2. Intro to Containers & Container Networking
Rohit Agarwalla, Senior Technical Leader
Duane DeCapite, Director Product Management and Strategy
BRKSDN-2115
BRKSDN-2115 2

3. © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Legal Disclaimer
Many of the products and features described herein
remain in varying stages of development and will be
offered on a when-and-if-available basis.
This roadmap is subject to change at the sole discretion
of Cisco, and Cisco will have no liability for delay in the
delivery or failure to deliver any of the products or
features set forth in this document.
BRKSDN-2115 3

4. • Container Overview
• Container Ecosystems
• Container Orchestration Systems
• Container Networking
• OpenStack and Containers with Demo
• Cisco Microservice Platform (Mantl/Shipped)
• Containers on Cisco Boxes
• Summary /Q&A
Agenda

5. Container Overview

6. © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
New Application Architectures
microservices
monolithic apps
Monolithic Apps Cloud Native Apps
server / hypervisor server clusters, containers
dependencies easy upgrade
stateful microservices
pets cattle
traditional dev and ops teams agile devops teams
BRKSDN-2115 6

7. © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Example: Stackanetes
[1] Alex Polvi, CoreOS. Microservices. https://www.youtube.com/watch?v=DPYJxYulxO4&feature=youtu.be
BRKSDN-2115 7

8. © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Milestones in Container History
2015 CNCF/OCI
Linux Foundation Collaborative
Projects
2014 Rocket (rkt)
CoreOS reference
images/runtime
2013 Docker Tools to build/retrieve images
2011 Cloud Foundry Warden Manage collection of containers
2007 Control Groups Merged into Linux kernel
2004 Solaris
Zones – similar to jails but
snapshot/clone ZFS
2000 FreeBSD Introduced jails
1979 Unix Developed chroot
source: pivotal.io
BRKSDN-2115 8

9. © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Containers – The Building Block
A container is a sandbox environment layered on top of a host OS that provides:
• Isolation – namespaces
• Resource Limits – control groups (cgroups)
[1] Intel Containers 101 (van de Ven)
source: intel
BRKSDN-2115 9

10. © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Linux Containers
• A Linux container lets you run a Linux system
within another Linux system.
• A container is a group of processes on a Linux
machine.
• Those processes form an isolated environment.
• Inside the container, it looks like a VM.
• Outside the container, it looks like normal
processes running on the machine.
• It looks like a VM, but it is more efficient:
Containers = Lightweight Virtualization
Zones
BRKSDN-2115 10

11. © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
App Containers enable Microservices
"The microservice architectural style is an approach to developing a single application as a suite of
small services, each running in its own process and communicating with lightweight mechanisms” 1
5 Architectural Constraints of Microservices2
1. Elastic – be able to scale, up or down, independently of other services in the same application.
2. Resilient – fail without impacting other services in the same application.
3. Composable – offer an interface that is uniform and is designed to support service composition
4. Minimal, and – only contain highly cohesive entities
5. Complete – be functionally complete
“Disruptor: Continuous Delivery with Containerized Microservices” – Adrian Cockcroft
[1] Martin Fowler. Microservices. http://martinfowler.com/articles/microservices.html
[2] Jim Bugwadia. http://nirmata.com/2015/02/microservices-five-architectural-constraints/
[3] Adrian Cockcroft,. On the State of Microservices, DockerCon Europe, 2014 http://thenewstack.io/dockercon-europe-adrian-cockcroft-on-the-state-of-microservices/
BRKSDN-2115 11

12. © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Containers are the Buzz because..
• Load fast
• Cloud Native
• Build once run (almost) anywhere
• Portable (easy to move between clouds)
• Light (share kernel)
• Scalable
• Enable microservices source: https://hub.docker.com
BRKSDN-2115 12

13. © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Containers are “almost” like little Virtual Machines
• Containers have their own network interface (and IP address)
• Can be bridged, routed... just like with Xen, KVM etc.
• Containers have their own file system
• For example a Debian host can run Fedora container (and vice-versa)
• Security: Containers are “isolated” from each other
• Two containers can't see each other (separate namespaces)
• Resource Control: Containers can have dedicated resources
• Soft & hard quotas for RAM, CPU, I/O... (cgroups)
BRKSDN-2115 13

14. © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Containers and Virtual Machines
App A
Bins/
Libs
Hypervisor (Type 2)
Host OS
Server
Host OS
Server
App A’
Bins/
Libs
Guest OS
App B
Bins/
Libs
Guest OS
Bins/Libs Bins/Libs
Container
Control
App
A
App
A’
App
B
App
B’
App
C’
App
C’
VM
Container
Containers are isolated but share OS and where
appropriate bins/libraries
Guest OS
BRKSDN-2115 14

15. © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Isolation: Container Namespaces
PID Namespace
• Processes in a PID namespace don't see
processes of the whole system
• Each pid namespace has a PID #1
• pid namespaces are actually nested
• A given process can have multiple PIDs
• One in each namespace it belongs to
• So you can easily access processes of children
namespace
• Can't see/affect processes in parent/sibling
namespace
Host
Container

16. Container Ecosystems

17. © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Container Ecosystem Overview
BRKSDN-2115 17

18. © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Docker Milestones
• Founded in 2010 by Solomon Hykes with contributions from others
• Originally developed at PaaS provider dotCloud
• Defined first container image standard
• Docker was released as open source in March 2013
• Moby Dock (mascot and logo) created by community in June 2013
• Created first container image repository (Docker Hub)
• Partnership with Red Hat for Fedora/RHEL and OpenShift in September 2013
• Docker, Inc closes $15M Series B funding in January 2014
BRKSDN-2115 18

19. © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Docker Milestones
• 100M container downloads in December 2014
• SocketPlane acquisition (libnetwork) in March 2015
• $95M Series D funding in April 2015
• 1B container downloads in November 2015
• Docker, Inc acquires Unikernal Systems in January 2016
• Docker Cloud launched in March 2016
BRKSDN-2115 19

20. © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Docker Engine
docker engine
• daemon – directly manages the containers on the host
• client – communicates with the docker daemon to control containers
• container – LXC or libcontainer (default)
docker --daemon=true
docker
docker
BRKSDN-2115 20

21. © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Docker Images and Containers
• Images layered via union file
system – enables multiple
layered file systems images to
be seen as one image.
kernel /bootfs
Ubuntu base image
add open-ssl
add apache
writeable
container
read-only
image
layers
copy-on-write
https://docs.docker.com/engine/userguide/storagedriver/imagesandcontainers/
BRKSDN-2115 21

22. © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
CoreOS Milestones
• Founded in January 2013
• Brandon Philips, developer at SUSE and Rackspace
• Alex Polvi, Mozilla, CloudKick, Rackspace
• Defined new container standard(rkt) in 2014
• No daemon
• Additional Security
• App Container (appc)
• Released CoreOS (MicroOS for containers)
BRKSDN-2115 22

23. © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
CoreOS Milestones
Released Tectonic (Kubernetes + CoreOS for Business)
BRKSDN-2115 23

24. © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
App Container Spec
“App Container (appc) is a well-specified and community developed specification that defines an image
format, runtime environment and discovery mechanism for application containers.”
The App Container (appc) spec aims to have the following properties:
• Composable - All tools for downloading, installing, and running containers should be well
integrated, but independent and composable.
• Secure - Isolation should be pluggable, and the cryptographic primitives for strong trust, image
auditing and application identity should exist from day one.
• Decentralized - Discovery of container images should be simple and facilitate a federated
namespace and distributed retrieval. This opens the possibility of alternative protocols, such as
BitTorrent, and deployments to private environments without the requirement of a registry.
• Open - The format and runtime should be well-specified and developed by a community. We want
independent implementations of tools to be able to run the same container consistently.
source: https://github.com/appc/spec/, coreos.com
BRKSDN-2115 24

25. © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
CoreOS/rkt can also run Docker
CoreOS Host
systemd
docker containers
$ sudo rkt –insecure-skip-verify fetch docker://redis
... (docker2aci converts docker image to ACI)
sha512-962bae14761e5e1ec121e4d49d010f29
$ sudo rkt run sha512-962bae14761e5e1ec121e4d49d010f29
$ sudo rkt –insecure-skip-verify fetch docker://ubuntu
$ sudo rkt run –interactive=true <image ID>
BRKSDN-2115 25

26. © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Additional CoreOS Projects
• etcd – sync cluster state, distributed key-value store, lock management, leader election
(Raft). Flannel stores routing in etcd. etcd is used by Kubernetes
• flannel – builds overlay network across machines. Used by Kubernetes.
• fleet – cross-cluster scheduler, combines systemd and etcd into a distributed init
• Tectonic – Kubernetes + CoreOS for business
• Enterprise Registry (powered by Quay.io) – private registry, public and private
options
BRKSDN-2115 26

27. © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Open Container Initiative (OCI)
Source: http://www.opencontainers.org
Project to create open industry standards around container formats and runtime
BRKSDN-2115 27

28. © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Containers are started as a child process of runC
and can be embedded into various other systems
without having to run a Docker daemon
runC is built on libcontainer, the same container
Docker images can be run with runC
Embeddable
Battle Hardened
Docker Compatible
runC: OCI Container Runtime
runC: is a CLI tool for spawning and running containers according to OCP specification
BRKSDN-2115 28

29. © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 29
Docker 1.11 based on OCI Runtime
source: https://blog.docker.com/2016/04/docker-engine-1-11-runc/
BRKSDN-2115

30. © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
OCI Image Spec
source: http://thenewstack.io/open-container-initiative-launches-container-image-format-spec/
Allow Developers to package and sign application containers
Run them in a variety of container engines
Use build tools and execution schemes that best meet their needs
Containers run without modification on rkt, Docker, Kubernetes, Amazon ECS
The project will be based on Docker v2.2 and draws from Core OS’s appc
spec
BRKSDN-2115 30

31. © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Cloud Native Computing Foundation
xxx
“Will create and drive the adoption of a new set of common container
technologies informed by technical merit and end user value, and inspired by
Internet-scale computing”
Platinum Sponsors
source: http://thenewstack.io/open-container-initiative-launches-container-image-format-spec/
BRKSDN-2115 31

32. © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Cloud Native Computing Foundation
Organization marks important milestone to advance essential infrastructure
“Cloud Native Computing Foundation Accepts Kubernetes as first hosted
project; Technical Oversight Committee”
- KubeCon, March 10, 2016 BRKSDN-2115 32

33. © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Cloud Native Computing Foundation
Organization marks important milestone to advance essential infrastructure
“Cloud Native Computing Foundation Accepts Prometheus as second
hosted project; Technical Oversight Committee”
-San Francisco, May 9, 2016
BRKSDN-2115 33

34. © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Intel Clear Containers
• Intel Clear Containers use Intel Virtualization Technology (VT-x) instead of
Namespaces for isolation between containers
• Lightweight hypervisor
• Let go of the PC in the Virtual Machine
• Result: Startup time 30 msec
• Supported rkt Stage 1 (0.8)
• www.clearlinux.org
BRKSDN-2115 34

35. © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 35
Windows Server and Hyper-V Containers (not Linux)
BRKSDN-2115

36. © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Mesosphere Milestones
• Founded in 2013 by Florian Leibert, Ben Hindman and Tobi Knaup, all with web scale
engineering experience from the likes of Twitter and Airbnb.
• 2014 – Headquartered in San Francisco with international operations in Hamburg, Germany
• Released Data Center Operating System (DC/OS) built on top of Mesos, Marathon and
Chronos. Docker & Linux container support.
• 2015 – announced Infinity Stack with Cisco for Big Data
• 2016 - open sourced DC/OS
• 2016 - support for Kubernetes.
• 2016 – Microsoft to build Azure Container Service on DC/OS
BRKSDN-2115 36

37. © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Additional Container OSs
• Red Hat RHEL 7 Project Atomic (March 2015) – fast transactional updates with rollback,
security (SELinux), Docker support, Kubernetes support, super-privileged containers
• Snappy Ubuntu Core (Dec 2014) – fast transactional updates with rollback, security
(AppArmor), Docker support
• VMware Photon (April 2015) – support for Docker, rkt and Garden
BRKSDN-2115 37

38. Container Orchestration Systems

39. © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Container Orchestration Platform
• Cloud native applications
consist several (100s-1000s) of
containers
• Requires a platform that can
provide –
• Clustering and Scheduling
• Networking, Storage and
Security
• Container Management
• Image Registry
Infrastructure Provisioning
Tool
Nodes (on-prem or cloud)
Logging
Monitoring
Operating System
Networking, Storage and
Security
Container Service Deployment
Image Registry
Scheduling
BRKSDN-2115 39

40. © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Container Orchestration Frameworks
List of a few frameworks -
• Docker Swarm
• Google Kubernetes
• CoreOS Tectonic
• Apache Mesos
• Kontena
• Google Container Engine
• Azure Container Service
• Amazon ECS
Selecting a framework depends on -
• Container run time
• Cloud v/s On-Prem
• Image Registry
• Open Source
• Networking
BRKSDN-2115 40

41. Docker Swarm

42. © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Docker and Docker Compose
• Inside Docker
• Docker Client
• Docker Images
• Docker Registry
• Docker Containers
• Docker Hosts
• Docker Compose
• Tool for defining and running multi-
container Docker applications
• Isolate environments on a single host
• Focused on Development and Testing workflows
• Docker Machine
• Automate Infrastructure Provisioning (driver
based EC2, Virtualbox etc)
docker run -it ubuntu /bin/bash
BRKSDN-2115 42

43. © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Common Docker commands
• docker run -it ubuntu:14.04 /bin/bash
• docker run -d ubuntu /bin/sh -c "while true; do echo
hello world; sleep 1; done”
• docker run -d -p 80:5000 training/webapp python
app.py
• docker port nostalgic_morse 5000
• docker ps
• docker logs -f nostalgic_morse
• docker inspect nostalgic_morse (JSON document
containing useful configuration and status
information for the specified container)
• docker start/stop nostalgic_morse
• docker rm nostalgic_morse
• docker images
• docker pull ubuntu
• docker commit -m "Added json gem" -a ”Cisco
Live!" 0b2616b0e5a8 ciscolive!/ubuntu:14.04.1
• FROM ubuntu:14.04
MAINTAINER Cisco Live! <user@cisco.com>
RUN apt-get update && apt-get install -y ruby ruby-
dev
RUN gem install sinatra
• docker build -t ciscolive!/ubuntu:14.04.1 <location
of Dockerfile>
• docker rmi ciscolive!/ubuntu:14.04.1
BRKSDN-2115 43

44. © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Docker Networking
• Bridge network driver (--driver=bridge)
• None network driver (--driver=none)
• Host network driver (--driver=host)
• Overlay network driver (--driver=overlay) – Multi-Host using VXLAN + Libkv
• Remote drivers - means of supporting drivers over a remote transport
• Uses IPTables for container external access and port mappings
BRKSDN-2115 44

45. © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Docker Networking Internals
eth0
iptables :
NAT/port-mapping
Docker0
eth0
C1
eth0
C2
iptables :
NAT/port-mapping
isolatedbridge
eth0
C3
eth0
C4
eth0
iptables :
NAT/port-mapping
docker_gw
eth0
C1
eth1
eth0
C3
eth1
veth pairs veth pairs
veth pairs
Distributed Store (Libkv - Consul, Etcd, Zookeeper)
Serf
Populate
Neighbor
Table
entries
IP address and VXLAN ID allocation
(Network, Subnet) – VXLAN ID
ov-net1 ov-net2
eth0
iptables :
NAT/port-mapping
docker_gw
eth0
C2
eth1
eth0
C4
eth1
veth pairs
ov-net1 ov-net2
Bridge Networking
Overlay Networking
ToR
BRKSDN-2115 45

46. © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Docker Networking Commands
• Three default networks
$ docker network ls
NETWORK ID NAME DRIVER
7fca4eb8c647 bridge bridge
9f904ee27bf5 none null
cf03ee007fb4 host host
• docker network ls
• docker network inspect bridge
• docker inspect --format='{{range
.NetworkSettings.Networks}}{{.IPAdd
ress}}{{end}}' web
• docker network connect/disconnect
my-bridge-network web
• docker network create -d bridge my-
bridge-network
• Docker Network Commands
BRKSDN-2115 46

47. © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
What is Docker Swarm ?
• Native clustering for Docker
that turns a pool of Docker
hosts into a single, virtual host.
• Distributed Store - store
metadata for service discovery
to register machines and
endpoints inside the cluster.
• Discovery Service – provides
node discovery in Swarm
• Scheduler – enables container
placement choices
Swarm Manager
(HA)
Docker Host (s)
Distributed Store (Libkv - Consul, Etcd, Zookeeper)
Docker Host (s)
Swarm Host (s)
Discovery
Service
Scheduler
CLI
API
Compose
Container
docker run swarm manage
docker run -d swarm join --advertise=172.30.0.69:2375 consul://172.30.0.161:8500
BRKSDN-2115 47

48. © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Scheduling - Filters
• Container – container placement, or on the availability of images
on a host
• Affinity
• Dependency
• Port
• Node - characteristics of the Docker host or on the configuration
of the Docker daemon
• Constraints
• Health
BRKSDN-2115 48

49. © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Scheduling – Strategies (Rank)
• Node (CPU, RAM, # of containers)
• Spread
• Bin Pack
• Random
BRKSDN-2115 49

50. © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Docker Universal Control Plane, Data Center
BRKSDN-2115 50

51. Google Kubernetes

52. © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
What is Kubernetes (K8S) ?
• Open Source system started by Google in 2014 for automating deployment,
operations, and scaling of containerized applications
• Written in Go (aka golang)
• Portable
• Public, Private and Hybrid
• Extensible
• Modular, Pluggable
• Self-Healing
• Auto-scaling, Auto-replication
• Contributed to CNCF in July 2015
Source: http://redmonk.com/fryan/2016/03/14/the-further-evolution-of-kubernetes/
BRKSDN-2115 52

53. © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Kubernetes Components
Master -
• Etcd – Distributed Key-Value Watchable
Storage
• API server - Processes REST operations
on the cluster, validates them, and updates
the corresponding objects in etcd
• Scheduler – Pluggable service that
schedules workloads onto nodes
• Controller Manager – Endpoint, Node and
Replication Controllers
Node -
• Kubelet – Manages Pods and
containers on the nodes
• Kube-Proxy – network proxy and
a load balancer
BRKSDN-2115 53

54. © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Kubernetes Deployment
Master(s)
Scheduler
Controller
Manager
Etcd
Node(s)
Docker
Kubelet
API
Kube-Proxy
Pod
Container
API
CLI
UI
Pod
Container
BRKSDN-2115 54

55. © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Kubernetes Concepts
• Cluster - A cluster is a set of physical or virtual machines and other infrastructure
resources used by Kubernetes to run your applications
• Namespaces - partition resources created by users into a logically named group
• Context - context defines a cluster, user, namespace tuple (all three are optional)
• Node - A node is a physical or virtual machine running Kubernetes, onto which pods can
be scheduled
• Pod - A pod is a co-located group of containers and volumes and is the smallest
deployable unit
• Replication controller - manage the lifecycle of pods and ensure that a specified number of
pods are running at any given time, by creating or killing pods as required
• Service – Defines IP address to access a set of pods and does basic load balancing
• Labels – key/value pair that is attached to a resource, such as a pod
BRKSDN-2115 55

56. © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
YAML Config files
apiVersion: v1
kind: Pod
metadata:
name: Front End v1
labels:
app: webapp
role: frontend
version: v1
spec:
containers:
- name: nginx
image: nginx
ports:
- containerPort: 80
Pod
apiVersion: v1
kind: ReplicationController
metadata:
name: Front End v1
spec:
replicas: 2
selector:
app: webapp
role: frontend
version: v1
template:
metadata:
name: Front End v1
labels:
app: webapp
role: frontend
version: v1
spec:
containers:
- name: nginx
image: nginx
ports:
- containerPort: 80
Replication Controller
apiVersion: v1
kind: Service
metadata:
name: Front End
spec:
selector:
app: webapp
role: frontend
version: v1
ports:
protocol: TCP
port: 443
targetPort: 443
Service
BRKSDN-2115 56

57. © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Kubernetes Service, Pods, Replication Controllers,
Labels
Front End ServiceS
app=webapp,role=frontend, version=v1
Front End v1 PodP
app=webapp, role=frontend,
version=v1
Front End v1 PodP
app=webapp, role=frontend,
version=v1
Front End v2 PodP
app=webapp, role=frontend,
version=v2
Front End v1 ControllerR
Desired Count = 2
app=webapp,role=frontend,version=v1
Front End v2 ControllerR
Desired Count = 1
app=webapp,role=frontend,version=v2
BRKSDN-2115 57

58. © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Kubernetes Common Commands
• kubectl config set-cluster e2e --server=https://1.2.3.4
• kubectl create -f ./file.yml (create a resource – namespace, pod, replication
controller, service)
• kubectl run nginx --image=nginx --replicas=5
• kubectl label pods <pod-name> new-label=awesome
• kubectl get namespace, nodes, services, pods, rc
• kubectl describe namespace, nodes, services, pods, rc
• kubectl exec <pod-name> -c <container-name> -- ls
• kubectl logs -f -c ngnix frontend
BRKSDN-2115 58

59. © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Kubernetes Networking
Fundamental requirements -
• All containers can communicate with all other containers without NAT
• All nodes can communicate with all containers (and vice-versa) without NAT
• Enable applications to communicate directly without port forwarding from nodes
to containers
Implementation Details -
• Containers within a pod share a common container network namespace
• IP’s are assigned per pod
• Nodes are assigned unique IP subnets
BRKSDN-2115 59

60. © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Kubernetes Networking Internals
Pod
C1
--net=container:infra
--ipc=container:infra
Communicate using localhost static ports
Shared namespace – IP address, IPC
Infra
10.0.1.3
C2
--net=container:infra
--ipc=container:infra
Container to Container
10.0.1.0/24
Node
10.0.1.4
Pod
C C
10.0.1.3
Pod
C C
10.0.2.0/24
Node
10.0.2.4
Pod
C C
10.0.3.0/24
Node
10.0.3.3
Pod
C C
BRKSDN-2115 60

61. © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Scheduling - Predicates
• Mandatory rules to schedule a new pod on the cluster
Predicate Node’s requirement
PodFitsPorts Needs to be able to host the pod without any port conflicts
PodFitsResources Has enough resources to host the pod
NoDiskConflict Has enough space to fit the pod and the volumes linked
MatchNodeSelector Match the selector query parameter defined in the pod’s description
HostName Has the name of the host parameter defined in the pod’s description
BRKSDN-2115 61

62. © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Scheduling - Priorities
• Used to find the most suitable node to run the pod out of machines shortlisted
based on predicates
Priorities Node(s) considered as the best(s)
LeastRequestedPriority Calculates the percentage of memory and CPU requested by the pods
that are already on the node. The node with the minimum percentage is
the best.
BalancedResourceAllocation Nodes that have a similar memory and CPU usage
ServiceSpreadingPriority Prefers the nodes that have different pods using them
EqualPriority Give an equal priority to all the nodes in the cluster
BRKSDN-2115 62

63. Mesos

64. © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
What is Mesos ?
• Originally developed at UC Berkley AMPLab as a research project
• An open-source Apache project that provides efficient resource allocation across
a cluster
• Provides a layer of abstraction for CPU, Memory, and Storage resources.
Essentially acts as the kernel for a datacenter.
• From a resource perspective,
• Pools server resources to be centrally managed as a single unit
• From an application perspective,
• Dispatches workloads to consume pooled resources
• Described as Data Center Operating System (DCOS)
BRKSDN-2115 64

65. © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
How does Mesos work ?
Elastic
Search
Mesos
Marathon Chronos Spark Hadoop
Node Node Node Node Node Node
Frameworks
Scheduler
Nodes
BRKSDN-2115 65

66. © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Mesos components
• Master
• Manages Mesos Agents
• Enables fine-grained sharing of resources
across frameworks in form of resource
offers
• Agent (Slave)
• Deployed on the nodes that run the tasks
• Distributed Store (zookeeper)
• Maintains cluster state and current master
• Frameworks
• Scheduler and Executor
Mesos Slave
Marathon
Executor
TasksTasks
Mesos Master (HA)
Distributed Store
(Zookeeper)
Marathon
Scheduler
Mesos Slave
Marathon
Executor
TasksTasks
Allocation Module
BRKSDN-2115 67

67. Container Networking

68. © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Container Networking Challenges
• Scale
• Several containers per host/cluster, more IP endpoints
• Speed
• Plumbing must be fast to match container lifecycle management
• Network Management
• Simple network integration with bare-metal, VMs and containers
• Segmentation
• Network tenant isolation
BRKSDN-2115 74

69. © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Container Networking Solutions
Flannel CoreOS
WeaveNet WeaveWorks
OVN VMWare
Contiv Cisco
Calico MetaSwitch Networks
Libnetwork Docker
OpenShift SDN RedHat
Nuage-SDN Nokia
OpenContrail Juniper
Contiv
BRKSDN-2115 75

70. © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Container Network Model (CNM)
Docker Container
Endpoint
Network
Sandbox
Green Network
Docker Container
Endpoint
Network
Sandbox
Blue
Network
• Proposed by Docker to provide
networking abstractions/API for
container networking
• Sandbox contains configuration of a
container's network stack (Linux
network namespace)
• An endpoint is container's interface
into a network (veth pair)
• A network is collection of endpoints
that can communicate with each other
(Linux Bridge, VLAN)
• A container can belong to multiple
endpoints (and therefore multiple
networks)
BRKSDN-2115 76

71. © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
CNM - Details
• CNM allows for co-existence of multiple drivers,
with a network managed by one driver
• IPAM Driver APIs - Request/Release Pool
(local/global), Allocate/Release IP Address
• Network Driver APIs - Network Create/Delete,
Endpoint Create/Delete/Join/Leave
• Capabilities - features the remote IPAM driver
can express during registration with libnetwork
(eg: MAC address)
• Used by Docker tools and other schedulers that
runs standard Docker containers e.g. Mesos
Docker Containerizer
Docker Daemon
Libnetwork (CNM)
Native Drivers Remote Drivers
BRKSDN-2115 77

72. © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Container Network Interface (CNI)
• Proposed by CoreOS as part of appc
specification
• Common interface between container
run time and network plugin
• Gives driver freedom to manipulate
network namespace
• Network described by JSON config
• Plugins support two commands:
- Add Container to Network
- Remove Container from Network
Container
Network
namespace
Driver
plumbing
Kubernetes, Rocket…
Container Network Interface
Plugins
BRKSDN-2115 78

73. © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
CNI (Container Network Interface) - Details
• Provides Container Create/Delete
events
• Need to provide information of network
namespace, container interface name to
the driver to plumb networking
• Networking and IPAM (both
executables) run using the network
configuration file
• Used by Kubernetes, Cloud Foundry,
Weave, Calico, Contiv
$ mkdir -p /etc/cni/net.d
$ cat >/etc/cni/net.d/10-mynet.conf <<EOF
{
"name": "mynet",
"type": "bridge",
"bridge": "cni0",
"isGateway": true,
"ipMasq": true,
"ipam": {
"type": "host-local",
"subnet": "10.22.0.0/16",
"routes": [
{ "dst": "0.0.0.0/0" }
]
}
} EOF
BRKSDN-2115 79

74. Contiv

75. © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Contiv – Introduction
• Container Networking and Storage with Ops Policies
• Provides REST interfaces
• Open Source at https://github.com/contiv
 Container Connectivity
 Policies for networking
 Variety of connectivity options
 Works with Kubernetes,
Docker, Mesos, Nomad
Contiv Networking
 Policy for volume allocation
 Snapshots, IOPs rate-limiting,
Garbage Collection, etc.
 Works with Docker
Contiv Storage
 Node Discovery, Inventory
 Node Life-Cycle Management
 Complete Stack, managed
 Works for cloud, optimized for
Bare-Metal
Contiv Cluster
Contiv UI
BRKSDN-2115 81

76. © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Contiv Network Components
- Distributed Cluster wide function
- Stateless: useful in node failure/restart, upgrade
- Implements cluster wide network and policy
- Manage Global Resources: IPAM, VLAN/VXLAN Pools
- Container Networking for:
. Kubernetes, Mesos, and Swam
- Route Distribution using BGP/EVPN
- Custom openflow pipeline for host networking
Contiv Master
Contiv Host Agent
- Tools to manipulate Contiv objects
- Implements CRUD using REST interface
Contiv CLI/UI
Docker Host
vswitch
TasksC1
Contiv Master (HA)
Docker Host
vswitch
TasksC2
CLI (netctl)/UI
Contiv
netplugin
Contiv
netplugin
BRKSDN-2115 82

77. © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Contiv Commands
• netctl net create contiv-net --subnet=10.1.1.0/24 --gateway=10.1.1.254 --pkt-
tag=100
• netctl policy create ciscolive!_policy
• netctl policy rule-add ciscolive!_policy 1 -direction=in -protocol=tcp -action=deny
• netctl policy rule-add ciscolive!_policy 2 -direction=in -protocol=tcp -port=80 -
action=allow -priority=10
• netctl policy rule-add ciscolive!_policy 3 -direction=in -protocol=tcp -port=443 -
action=allow -priority=10
• netctl group create contiv-net web -policy=ciscolive!_policy
• docker run -itd --net web.contiv-net ubuntu bash
BRKSDN-2115 84

78. FD.IO

79. © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
VPP overview
• User-space network packet processing stack for commodity hardware
• Uses Intel DPDK open source network I/O technology
• High performance
• ~14 Mpps (Millions packets per second) from a single x86-64 CPU core
• Processing multiple packets at a time to optimize the use of commodity hardware
resources.
• Uses a completely different software architecture compared to the traditional scalar
approach of processing one packet at a time
• Built on a packet processing graph
• Rich multi-layer networking functionality ― Layer 2, IPv4, and IPv6 forwarding
with large tables,VRFs, multiple types of tunneling etc
• Extensible through the use of plugins for writing new features
BRKSDN-2115 86

80. © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
VPP Operation
• Grab all available packets (pointers)
from device RX ring buffer (using
DPDK)
• Form a frame (vector) comprising
packet indices in received order (similar
packets sampled)
• Process frames using a directed graph
of nodes
• No I-cache thrashing problem
• Mitigates the dependent read latency
problem due to big MAC or IP tables
• Reduces stack depth and addresses D-
cache misses on stack addresses Source: https://www.metaswitch.com/the-switch/fd.io-takes-over-vpp
BRKSDN-2115 88

81. © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Container Integration
• Netlink Server to ‘intercept’ netlink
calls from existing container control
planes
• Netlink Server then programs VPP as
appropriate
• Container control planes can evolve to
interfacing directly with VPP at their
own pace
VPP
ControlPlaneDataPlane
Netlink Server
libnetwork
cni
weave
calico…
flannel
BRKSDN-2115 90

82. Containers and OpenStack

83. © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Kolla Mission Statement
Production-ready containers and deployment tools for operating OpenStack
clouds
“Kolla provides Docker containers and Ansible playbooks to meet
Kolla’s mission”
K LLA
BRKSDN-2115 93

84. © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
• Services include ceph, mariadb, rabbitmq, memcached, glance, keystone, nova, neutron
(ovs & linuxbridge), murano, heat, cinder, swift, ironic
• Ceph backed storage
• Distro choice of CentOS, Oracle Linux, RHEL, Ubuntu
• Deploy the big tent at 100 node scale
• Small runtime dependency footprint of docker-py and docker-engine
• Minimal operational dependencies
Kolla Liberty: Deploy OpenStack Clouds
BRKSDN-2115 94

85. © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
• Security enhancements
• Added upgrade action
• Added reconfigure action
• Diagnostics
• Deployment time reduced 80%
• Named Volumes for persistence
• New services: ElasticSearch, Heka, Kibana, Manila, Mistral
Kolla Mitaka: Operate OpenStack Clouds
BRKSDN-2115 95

86. © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Magnum in OpenStack
Making Containers a First Class Resource
Magnum is First
HTTP/1.1 201 Created
Not Re-Implementing
Orchestration
SSO Experience for Cloud
Users
Multi-Tenant Control and Data
Planes
Asynchronous API
Uses OpenStack Orchestration
(Heat)
Uses OpenStack Identity
(Keystone)
Source: Mid Cycle Meetup
BRKSDN-2115 96

87. © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 97
Magnum: Multi-Tenant CaaS
Heat orchestrates OS image with Docker & k8s
BRKSDN-2115

88. © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 98
Kuryr: Container Networking with Neutron
Docker Libnetwork driver for OpenStack Neutron
BRKSDN-2115

89. © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 99
Calico
vRouter in each compute node (No Overlay)
CNI, libnetwork
BRKSDN-2115

90. © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Cisco Physical Infrastructure
Network VIM
Linux (RHEL OS), Hyper Visor (KVM), Host Packages, Software Defined Storage
NFVI Scope
NetworkCompute (UCS) Storage Ceph
UnifiedManagement
withassurance.
UCSD
API
GUI
Virtual Infrastructure Manager
RHEL OSP
Assurance
Cisco NFVI Solution
Leading Industry Partnerships
Performance Acceleration,
Enhanced Platform Awareness
Certified by Red Hat
Joint Engineering
Integrated platform Design and Validation
Legend
Simple Access to Support
Single Point of Contact
VTS VPP OVSor or or
BRKSDN-2115 100

91. Mantl/Shipped

92. © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
What Is Mantl?
Cisco’s Open Source Containerization Platform
An end to end, cloud agnostic, highly extensible, integrated
stack for running container workloads and big data.
Including deployment automation, security, and monitoring.
Designed to grow into a platform for applications and data
services.
Mantl.io
BRKSDN-2115 102

93. © 2016 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKSDN-2115 103

94. © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
ciscoshipped.io
Interface for cloud native
development
Developer/Operations
friendly
Easy-to-adopt, easy-to-
use
Builds a ‘Developer-
Experience / PaaS layer’
Deployed on top of our
Mantl Container stack
What Is SHIPPED?
BRKSDN-2115 104

95. © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Project Shipped
Project
Management
Continuous
Integration
Source
Control
Continuous
Deployment
Application
Orchestration
Developer
Tools
Service
Assurance
Jira
Rally
GitHub
Consul
InfluxDB
Elasticsearch
Apache Spark
ZoomData
LogStash
Issue
Management
Mesos (Cloud Foundry)
Consul
Kubernetes (OS)
Kafka
Terraform
(UCS, CF, OS)
Drone
GitLab
Shipped CICD
GitHub
GitLab
Build Packs
Eclipse
Vagrant
Docker Compose
Cloud Foundry/Openshift
Jira
Rally
GitHub
Collaboration
Cisco Spark
BRKSDN-2115 105

96. Containers on Cisco Boxes

97. © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Containers on Cisco Boxes
(BRKSDN-2116)
BRKSDN-2115 107

98. Summary/Q&A

99. © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 109
• Container ecosystems and orchestrations are evolving
• OCI and CNCF will create common standard
• Containers can be deployed as part of OpenStack
• Kolla and Magnum are key projects
• Visit Cisco booth for more NFVI/Mantl/Contiv demos
Summary/Q&A
BRKSDN-2115

100. © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Complete Your Online Session Evaluation
Don’t forget: Cisco Live sessions will be available
for viewing on-demand after the event at
CiscoLive.com/Online
• Give us your feedback to be
entered into a Daily Survey
Drawing. A daily winner will
receive a $750 Amazon gift card.
• Complete your session surveys
through the Cisco Live mobile
app or from the Session Catalog
on CiscoLive.com/us.
BRKSDN-2115 110

101. © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Continue Your Education
• Demos in the Cisco campus
• Walk-in Self-Paced Labs
• Lunch & Learn
• Meet the Engineer 1:1 meetings
• Related sessions
BRKSDN-2115 111

102. Please join us for the Service Provider Innovation Talk featuring:
Yvette Kanouff | Senior Vice President and General Manager, SP Business
Joe Cozzolino | Senior Vice President, Cisco Services
Thursday, July 14th, 2016
11:30 am - 12:30pm, In the Oceanside A room
What to expect from this innovation talk
• Insights on market trends and forecasts
• Preview of key technologies and capabilities
• Innovative demonstrations of the latest and greatest products
• Better understanding of how Cisco can help you succeed
Register to attend the session live now or
watch the broadcast on cisco.com

103. Thank you

104. © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Sources Include
• BRKDEV-1002, Rosenbloom
• BRKDEV-2116, Brockners
• Pivotal.io
• Intel.com
• Redhat.com
• Docker.com
• Coreos.com
• CNCF.io
• opencontainers.org
• Openstackfoundation.org
BRKSDN-2115 114