Presentation given at the 2017 LinuxCon China With the booming of Container technology, it brings obvious advantages for cloud: simple and faster deployment, portability and lightweight cost. But the networking challenges are significant. Users need to restructure their network and support container deployment with current cloud framework, like container and VMs. In this presentation, we will introduce new container networking solution, which provides one management framework to work with different network componenets through Open/friendly modelling mechnism. iCAN can simplify network deployment and management with most orchestration systems and a variety of data plane components, and design extendsible architect to define and validate Service Level Agreement(SLA) for cloud native applications, which is important factor for enterprise to deliver successful and stable service via containers.

1. Simplify Networking for Containers
叶磊 曹水
华为 中央软件院 云网络实验室

2. 2
The Nature of Container Network

3. 3
cloud native and containerised micro-services
high density/
dynamic
complex deployment
scenarios
online monitoring
and control
E2E Monitoring
VM Containers
Public
Cloud
Private
Cloud
L2/L3 Overlay Tunnel
SLA (Application to
Application)
more applications and micro services
are deployed in containers

4. 4
deployment complexity
public clouds:
AWS/Azure/HEC
NFV: SR-
IOV/L2/L3
private clouds:
openstack/vmware/
baremetal
simpleflatcontainer
networkmodel:CNI
complexdeployment
scenarios

5. 5
deployment complexity
public clouds:
AWS/Azure/HEC
NFV: SR-
IOV/L2/L3
private clouds:
openstack/vmware/
baremetal
simpleflatcontainer
networkmodel:CNI
complexdeployment
scenarios
existing solutions are
suitable for limited cases
with hard-coded “plugins”
require a flexible solution that
always adapts the best technology
based on specific situation

6. 6
Neutron
Kuryr
Bare
mental
Traditional OS
Socket
OpenStack
Backend
TCP/IP
Stack
Container
OS
Socket
TCP/IP
Stack
vSwitch
OVERLAY
NICs
Driver
Underlay
vRouter
Underlay
Kuryr
OVERLAY
Cloud Provider
XEN
KVM
IRONIC
Container OS
Socket
vNIC
DRIVER
TCP/IP
Stack
Container OS
Socket
vNIC
DRIVER
TCP/IP
Stack
Container OS
NIC
DRIVER
Network
Stack (Iaas)
vSwitch
OVERLAY
VPC
vRouter
vSwitch
OVERLAY
vSwitch
OVERLAY
Traditional
OS
Socket
TCP/IP
Stack
Bridge
OVS L2
OVERLAY OVERLAY OVERLAY
Socket
TCP/IP
Stack
SND
backend
HostGW
SuSE12
Socket
Native
Driver
TCP/IP
Stack
Bridge
OVS L2
OVERLAY
Container
OS
L2
Bare
mental
Bare
mental
Hetero
OS
Pass
through
Contain
erOS
DPDK
API
VF
DPDK
PMD
Containe
rOS
DPDK
API
vNIC
DPDK
PMD
VF
DPDK
PMD
VF PassThrough
OVERLAY L2
Container
OS
DPDK
API
vNIC
DPDK
PMD
VF
DPDK
PMD
Cloud
Provider
Bare mental
OVERLAY
Container
OS
Socket
TCP/IP
Stack
vSwitch
OVERLAY
NICs
Driver
L2
Container
OS
vSwitch
OVERLAY
NICs
Driver
Bare Mental Host
Socket
TCP/IP
Stack
vSwitch
L2
NICs
Driver
Socket
TCP/IP
Stack
vSwitch
OVERLAY
vSwitch
L2
Public
Cloud
Private
Cloud
How we deal with so many scenarios for
containers?

7. 7
Kernel Network Stack
vNIC
@Container
vNIC
@Container
vNIC
@Container
Function
feature
Rich, identical to
Kernel
Performance Normal
Compatibility Very good
Customized Network Stack
Customized
Socket Lib
Customized
Socket Lib
Customized
Socket Lib
Function
feature
Normal, according
to Customized Stack
Performance Good, about 3 times
than Kenel
Compatibility Normal, maybe miss
some socket function
DPDK PMD
DPDK
DPDK Client
@ Container
Function
feature
Poor, according to
DPDK application
Performance Very good, identical
to wire speed
Compatibility Poor, only DPDK ENV
DPDK
DPDK Client
@ Container
DPDK
DPDK Client
@ Container
Why we need so many models

8. 8
Our solution: iCAN (intelligent Container
Network)
an extensible framework to
•program various container network data path and
policies
•adapt to different orchestrators
•support end-to-end SLA between containerised
applications

9. 9
iCAN architecture
kubernetes master
iCAN SLA schd ext.
iCAN master
SLA-annotated
policy
etcd
Standard Netwok
Component (SNC)
models
node
iCAN agent
kubernets agent
iCAN monitor
master
node
node
iCAN agent
kubernets agent
node
iCAN agent
kubernets agent
SNC
configurations
monitoring
report
aggregated
report
aggregated
report
aggregated
report

10. 10
CNI Interface Extension
br-intX
Node
PodX
Eth0
br-intY
PodY
Eth1
PhyNet1
PhyNet2
PaaS
①CNI ADD
{
"cniVersion": "0.2.0",
"name": "IDM-M",
"type": "bridge-veth",
// type (plugin) specific
"vlanID": 42,
"ipam": {
"type": "dhcp",
"routes": [ { "dst": "10.3.0.0/16" }, { "dst": "10.4.0.0/16" } ]
}
// args may be ignored by plugins
"args": {
"labels" : {
" phynet " : " Phy_Net1"
}
}
}
{
"cniVersion": "0.2.0",
"name": "IDM-M",
"type": "bridge-veth",
// type (plugin) specific
"vlanID": 42,
"ipam": {
"type": "dhcp",
"routes": [ { "dst": "10.3.0.0/16" }, { "dst": "10.4.0.0/16" } ]
}
// args may be ignored by plugins
"args": {
"labels" : {
" phynet " : " Phy_Net1"
}
}
}
{
"cniVersion": "0.2.0",
"name": "IDM-C",
"type": "bridge-veth",
// type (plugin) specific
"vlanID": 43,
"ipam": {
"type": "dhcp",
"routes": [ { "dst": "10.3.0.0/16" }, { "dst": "10.4.0.0/16" } ]
}
// args may be ignored by plugins
"args": {
"labels" : {
" phynet " : " Phy_Net2"
}
}
}
①CNI ADD
②CNI Network Configuration
②CNI Network Configuration
Once with one ticket Once with multi ticket
1） Parameters on CNI Network Configuration ，
support Once or Multi entry；
2） Reuse the CNI’s common agreement, all
customized fields within ”args” segment；

11. 11
Standard Network Component (SNC) model
abstract for network components in data-path
• interfaces, devices and templates
L2 device
l2 interface
L2 devices:
bridge/macvlan/ovs/…
L3 device
l3 interface
L3 devices:
router/ipvlan/…
L2 dev:linux bridge
L3 dev: IPS
a template for Flannel
data path
l3 interface
l2 paired
interface
l2 interface
l2 paired
interface

12. 12
Unified Framework For Multi Models
Container
MNG
Flannel Plugin Calico Plugin ……
Linux BR
Kernel
VxLAN
Kernel Route
Kernel Route
PGP Route
Sync
Linux BR
Kernel
Route
Container
MNG
Canal Plugin
Linux BR
Kernel VxLAN
Kernel Route
Kernel Route
PGP Route
Sync
GRE Tunnel
IPIP
Tunnel
Flannel Type Calico Type SR-IOV type
User
Stack
SR-IVO
Through
● Existing every Plugins only support its own model
● Though they employ common data module, the
function is isolated
● After deconstruct different data path , we setup a DSL
language to describe them ,using abstracted standard
component
● Unified Framework with Pluggable drivers for additional
vSwitches, Linux BR, SR-IOV, ...

13. 13
Big Pic of Multi-modes && Multi-planes
PHY-NET
PHY-OM PHY-MNG PHY-DATA
NIC NIC
NIC NIC NIC NIC NIC NIC NIC NIC
NICNIC
DPDK
PMD
User
vSwitch
User
Stack
@Host
Container
User
Socket Lib
Container
User
Socket Lib
bonding
DPDK
PMD
User
vSwitch
User
Stack
@Host
Process
APP
Container
User
Socket Lib
Container
User
Socket Lib
Container
DPDK
APP
UIO UIO
DPDK
PMD
Container
DPDK
APP
UIO UIO
DPDK
PMDvSwitch
Container
Kernel
Stack
bonding
Process
APPvSwitch
Container
Kernel
Stack

14. 14
Open stack Neutron Ml2 Solution
Neutron Server
ML2
Plugin
Host A
Linuxbridge
Agent
Host B
Hyper-V
Agent
Host C
Open vSwitch
Agent
Host D
Open vSwitch
Agent
API Network
Neutron Server
ML2
Plugin
Host A
Modular
Agent
Host B
Modular
Agent
Host C
Modular
Agent
Host D
Modular
Agent
API Network
● Existing ML2 Plugin works with existing agents
● Separate agents for Linuxbridge, Open vSwitch,
and Hyper-V
● Combine Open Source Agents, a single agent which can
support Linuxbridge and Open vSwitch
● Pluggable drivers for additional vSwitches, Infiniband, SR-
IOV, ...

15. 15
iCAN Control Plane Integrated with Openstack
Local
Node
Kuberlet
CANAL
Agent
C C C C C C
CANAL Master
Distributed
KV store
(etcd)
Kubernetes
Master
Monitoring
controller
SLA Manager
IPAM
Neutron
controller
Openstack
Neutron Server
Kuryr Agent
Control
Node
Neutron
controller

16. 16
Monitoring based SNC Modeling
pDev
pPort
pIF
pDev
pPort
pIF
vDev
vPort
vIF
pIF
vPort
vIF
vPort
C1
vIF
C2
vIF
vDev
vPort
vIF
pIF
vPort
C3
vIF
•E2E Monitoring
Point Monitor Item
Source Dest
T1
T4
T2
T3
Latency = ((T4 - T1) - (T3 - T2)) / 2
Monitoring on local SNC components ：
Latency：
Generate E2E monitoring data in master node：
Monitorin
g Agent
Monitorin
g Agent
… …
Monitoring
Master
•E2Ethrought：minimal throughput
•E2E Drop rate： deviations between RX
and TX
•Throughput Analysis：data from local
node
 Bandwidth
 Throughput
 Status
 QoS
 CPU utilization

17. 17
Simplify Network SLA modeling
 iCAN provides north bound interfaces for orchestration and applications to define their requirements through PG(Pod Group: a group of pods with
the same functions), Linking (network requirement between PG) , SLA Service types and Service LB Type.
 Given topology and link bandwidth, evaluate the offers when deploying pods. Essentially a evaluation for pod placement, and validate the
deployment.
 2-Tiers Network topology management Underlay Network（Stable and Predictable） and Overlay Network (Customizable and Dynamic)
 Support: bandwidth, latency and drop rate
 Bandwidth <5%
 Latency <10%, more non-deterministic, affected by many factors such as queuing in software switch and hardware, application response,
server IO, etc
Web
Web
DB
DB
Web
Internet
10Mbps (x3)
5Mbps (x6)
Web
Web
DB
DBInternet
10Mbps (x2)
Latency: Low
User 1
User 2
Polices Deployment
Scheduler
validation
Convert link requirement
to node requirement

18. 18
iCAN Container networking
Multi-dimension SLA& Security
Performance Isolation with bandwidth, latency, drop rate(Proactive Network SLA and
Reactive Network SLA )
Security Isolation: VLAN/VXLAN, ACL
Rich Network Support
Powerful network component modeling : SNC and Modeling via Yang
Rich network schemes, support L2, Overlay, NAT, VLAN, L3, BGP, VPC
Accelerated Network Stack
Powerful Monitoring
Implement “monitoring on-demand ”and “E-to-E monitoring” based on the topology
Facilitate on-demand DSL based troubleshooting
Cooperate with the SLA subsystem to assess the SLA quality

19. Copyright©2016 Huawei Technologies Co., Ltd. All Rights Reserved.
The information in this document may contain predictive statements including, without
limitation, statements regarding the future financial and operating results, future product
portfolio, new technology, etc. There are a number of factors that could cause actual
results and developments to differ materially from those expressed or implied in the
predictive statements. Therefore, such information is provided for reference purpose
only and constitutes neither an offer nor an acceptance. Huawei may change the
information at any time without notice.
Thank You.

20. 20
SNC Template Execution Workflow
❶ Network-Agent Local initialized
base on Node Network Pool
Configuration and Network Capability
Strategy， generate Node Network
Capability Configuration(NNCC) .
❷ Node received template
deployment request, check NNCC. If
node can’t meet requirement, return
failure, otherwise will return Network
Configuration Deployment Template
(NCDT) with information ❸；
❹After, send network deployment
request to Network-Element as NCDT
defined, Finally executed by related
network driver;
Linux-
Bridge
Node NetworkPool
Configuration
Name Location
Linux-
Bridge
Host
Linux-
Bridge
Container
IP-Filter Host
OVS Host
NetworkCapability
Strategy
Cap-
Name
Reques
t
Priority
-Index
GRE Linux-
Bridge；
GRE-Ko;
IP-Filter
0
GRE OVS;
IP-Filter
1
VxLAN OVS;
IP-Filter
0
VxLan EVS;IP-
Filter
1
L2-
DHCP
Dnsmas
q;Linux-
Bridge;
0
Cap-
Name
Locati
on
Priorit
y-
Index
GRE Conta
iner
1
GRE Host 1
L2-
DHCP
Host 0
Node Network
Capability
Configuration
NetworkConfiguration DeployTemplate
Resource Parameter
Bridge @Host,VxLan-cap,GRE-cap…….
GRE-Tunnel @Container…….
NetworkConfigurationTemplate
Resource Deploy-with Parameter
Bridge OVS @Host,VxLan-
cap,GRE-
cap…….
GRE-Tunnel GRE-Ko @Container…….
Internal-Link IP-Filter veth
Bridge Linux-Bridge @Container…
Resource Driver-
func
Bridge Create();Del
ete();Config
ure()…..
OVS
Resource Driver-
func
Bridge Create();Del
ete();Config
ure()…..
GRE-
Tunneel
Create();Del
ete();Config
ure()…
IP-
Filter
Resource Driver-
func
Internal-
Link
Create();Del
ete();Config
ure()…..
❷
❸
❹ ❹
❶
❹

21. 21
Modeling for Standard Network Component
Standard Network Component can help to :
Decouple network control with implementation
Replace and upgrade network components seperately
 Provide on-demanding network solution and SLA for application
L3: IPVLAN
L3_IF L3_IF L3_IF
L2_IF
IPVLAN SNC： L3_IF
L3_DEV
L2_IF
L2: MACVLAN
L3_IF L3_IF L3_IF
L2_IF
MACVLAN SNC：L3_IF
L2_DEV
L2_IF
L2-dev
Paired-IF Paired-IF Paired-IF
IFD/IPA
L3: IPS
L3_IF
CALICO SNC：
Paired-IF
L2_DEV
L3_DEV
L3_IF

22. 22
Example: Support with Flannel(VxLAN backend mode)via SNC
Modeling (kernel based Overlay)
== Operating abstraction:
- CreateSubnet() -- get subnet information via etcd API
- L2:SW.CreateDevice() => "l2_sw_dev"
- L2:SW.CreatePort(port_L)
- L2:SW.CreatePort(port_R)
- Overlay:Flannel.CreateDevice() => "flannel_dev"
- Overlay:Flannel.Connect(flannel_dev.inf_L,
l2_sw_dev.port_R)
- Overlay:Flannel.Connect(flannel_dev.inf_R, eth0)
- Link:vNIC-pair.CreateDevice() => "link_dev"
- Link:vNIC-pair.Connect(link_dev.inf_R, l2_sw_dev.port_L)
L2-Device:
vSwitch
Overlay:
flannel
Link-Device:
vNIC-pair
Flannel Template:
Port_L Port_R
SNC interfaces: /* L2:SW device definition */
{
/* members */
string port[];
/* methods */
CreateDevice(); // creat L2:SW device
CreatePort(string port_name);
}
/* Overlay:Flannel device definition */
{
/* members */
string inf_L;
string inf_R;
/* methods */
CreateDevice();
Connect(string inf, string port);
}
/* Link:VNIC-pair device definition */
{
/* members */
string inf_L;
string inf_R;
/* methods */
CreateDevice();
Connect(string inf, string port)
}
Page 12 模板化实例