This document discusses integrating programmable packet processing (P4), traditional software (C), and hardware acceleration using Agilio SmartNICs with OpenStack networking. It reviews traditional OpenStack networking options and their performance issues. It then discusses how P4, C extensions, and SmartNICs can provide flexible, high-performance networking by offloading or extending the OpenStack networking datapaths like OVS and Contrail vRouter. Examples are provided of running P4/C firmware on the SmartNIC to implement a virtual switch or extending existing software. Integration with OpenStack and implications are discussed throughout.

1. ©2016 Open-NFP 1
Stacks and Layers:
Integrating P4, C, OVS and OpenStack
Johann Tönsing
September 21, 2016

2. ©2016 Open-NFP 2
Agenda
• Review of Traditional Cloud Networking Stacks
OVS (with/without connection tracking), Contrail vRouter, SR-IOV, VirtIO…
• Issues: Flexibility, Performance, CPU Utilization
• Accelerating and Offloading Cloud Networking using Agilio™ SmartNICs
• Implementation Details: Hardware and Software Architectures
• Introducing Additional Programmability
• Implications of Supporting P4, C and eBPF Extensions
• Results: Performance Comparisons
• Update on Integration / Open Sourcing Activities
• Conclusions

3. ©2016 Open-NFP 3
Traditional OpenStack Networking Options
Forwarding /
Virtual Switching
Technology
OpenStack
Control Plane
Forwarding Overlays (Tunnels)
Security Groups
(Microsegmentation,
Stateful Firewalling)
SR-IOV with
regular NIC
Vendor specific plugin Limited
E.g. MAC/VLAN
based directing
None None
OVS
ML2 plugin
(optional: SDN controller)
OVN driver
L2 / L3
using OpenFlow
match/action tables
VXLAN, GRE,
(for OVN) GENEVE
…
Emerging:
NSH (for NFV)
OVS 2.5+ can interface
to Linux conntrack
(Previously Linux bridge
with iptables/nftables)
Contrail vRouter
Contrail driver
(via Contrail controller)
L2 / L3
using dedicated
forwarding tables
MPLS in UDP,
MPLS in GRE,
VXLAN
…
Built in, via flow table
Additional features: load balancing, NAT etc.

4. ©2016 Open-NFP 4
OpenStack Networking Options Evaluated
Forwarding /
Virtual Switching
Technology
Traditional Approach Agilio™ SmartNIC Accelerated Approach
SR-IOV
Limited expressiveness to direct traffic to
VMs (no support for general match/action
rules, tunnel termination, stateful firewalling)
High throughput
No VM migration support
Full OVS or vRouter virtual switching incl.
tunnel termination, stateless/stateful firewalling
and SR-IOV based data delivery to VMs
High throughput
VirtIO integration, supporting VM migration
OVS
and
Contrail vRouter
High expressiveness - match/action,
tunnels, stateless/stateful firewalling etc.
Limited throughput
High CPU utilization (e.g. 50% of cores)
Same expressiveness - match/action, tunnels,
stateless/stateful firewalling etc.
Higher throughput (~5x higher)
Lower CPU utilization (~10x lower)

5. ©2016 Open-NFP 5
Agilio™ CX SmartNIC Hardware
• Optimized for standard server based cloud data centers
• Low Profile Half Length PCIe form factor, power < 25W
• Based on Netronome’s NFP-4xxx silicon (72 C programmable cores, 8 threads each)
• 2GB DRAM for lookup tables / state tables (millions of entries)
• Dataplane fully implemented in software
1x 40GbE 2x 40GbE2x 10GbE 2x 25GbE
Also available: Agilio™ LX 2x40G / 1x100G with dual PCIe interfaces, 120 cores, 8GB DRAM…

6. ©2016 Open-NFP 6
Offload Model: Agilio™ OVS Acceleration
Open vSwitch Subsystem
OVS
Agent
OpenFlow
Virtual Machine
Virtual Machine
Virtual Machine
x86 Kernel
x86 Userspace
PCIe
Virtual Machine
SR-IOV /
VirtIO VFs
SR-IOV /
VirtIO VFs
Agilio™
SmartNIC
Apps
Apps
1
1
netdev or DPDK
netdev or DPDK
Apps
netdev or DPDK
Apps
netdev or DPDK
Apps
netdev or DPDK
OVS CLI
Callable
API
1 Configuration via controller, CLI, or Callable API
2
2 OVS userspace agent populates kernel cache
(Nova, Neutron)
Execute
Action
Open vSwitch Datapath
Execute Action
(e.g. Entunnel,
Deliver to VM,
Send to Port)
Conn
track
3 Offload datapath: copy match tables, sync stats
3
5 Offload connection tracking: synchronize state
5
Conn
track
FTP
SIP
4 Flow tracking: per-microflow state learning
4
Self Learning
Exact Match
Flow Tracker
Miss
Hit
OVS
Kernel DP
Match/Act
OVS
Kernel DP
Match/Act
Miss
Miss

7. ©2016 Open-NFP
vRouter Datapath
vRouter
Flow
Classify
Execute Action
(e.g. Entunnel,
Deliver to VM,
Send to Port)
vRouter
Forwarding
vRouter
Forwarding
7
Agilio™ vRouter Acceleration
vRouter
Subsystem
vRouter
Agent
Virtual Machine
Virtual Machine
Virtual Machine
x86 Kernel
x86 Userspace
PCIe
Virtual Machine
SR-IOV /
VirtIO VFs
SR-IOV /
VirtIO VFs
Agilio™
SmartNIC
Apps
Apps
1
1
netdev or DPDK
netdev or DPDK
Apps
netdev or DPDK
Apps
netdev or DPDK
Apps
netdev or DPDK
vRouter
CLI
1 Configuration via controller or CLI: BGP over XMPP
3
3 Userspace agent populates kernel tables
(Nova, Neutron)
Execute
Action
4 Offload flows: copy tables, sync stats
4
vRouter
Flow
Classify
Miss
Miss
vRouter
Forwarding
vRouter
Forwarding
Notes:
- Control protocol: BGP over XMPP
- Tunnel protocols: VXLAN, MPLS over UDP / GRE
- Distinct policy (flows - ACLs) and forwarding (IP/MPLS)
- Flow table is used to implement stateful firewalling
2
2 Offload forwarding: pre-emptively copied

8. ©2016 Open-NFP 8
Offload Existing Datapath: Characteristics
•Flexibility defined by server’s existing datapath software
•OVS: Configure match/action tables (forwarding / policies)
•Contrail vRouter: Configure forwarding and policies separately
•Integration via drivers / plugins
•OVS: OpenStack ML2 plugin (with/without SDN controller)
•OVS: OpenStack driver for OVN
•Contrail vRouter: OpenStack driver for Contrail
Extend OpenStack to support new concept - SR-IOV path
directly to VM while offloading virtual switching to NIC

9. ©2016 Open-NFP 9
SmartNIC Firmware: Pre-programmed or Custom
Compiler Debugger
Run-Time
app.P4 app.C
Editor
• Firmware can be developed in P4 and/or C
• To discuss: integrating custom programmed
datapaths (implemented in P4 / C etc.) into
existing code (OVS / OpenStack…)
• SmartNIC with dynamically downloadable
firmware
Contrail OVS
OpenStack ONOS ODL
Linux BSD
• OVS / Contrail / Linux eBPF datapath on host
can be accelerated by SmartNIC
D
P
D
K
eBPF

10. ©2016 Open-NFP 10
Example: P4 “main” implementing a simple NIC
header_type eth_hdr {
fields {
dst : 48;
src : 48;
etype : 16;
}
}
header eth_hdr eth;
parser start {
return eth_parse;
}
parser eth_parse {
extract(eth);
return ingress;
}
action drop_act() {
drop();
}
action fwd_act(port) {
modify_field(standard_metadata.egress_spec,
port);
}
table in_tbl {
reads {
standard_metadata.ingress_port : exact;
}
actions {
fwd_act;
drop_act;
}
}
control ingress {
apply(in_tbl);
}

11. ©2016 Open-NFP
Datapath
Program
in
HL lang e.g. P4
• Config
time
Compiler back end
• Protocol Independent Forwarding (PIF)
Configured (“Programmed”) Datapath
• Programs in language(s) like P4 describe datapath
• Parse tree => protocol independent
• Match/action tables (control flow arranges table sequence)
• Packet metadata, per table or global state
• QoS
Datapath
Program
in IR
Compiler front end
11
Protocol Independent Forwarding => Flexibly Programmed Datapath
• Predefined Protocol Forwarding
• OpenFlow specification defines protocols / fields,
match/action behavior, overall control flow
(tables can influence)
• Set of supported protocols fixed by
implementation
OpenFlow
• Either way - need “southbound” interface
for run-time interaction with switch...
• Populate tables, receive statistics + events
OpenFlow++
OpenFlow
Controller
• Run
time

12. ©2016 Open-NFP 12
Extending OVS using P4/C Plugins
Open vSwitch Subsystem
OVS
Agent
OpenFlow
Virtual Machine
Virtual Machine
Virtual Machine
x86 Kernel
x86 Userspace
PCIe
Virtual Machine
SR-IOV /
VirtIO VFs
SR-IOV /
VirtIO VFs
Agilio™
SmartNIC
Apps
Apps
1
1
netdev or DPDK
netdev or DPDK
Apps
netdev or DPDK
Apps
netdev or DPDK
Apps
netdev or DPDK
OVS CLI
Callable
API
1 Configuration via controller, CLI, or Callable API
2
2 OVS userspace agent populates kernel cache
(Nova, Neutron)
Execute
Action
Open vSwitch Datapath
Execute Action
(e.g. Entunnel,
Deliver to VM,
Send to Port)
3 Offload datapath: copy match tables, sync stats
3
OVS
Kernel DP
Match/Act
Datapath
Extension
or Plugin
P4 / C
in Sandbox
DP Ext.
4 Datapath extension software
4
4
OVS
Kernel DP
Match/Act
Miss
Miss

13. ©2016 Open-NFP 13
Plugin in Datapath: Characteristics
• Some flexibility
– Easy to implement custom actions - e.g. filter / modify packet
– Difficult to implement custom classification (combined with OVS)
– Can implement inner protocols (OVS must support outer protocols)
• Integration effort varies
– Can model as custom port
» Similar to VNF in network (on dedicated interface or attached to tunnel)
from controller perspective => least intrusive
– Can model as custom action
» Requires OpenFlow protocol extension (controller side, OVS side)
» May require changes to OVN, OpenStack ML2 plugin / net driver, etc.

14. ©2016 Open-NFP 14
OVS “on” SmartNIC P4 Datapath
Open vSwitch Subsystem
OVS
Agent
OpenFlow
Virtual Machine
Virtual Machine
Virtual Machine
x86 Kernel
x86 Userspace
PCIe
Virtual Machine
SR-IOV /
VirtIO VFs
SR-IOV /
VirtIO VFs
Agilio™
SmartNIC
Apps
Apps
1
netdev or DPDK
netdev or DPDK
Apps
netdev or DPDK
Apps
netdev or DPDK
Apps
netdev or DPDK
OVS CLI
Callable
API
(Nova, Neutron)
Execute
Action
P4 Generated Datapath
Execute
P4
Action
OVS
Kernel DP
Match/Act
P4
Matching
Fallback
Fallback

15. ©2016 Open-NFP 15
P4 Datapath on SmartNIC: Characteristics
• Some flexibility
– In theory easy to implement offloaded behavior on SmartNIC
(new protocols, new actions etc.) - using high productivity
language (P4)
– Note however OpenFlow matching is more flexible - can
augment any table at any time with any field / action
– Furthermore limited to what OVS on host supports (C code)
• Integration effort modest
– Already done if offloading existing OVS code
– Must extend OpenFlow+OVSDB or OVN if enhancing OVS

16. ©2016 Open-NFP 16
P4 “into” OVS Datapath
Open vSwitch Subsystem
OVS
Agent
OpenFlow
Virtual Machine
Virtual Machine
Virtual Machine
x86 Kernel
x86 Userspace
PCIe
Virtual Machine
SR-IOV /
VirtIO VFs
SR-IOV /
VirtIO VFs
Agilio™
SmartNIC
Apps
Apps
1
netdev or DPDK
netdev or DPDK
Apps
netdev or DPDK
Apps
netdev or DPDK
Apps
netdev or DPDK
OVS CLI
Callable
API
(Nova, Neutron)
P4 Generated Datapath
Execute
P4 / OVS
Action
Execute
P4 / OVS
Action
P4 / OVS
Matching
P4 / OVS
Matching
Fallback
Fallback

17. ©2016 Open-NFP 17
P4 Datapath in Kernel: Characteristics
• Mixed flexibility
– Easy to implement behavior (new protocols, new actions etc.) by
using high productivity language (P4)
– Note however OpenFlow matching is more flexible - can
augment any table at any time with any field / action
» Regenerate program on demand to correspond to OpenFlow tables
» Implement program based on assumed model
• Integration effort considerable
– Need to re-implement OVS on P4
(TBD how compatible userspace will be with regular OVS)
– Offloading easier once infrastructure in place

18. ©2016 Open-NFP 18
P4 “instead of” OVS (Datapath and Control)
Host Code
P4 Agent
Virtual Machine
Virtual Machine
Virtual Machine
x86 Kernel
x86 Userspace
PCIe
Virtual Machine
SR-IOV /
VirtIO VFs
SR-IOV /
VirtIO VFs
Agilio™
SmartNIC
Apps
Apps
1
netdev or DPDK
netdev or DPDK
Apps
netdev or DPDK
Apps
netdev or DPDK
Apps
netdev or DPDK
(Nova, Neutron)
P4 Generated Datapath
Execute
Action
Execute
Action
P4
or eBPF
or XDP
P4
Matching
or eBPF…
Fallback
Fallback
1 Protocol(s) to be discussed
(could become callable API)
Other open issues:
- Downloading programs via
OpenStack or other systems
- Scheduling VMs to run on
nodes with acceleration hardware
(Nova)
Acceleration discussion started at
OpenStack Spring 2016 (Austin)
or offloaded eBPF / XDP…

19. ©2016 Open-NFP 19
New Datapath/Control: Characteristics
• Full flexibility
– Easy to implement behavior (new protocols, new actions etc.)
by using high productivity language (P4)
– Can deploy completely different control plane
• Integration effort considerable
– New infrastructure required in OpenStack - new ML2 plugin /
driver, new southbound protocol or API, etc.

20. ©2016 Open-NFP
SmartNIC Firmware (P4/C)
20
Example of Fully Customized Datapath (P4 / C)
Run-Time
Interface
Server (x86 - Linux)
PCIe
Agilio™ SmartNIC
Virtual Machine 1
VNF
Kernel
Mode
(C)
netdev
P4 / C
Development
Environment
Edit - Debug
Control
App
Populate
tables,
display
statistics
Security
µVNF (C)
Timestamp
µVNF (C)
Latency Stats
µVNF (C)
Virtual Machine 2
VNF
User
Mode
(C)
DPDK
Timestamp
µVNF (C)
Latency Stats
µVNF (C)
Match
Protocol
Meter
Other
TCP
Concepts:
• P4 and C running on SmartNIC
implements datapath - e.g.
defines protocols, match / action
behavior
• Datapath steers traffic to VNFs
running on x86 server and
on SmartNIC

21. ©2016 Open-NFP 21
OVS Throughput vs. Number of Rules
5
10
15
20
25
30
OVS in Kernel
Space
OVS in User Space
on DPDK
100 Wildcard Rules
1000 Wildcard Rules
10000 Wildcard Rules
64000 Wildcard Rules
MillionsofPacketsperSecond
12 CPU Cores
12 CPU Cores
OVS Offloaded to
Agilio™ CX-4000
1 CPU Core
5X Throughput Improvement + 90% CPU Savings
OVS L2/L3 Forwarding to 8 VMs with 64K Flows

22. ©2016 Open-NFP 22
Efficiency: Throughput Per Server CPU Core
Throughput with single server CPU core
dedicated to network related processing
MillionsofPacketsperSecond
• 50x Efficiency vs. Kernel OVS
• 20x Efficiency vs. User Mode (DPDK) OVS
=> Replace 3-6 racks with 1 rack!
Throughput / efficiency similar for
Agilio™ Contrail vRouter
Benefit for your use case:
search for
“netronome.com roi calculator”

23. ©2016 Open-NFP 23
Integration / Open Sourcing Activities
Area Activities
Linux drivers
VF netdev (kernel device
driver) upstreamed in
kernel 4.5
Patches available for
older kernels, or use
VirtIO
Representative netdev
(for fallback processing
of traffic) proposals
imminent
FreeBSD drivers Kernel device driver implemented
DPDK drivers
Poll mode driver
upstreamed in DPDK 2.2
Patches available for
older DPDK versions
Open vSwitch
acceleration
integration
Hooks for acceleration
proposed in October
2014
Iteration in progress
OpenStack
integration
Proposing plugins and
agents to support virtual
switching acceleration
via RFE process
Integration for OVS in
process - Mirantis,
Ericsson CEE etc.
Integration for Contrail
vRouter in process -
Juniper etc.
Also: P4 / protocol independent forwarding and OpenFlow run-time API, eBPF / XDP acceleration…
Participation appreciated — join us at Linux, Open vSwitch, Contrail vRouter, OpenStack, p4.org, OpenSourceSDN.org, …

24. ©2016 Open-NFP 24
Next Steps
• Use Agilio™ SmartNICs with existing dataplanes
• Use Agilio™ OVS (with / without Conntrack)
• Use Agilio™ vRouter
• Future: Agilio™ eBPF/XDP, etc.
• Program Agilio™ SmartNICs
• Use APIs (on x86 servers) - with above dataplanes
• Program in P4 and/or C (on SmartNIC / on x86)
• Improve performance + free up server resources!

25. ©2016 Open-NFP 25
Thank You!
More information: netronome.com
and, of course: open-nfp.org