By Andy Wingo. It used to be that to set up a serious network, you needed to stock racks and racks with specialized proprietary single-purpose boxes. This was because only specialized hardware could handle the hundreds of gigabits per second that might flow through any given box. Things have changed. With the rise of cheap commodity Xeon-based servers and widespread availability of 10 gigabit network cards, an off-the-shelf server with a few NICs can now handle the workload. The age of open source software-driven routers is fully here -- but it doesn't look like what we thought it would, 10 years ago. We thought it would be Linux everywhere, but it turns out that Linux's networking stack is just too slow. To get around this problem, modern high-speed software switches bypass the kernel entirely, instead booting network cards and handling traffic entirely from user-space. The up-side of this is that now we have the possibility of using pleasant, hackable, open source, standalone software stacks to deliver network applications that are tailored to specific needs. This talk presents Snabb, a toolkit for building user-space network functions. Snabb is entirely written in the expressive Lua language, minimizing the amount of code that you have to write to get stuff done. Snabb specifically uses the LuaJIT implementation of Lua, giving us excellent code generation as well as efficient access to low-level binary data and AVX2 assembly generation. Snabb's goal is to be "rewritable software": software that's so simple that you could explain it to someone and they could write their own. By the end of the presentation, you too should have this feeling. We will also describe how Snabb is used in practice in major telecoms and ISPs to provide IPv6 transition technologies to entire countries. Using Snabb allowed a small team of open-source hackers to ship a product that competed favorably against offerings from traditional network vendors. (c) linux.conf.au 2017, CC-BY-SA Hobart, 16-20 January 2017 https://linux.conf.au

1. Production
Snabb
Simple, fast software networking
with Snabb
20 January 2017 – linux.conf.au
Andy Wingo wingo@igalia.com
@andywingo

2. hey
hacker
User-space networking is for us!
Snabb is a great way to do it!
Make a thing with Snabb!

3. (hi)story You are an ISP
The distant past: the year 2000
To set up: you lease DSL exchanges,
bandwidth, core routers
Mission accomplished!

4. (hi)story The distant past: the year 2005
You still pay for DSL, bandwidth,
routers
Also you have some boxes doing
VoIP (more cash)

5. (hi)story The distant past: the year 2010
You still pay for DSL, bandwidth,
routers, VoIP
OMG TV!!!
Also we are running out of IPv4!!!
Also the subscriber fee is still the
same!!!!!!!

6. (hi)story Trend: ISPs have to do more (VoIP,
TV, VOD, cloud, carrier NAT)
“Doing more”: more expensive
boxes in the rack ($70k/port?)
Same story with many other users
Isn’t there a better way?

7. material
conditions
In the meantime, commodity
hardware caught up
Xeon dual-socket, >12 core/
socket
❧
Many 10Gbps PCIe network cards
(NICs)
❧
100-200 Gbps/server
10-15 million packets per second
(MPPS) per core+NIC pair
70 ns/packet
Let’s do it!

8. alternate
(hi)story
The teleology of open source: “one
day this will all run Linux”
Conventional wisdom: if I walk the
racks of a big ISP, it’s probably all
Linux

9. linux? The teleology of open source: “one
day this will all run Linux”
Conventional wisdom: if I walk the
racks of a big ISP, it’s probably all
Linux
Q: The hardware is ready for 10
MPPS on a core. Is Linux?

10. not
linux
The teleology of open source: “one
day this will all run Linux”
Conventional wisdom: if I walk the
racks of a big ISP, it’s probably all
Linux
Q: The hardware is ready for 10
MPPS on a core. Is Linux?
A: Nope

11. why
not
linux
Heavyweight networking stack
System/user barrier splits your
single network function into two
programs
Associated communication costs

12. user-
space
networking
Cut Linux-the-kernel out of the
picture; bring up card from user
space
tell Linux to forget about this PCI
device
❧
mmap device’s PCI registers into
address space
❧
poke registers as needed❧
set up a ring buffer for receive/
transmit
❧
profit!❧

13. (hi)story
time
The distant past: the year 2017
Multiple open source user-space
networking projects having success
Prominent ones: Snabb (2012), DPDK
(2012), VPP/fd.io (2016)
Deutsche Telekom’s TeraStream:
Vendors provide network functions
as software, not physical machines
How do software network functions
work?

14. aside Snabb aims to be rewritable software
The hard part: searching program-
space for elegant hacks
“Is that all? I could rewrite that in a
weekend.”

15. nutshell A snabb program consists of a graph
of apps
Apps are connected by directional
links
A snabb program processes packets
in units of breaths

16. local Intel82599 =
require("apps.intel.intel_app").Intel82599
local PcapFilter =
require("apps.packet_filter.pcap_filter").PcapFilter
local c = config.new()
config.app(c, "nic", Intel82599, {pciaddr="82:00.0"})
config.app(c, "filter", PcapFilter, {filter="tcp port 80"})
config.link(c, "nic.tx -> filter.input")
config.link(c, "filter.output -> nic.rx")
engine.configure(c)
while true do engine.breathe() end

17. breaths Each breath has two phases:
inhale a batch of packets into the
network
❧
process those packets❧
To inhale, run pull functions on
apps that have them
To process, run push functions on
apps that have them

18. function Intel82599:pull ()
for i = 1, engine.pull_npackets do
if not self.dev:can_receive() then
break
end
local pkt = self.dev:receive()
link.transmit(self.output.tx, pkt)
end
end

19. function PcapFilter:push ()
while not link.empty(self.input.rx) do
local p = link.receive(self.input.rx)
if self.accept_fn(p.data, p.length) then
link.transmit(self.output.tx, p)
else
packet.free(p)
end
end
end

20. packets struct packet {
uint16_t length;
unsigned char data[10*1024];
};

21. links struct link {
struct packet *packets[1024];
// the next element to be read
int read;
// the next element to be written
int write;
};
// (Some statistics counters elided)

22. voilà At this point, you can rewrite Snabb
(Please do!)
But you might want to use it as-is...

23. tao Snabby design principles
Simple > Complex❧
Small > Large❧
Commodity > Proprietary❧

24. simple Compose network functions from
simple parts
intel10g | reassemble | filter |
fragment | intel10g
Apps independently developed
Linked together at run-time
Communicating over simple
interfaces (packets and links)

25. small Early code budget: 10000 lines
Build in a minute
Constraints driving creativity
Secret weapon: Lua via LuaJIT
High performance with minimal fuss

26. small Minimize dependencies
1 minute make budget includes Snabb
and all deps (luajit, pflua, ljsyscall,
dynasm)
Deliverable is single binary
./snabb --help
./snabb top
./snabb lwaftr run ...

27. small Writing our own drivers, in Lua
User-space networking
The data plane is our domain, not
the kernel’s
❧
Not DPDK’s either!❧
Fits in 10000-line budget❧

28. commodity What’s special about a Snabb
network function?
Not the platform (assume recent
Xeon)
Not the NIC (just need a driver to
inhale some packets)
Not Snabb itself (it’s Apache 2.0)

29. commodity Open data sheets
Intel 82599 10Gb
Mellanox ConnectX-4 (10, 25, 40,
100Gb)
Also Linux tap interfaces, virtio host
and guest

30. commodity Prefer CPU over NIC where possible
Commoditize NICs – no offload
Double down on 64-bit x86 servers

31. status Going on 5 years old
27 patch authors last year, 1400 non-
merge commits
Deployed in a dozen sites or so
Biggest programs: NFV virtual
switch, lwAFTR IPv6 transition core
router, SWITCH.ch VPN
New in 2016: multi-process, guest
support, 100G, control plane
integration

32. production Igalia developed “lwAFTR”
(lightweight address family
translation router)
Central router component of
“lightweight 4-over-6” deployment
lw4o6: IPv4-as-a-service over pure
IPv6 network
Think of it like a big carrier-grade
NAT
20Gbps, 4MPPS per core

33. challenges (1) Make it fast
(2) Make it not lose any packets
(3) Make it integrate
(4) Make it scale up and out

34. fast LuaJIT does most of the work
App graph plays to LuaJIT’s
strengths: lots of little loops
Loop-invariant code motion boils
away Lua dynamism
❧
Trace compilation punches
through procedural and data
abstractions
❧
Scalar replacement eliminates all
intermediate allocations
❧

35. fast Speed tips could fill a talk
Prefer FFI data structures (Lua
arrays can be fine too)
Avoid data dependency chains
4MPPS: 250 ns/packet
One memory reference: 80ns
Example: hash table lookups

36. lossless Max average latency for 100 packets
at 4MPPS: 25 us
Max latency (512-packet receive ring
buffer): 128 us
Avoid allocation
Avoid syscalls
Avoid preemption – reserved CPU
cores, no hyperthreads
Avoid faults – NUMA / TLB /
hugepages
Lots of tuning

37. integrate Operators have monitoring and
control infrastructure – command
line necessary but not sufficient
Snabb now does enough YANG to
integrate with an external NETCONF
agents
Runtime configuration and state
query, update
Avoid packet loss via multi-process
protocol

38. scale 2017 is the year of 100G in
production Snabb; multiple
coordinated data-plane processes
Also horizontal scaling via BGP/
ECMP: terabit lw4o6 deployments
Work in progress!

39. more Pflua: tcpdump / BPF compiler (now
with native codegen!)
NFV: fast virtual switch
Perf tuning: “x-ray diffraction” of
internal CPU structure via PMU
registers and timelines
DynASM: generating machine code
at run-time optimized for particular
data structures
Automated benchmarking via Nix,
Hydra, and RMarkdown!
[Your cool hack here!]

40. thanks! Make a thing with Snabb!
git clone https://github.com/SnabbCo/snabb
cd snabb
make
wingo@igalia.com
@andywingo

41. oh no here comes the hidden track!

42. Storytime! Modern x86: who’s winning?
Clock speed same since years ago
Main memory just as far away

43. HPC
people
are
winning
“We need to do work on data... but
there’s just so much of it and it’s
really far away.”
Three primary improvements:
CPU can work on more data per
cycle, once data in registers
❧
CPU can load more data per
cycle, once it’s in cache
❧
CPU can make more parallel
fetches to L3 and RAM at once
❧

44. Networking
folks
can
win
too
Instead of chasing zero-copy, tying
yourself to ever-more-proprietary
features of your NIC, just take the hit
once: DDIO into L3.
Copy if you need to – copies with L3
not expensive.
Software will eat the world!

45. Networking
folks
can
win
too
Once in L3, you have:
wide loads and stores via AVX2
and soon AVX-512 (64 bytes!)
❧
pretty good instruction-level
parallelism: up to 16 concurrent
L2 misses per core on haswell
❧
wide SIMD: checksum in
software!
❧
software, not firmware❧