mSwitch paper talk at SOSR 2015. Paper: http://dl.acm.org/citation.cfm?id=2775065

1. mSwitch:
A
Highly-­‐Scalable,
Modular
Software
Switch
Michio
Honda
(NetApp)
*
Felipe
Huici
(NEC),
Giuseppe
Lettieri
and
Luigi
Rizzo
(Università
di
Pisa)
ACM
SOSR’15
June
17
*
this
work
was
mostly
done
at
NEC

2. Motivation
• Software
switches
are
important
–Interconnection
between
VMs/containers
and
NICs
–Middleboxes,
SDN,
NFV
• Requirements
–Throughput
(e.g.,
10
Gbps)
–Scalability
(e.g.,
100
ports)
–Flexibility
(e.g.,
forwarding
decision,
packet
modification)
–CPU
efficiency
(e.g.,
allocate
as
many
CPU
resources
to
VMs
as
possible)
Are
existing
software
switches
able
to
meet
these
requirements?
Software Switch
VM VM VM
NIC NIC

3. Existing
Software
Switches
• OS-­‐standard
ones
don’t
provide
high
throughput
1
2
4
6
8
10
60 124 252 508 1020 1514
Throughput(Gbps)
Packet size (bytes, excluding CRC)
FreeBSD bridge
Linux OVS
DPDK vSwitch
VALE
• High
throughput
ones
lack
port
scalability
and/or
flexibility
Finally, CuckooSwitch targets physical NICs (i.e., no virtual
ports), so the experiments presented in that paper are limited
to 8 ports total.
Flexibility: Most of the software switches currently avail-
able do not expressly target a flexible forwarding plane, lim-
iting themselves to L2 forwarding. This is the case for the
standard FreeBSD and Linux bridges, but also for newer
systems such as VALE and CuckooSwitch. Instead, Open
vSwitch supports the OpenFlow protocol, and as such pro-
vides the ability to match packets against a fairly compre-
hensive number of packet headers, and to apply actions to
matching packets. However, as shown in figure 1 and in [19],
Open vSwitch does not yield high throughput.
Throughput CPU Usage Density Flexibility
FreeBSD switch ⇥
p p
⇥
Linux switch ⇥
p p
⇥
Open vSwitch ⇥
p p p
Hyper-Switch ⇥
p
⇥
p
DPDK vSwitch
p
⇥ ⇥
p
CuckooSwitch
p
⇥ ⇥ ⇥
VALE
p p
⇥ ⇥
Table 1. Characteristics of software switches with respect
to throughput, CPU usage, port density and flexibility.
DPDK vSwitch takes the Open vSwitch code base and ac-
celerates it through the use of the DPDK packet framework.

4. • Separation
into
fabric
and
logic
• Fabric:
switches
packets
between
ports
• Logic:
modular
forwarding
decisions
mSwitch
Design
Decisions
4
Switching
fabric
Switching
logic
OS
stack
Sock.
API
Apps
Virtual
Ports
App/VMApp/VM
netmap
API
netmap
API
User
Kernel
NIC
• Interrupt
model
• Efficient,
flexible
CPU
utilization
• Runs
in
the
kernel
• To
efficiently
handle
interrupts
• Integration
with
OS
subsystems
• network
stack,
device
drivers
etc
• Separate,
per-­‐port
packet
buffers
• Isolation
• Copy
is
anyways
inexpensive

5. • Output
queue:
reserve
destination
buffers
w/
lock
and
copy
packets
w/o
lock
• concurrent
senders
can
perform
copy
Scalable
Packet
Switching
Algorithms
• Input
queue:
group
packets
for
each
destination
port
before
forwarding
• For
a
batch
of
input
packets,
lock
each
destination
port
and
access
its
device
register
only
once
5
sender1
sender2
Output
queue

6. Modular
Switching
Logic
• Switching
logics
are
implemented
as
separate
kernel
modules
that
implement
a
lookup
function:
• Return
value
indicates
a
destination
switch
port
index,
drop
or
broadcast
• L2
learning
is
default
but
we
can
change
it
at
anytime
while
the
switch
is
running

7. A
Full
mSwitch
Module
u_int
my_lookup(u_char *buf, const struct net_device *dev)
{
struct ether_hdr *eh;
eh = (struct ether_hdr *)buf;
/* least significant byte */
return eh->ether_dst[0];
}

8. CPU
Utilization
• mSwitch
efficiently
utilizes
CPUs
1
2
3
4
5
6
7
1 2 3 4 5 6 7 8 9
100
200
300
400
500
600
700
800
900
1000
Throughput(Gbps)
CumulativeCPUutilization(%)
# of destination virtual ports (# of CPU cores - 1)
mSwitch
DPDK vSwitch
mSwitch (% CPU)
DPDK vSwitch (% CPU)
NIC
AppApp AppApp
virtual
ports
One
CPU
core
for
the
NIC
and
per
virtual
port

9. Ports
Scalability
• mSwitch
scales
to
many
ports
1
2
3
4
5
6
7
1 10 20 30 40 50 60 70 80 90 100 110 120
10
20
30
40
50
60
70
80
90
100
Throughput(Gbps)
CPUutilization(%)
# of destination virtual ports
mSwitch (Throughput)
VALE (Throughput)
mSwitch (NIC-CPU)
VALE (NIC-CPU)
mSwitch (App-CPU)
VALE (App-CPU)
NIC
AppApp AppApp
virtual
ports
One
CPU
core
for
the
NIC
Another
one
for
all
virtual
ports

10. mSwitch
Module
Use
Cases
10
VPVP VPkernel
NIC
Open vSwitch
datapath
VM
user
VM VM
NIC
VPVPkernel
NIC
UDP/TCP port
filter
App/VM
user
App/VM
NIC
(middlebox for
TCP 80 and 443)
(middlebox for
UDP/TCP 5004)
Mux/Demux
(3 tuple)
VPVP
App+
stack
Sock. API
OS stack
App+
stack
kernel
user
(TCP 22)
(TCP 80) (TCP 53) App
NIC
• Accelerated
Open
vSwitch
datapath
• 3x
speedup
• Filtering
for
virtualized
middleboxes
• Efficiently
directs
relevant
packets
to
middleboxes
on
virtual
ports
• Support
for
user-­‐space
protocols:
• With
isolation
• Can
still
use
OS’s
stack

11. Conclusion
• A
highly-­‐scalable,
modular
software
switch
• Higher
scalability
and
flexibility
compared
to
DPDK
vSwitch
and
VALE
• Already
integrated
into
netmap/VALE
implementation
• https://code.google.com/p/netmap/
• Upstreamed
into
FreeBSD,
works
in
Linux
• All
the
modules
(e.g.,
Open
vSwitch
acceleration)
are
publicly
available
• https://github.com/cnplab
• The
paper
is
open
access:
http://web.sfc.wide.ad.jp/~micchie/papers/a1-­‐honda-­‐sosr15.pdf
• Other
papers
using
mSwitch:
• Martins
et
al.
“ClickOS
and
the
art
of
network
function
virtualization”,
USENIX
NSDI’14
• Honda
et
al.
“Rekindling
network
protocol
innovation
with
user-­‐level
stacks”,
ACM
CCR
201411

12. Module
complexity
and
performance
12
the
ces
an-
re-
ons
S
S
2
4
6
8
10
12
14
1.2 1.5 2.1 2.4 3.2
Throughput(Mpps)
CPU Clock Frequency (Ghz)
Baseline
Filter
L2 learn
3-tuple
mSwitch-OVS
Figure 16: Throughput comparison between di↵er-
ent mSwitch modules for 60 Byte packets.
Measurement
results
for
minimum-­‐sized
packet
forwarding
between
two
NICs