Big networking vendors have discovered network virtualization for themselves. However, not only hardware appliances, but also open-source solutions have various means of virtualising networks.Hosting an IaaS cloud, you are faced with the challenge of isolating VMs, implementing private internal networks, billing and accounting, firewalls and shaping. And all these challenges should not affect the rest of your (non-virtualized) network. Using OpenVSwitch, you can tackle many of these tasks. In this session, we show you the caveats, but also the exciting possibilities of open-source network virtualization in practical examples.

1. Heartbleed
...and why yours should, too

2. You are in the right session
_ This
is
an
emergency
service
announcement
_ Due
to
events
that
transpired
on
Tuesday
_ I
thought
it‘d
be
good
to
have
some
info
10.04.14 OSDC 2014 2

3. About me
_ Dr.
Christopher
Kunz
_ Studied
CompSci
in
Hannover,
PhD
in
2012
_ Works
as
a
hoster
for
15
years
_ Some
admin
experience
_ Used
to
do
a
lot
of
PHP
_ Author,
„PHP-­‐Sicherheit“,
ed.
1-­‐3
_ And
don‘t
get
me
started
about
swords!
10.04.14 OSDC 2014 3

4. About filoo
_ hQps://www.filoo.de
_ Quickly-­‐growing
hosVng
company
_ Data
center
in
Frankfurt,
Germany
_ Developed
own
IaaS
middleware
_ QEMU/KVM,
OVS,
Ceph
_ Offer
hosVng,
co-­‐locaVon,
cloud
services
_ 100%
subsidiary
of
Thomas-­‐Krenn.AG
_ Visit
their
booth!
10.04.14 OSDC 2014 4

5. Heartbleed in a nutshell
_ A
bug
with
a
cute
name
_ ...and
not
so
cute
effects
_ Pre-­‐auth,
pre-­‐logging
universal
TLS/SSL
bug
_ Introduced
in
OpenSSL
1.0.1a
(2012)
_ Allows
to
make
64kb
memory
dumps
of
the
server‘s
memory
10.04.14 OSDC 2014 5

6. Wait. What?
_ Yes,
remote
memory
dumps
_ Due
to
an
unchecked
buffer
length,
a
TLS
enabled
server
may
dump
memory
contents
to
the
client
_ Limit
of
64k
per
reply
_ MulVple
replies
possible
_ Memdump
may
contain...
_ URLs
and
GET
/
POST
variables
_ Random
excerpts
from
whatever
_ Source
code
of
scripts/whatever
else
_ SSL
cerVficate
private
keys
10.04.14 OSDC 2014 6

7. About DTLS heartbeats
_ RFC
6520,
Transport
Layer
Security
(TLS)
and
Datagram
Transport
Layer
Security
(DTLS)
Heartbeat
Extension
_ Provides
a
heartbeat
for
TLS
(TCP)
and
DTLS
(mostly
UDP)
sessions
_ Intended
to
add
stability
to
unstable
connecVons
and
prevent
renegoVaVons
_ Implemented
in
OpenSSL
as
part
of
a
PhD
thesis
_ Patch
commiQed
Dec
15,
2011
10.04.14 OSDC 2014 7

8. What this bug is not
_ This
is
not
a
crypto
bug
_ At
least
not
in
its
primary
funcVon
_ This
is
not
a
fully
arbitrary
mem
disclosure
_ Only
memory
belonging
to
aQacked
daemon
can
be
dumped
_ This
is
not
a
remote
root
hole
_ Hence
the
relaVvely
low
CVE
score
of
5.0
10.04.14 OSDC 2014 8

9. Anatomy of the bug 1
struct {
HeartbeatMessageType type;
uint16 payload_length;
opaque
payload[HeartbeatMessage.payload_length];
opaque padding[padding_length];
} HeartbeatMessage;
_ From RFC6520:
_ payload_length: The length of the payload.
_ payload: The payload consists of arbitrary content.
10.04.14 OSDC 2014 9

10. Anatomy of the bug 2
_ ssl/d1_both.c,
line
1474+:
buffer = OPENSSL_malloc(1 + 2 + payload +
padding);
bp = buffer;
[..]
memcpy(bp, pl, payload);
_ From: https://github.com/openssl/openssl/commit/
4817504d069b4c5082161b02a22116ad75f822b1
10.04.14 OSDC 2014 10

11. Anatomy of the bug
_ The
heartbeat
extension
allocates
payload+19
bytes
of
memory
_ Copies
pl
bytes
of
arbitrary
user-­‐supplied
data
payload
via
memcpy()
to
construct
response
_ Client
sets
pl
to
65535
_ Client
sends
only
1
byte
of
data
in
payload
_ Response
contains
1
byte
of
client-­‐supplied
payload
_ ...and
64K
of
RAM
from
the
memcpy()
call
_ Analysis
in:
hQp://blog.existenValize.com/diagnosis-­‐of-­‐
the-­‐openssl-­‐heartbleed-­‐bug.html
10.04.14 OSDC 2014 11

12. Test vulnerability
_ Python
script
at:
hQps://gist.github.com/takeshixx/
10107280
_ Can
test
any
SSL/TLS
enabled
TCP
service
_ Has
support
for
StartTLS
(-­‐s
opVon)
_ Conveniently
dumps
64kb
of
memory
for
you
10.04.14 OSDC 2014 12
00d0: 10 00 11 00 23 00 00 00 0F 00 01 01 33 41 31 25 ....#.......3A1%
00e0: 32 43 25 32 32 5F 6D 6F 64 65 25 32 32 25 33 41 2C%22_mode%22%3A
00f0: 25 32 32 6A 73 6F 6E 25 32 32 25 32 43 25 32 32 %22json%22%2C%22
0100: 5F 69 64 25 32 32 25 33 41 25 32 32 70 5F 33 30 _id%22%3A%22p_30
0110: 33 34 35 38 31 38 25 32 32 25 32 43 25 32 32 5F 345818%22%2C%22_
0120: 63 6F 6E 74 61 69 6E 65 72 25 32 32 25 33 41 30 container%22%3A0
0130: 25 32 43 25 32 32 5F 61 63 74 69 6F 6E 25 32 32 %2C%22_action%22
0140: 25 33 41 25 32 32 76 69 65 77 25 32 32 25 32 43 %3A%22view%22%2C

13. Memdump
_ From:
hQps://twiQer.com/markloman/status/
453502888447586304
10.04.14 OSDC 2014 13

14. Memdump
_ Memory
contents
is
non-­‐determinisVc
_ SomeVmes
exciVng,
mostly
boring
_ while true
do python hb-test.py yahoo.com | grep
-C 2 login >> /tmp/out;
sleep 1;
done"
_ Profit!
10.04.14 OSDC 2014 14

15. Detect exploitation
_ No
logging
on
the
machine
_ All
exploitaVon
is
pre-­‐logging,
pre-­‐applicaVon
_ IDS
vendors
are
pushing
out
signatures
already
10.04.14 OSDC 2014 15

16. Affected services
_ Above
all,
SSL-­‐enabled
web
servers
_ Any
that
uses
OpenSSL,
anyway
_ Mail
servers
_ IMAP
over
SSL,
POP
over
SSL,
SMTP
over
SSL,
StartTLS
_ VPN
tunnels
_ OpenVPN
when
using
cert
auth
(maybe?)
_ PotenVally
others
_ IRC
servers,
XMPP,
FTP
over
TLS
_ Android
4.1.1
is
vulnerable
_ OpenSSH
is
not
vulnerable
10.04.14 OSDC 2014 16

17. Linux versions affected
_ OpenSSL
1.0.1
a
thru
f
_ Debian
Wheezy,
Jessie,
Sid
_ Fixed
for
Wheezy
&
Sid
_ Ubuntu
10.04,
12.04,
12.10,
13.10,
14.04
_ Fixed
packages
exist
_ RHEL
6
_ Patch
exists
_ And
all
others
that
ship
OpenSSL
_ Clients
are
also
vulnerable!
10.04.14 OSDC 2014 17

18. Other affected stuff
_ Cisco
devices:
„We
use
Cisco
SSL
which
is
not
OpenSSL.“;
SSL
VPN
products
potenVally
affected
_ Juniper
has
released
fixes
for
their
SSL
VPN,
none
for
J-­‐
Web
etc.
yet
_ Big
IP?
Kemp?
Fritz.Box?
Your
home
NAS?
_ More
info
(hopefully)
here:
hQp://www.kb.cert.org/
vuls/byvendor?searchview&Query=FIELD
+Reference=720951&SearchOrder=4
10.04.14 OSDC 2014 18

19. Mitigation & cleanup
_ First,
upgrade
to
fixed
openssl
_ apt-­‐get
install
openssl
libssl-­‐1.0.0
_ Next,
restart
all
services
that
load
old
lib
_ Use
checkrestart
or
lsof
–n
|
grep
DEL
|
grep
ssl
_ If
you
use
staVc
binaries,
recompile
everything
_ If
you
use
Google‘s
mod_spdy
on
Apache2.2,
don‘t
_ It
has
its
own
staVcally
linked
mod_ssl
which
is
shamefully
out
of
date
10.04.14 OSDC 2014 19

20. What about certs?
_ It
is
possible
that
privkeys
have
leaked
_ If
so,
you
need
to
revoke&reissue
certs
_ Some
CAs
offer
free
reissue
_ If
you
don‘t
have
PFS,
you
have
a
problem
_ AQackers
who
sniffed
your
traffic
might
be
able
to
decode
it
10.04.14 OSDC 2014 20

21. Thank you
_ .Do
not
despair,
there
is
hope!
_ ...and
now,
back
to
our
regular
scheduled
programme!
10.04.14 OSDC 2014 21
hQp://xkcd.com/1353/

22. Software-defined Networking
In an open-source cloud

23. Agenda
_ High-­‐Level
overview:
What
is
this
about?
_ The
use
case
–
virtualized
networks
for
IaaS
_ Intro
to
OpenVSwitch
_ How-­‐to:
Deploy
OpenVSwitch
_ Frontnet,
Backnet,
public
net
_ Firewalling
_ Tying
it
all
together
10.04.14 OSDC 2014 23

24. So what‘s the hype?
_ Sovware-­‐Defined
Networking
is
the
hype
_ I‘m
not
good
with
hype
_ Networking
is
decoupled
from
bare
metal
_ EssenVally
you
virtualize
parts
of
your
network
_ Control
and
data
plane
are
decoupled
_ Many
vendors
jumped
on
the
train
_ HP,
Cisco,
VMWare,
you
name
it
10.04.14 OSDC 2014 24

25. OpenFlow
_ ImperaVve
control
_ Switches
are
dumb
–
they
only
forward
according
to
rules
_ OpenFlow
controllers
make
the
rules
_ First
packet
of
each
type
is
sent
thru
OpenFlow
controller
_ Subsequent
ones
go
directly
through
switch
10.04.14 OSDC 2014 25

26. OpFlex
_ Cisco‘s
answer
to
OpenFlow
_ Other
vendors
on
board:
Citrix,
MSFT,
RHAT,
Canonical
_ Not
on
board:
J,
HP,
Huawei,
vmWare
_ Balance
intelligence
between
switch
and
controller
_ „DeclaraVve
control“;
just
declare
how
you
want
it
and
the
switch
interprets
that
rule
_ IETF
proposed
standard
_ Drav-­‐smith-­‐opflex
_ Open
APIs
_ AltruisVc
goal:
Eliminate
SPOF
(the
controller)
_ EgoisVc
goal:
Sell
smarter
(=$++)
switches
10.04.14 OSDC 2014 26

27. The OSS Contender
_ OpenVSwitch
_ Openvswitch.org
_ Open
Source
_ Apache
2.0
license,
non-­‐viral
_ GPLv2
_ MulVlayer
(2,3)
virtual
switch
_ Supports
lots
of
interesVng
features
_ VLANs,
Ne{low,
sFlow,
LACP,
filtering,
...
10.04.14 OSDC 2014 27

28. OVS Overview
_ Shamelessly
lived
from
[1]
10.04.14 OSDC 2014 28
ovs-vswitchd
OVS Kernel Module
Control Cluster
ovsdb-server
Off-box
User Kernel
Management Protocol (6632/TCP)
OpenFlow (6633/TCP)
Netlink

29. OSVDB
_ Database
holds
configuraVon
items
_ DefiniVons
for
bridges,
tunnels,
interfaces
_ Controller
addresses
_ ConfiguraVon
is
reboot-­‐safe
_ Custom
database
system,
not
MySQLiteMongoDB
_ Speaks
custom
protocol
(OSVDB)
_ Log
based
_ osvdb-­‐tool
show-­‐log
shows
all
changes
_ Nivy
for
debug
/
change
management!
10.04.14 OSDC 2014 29

30. How ovs works
_ ImperaVve
control
_ All
intelligence
is
in
the
controller
_ Data
path
only
carries
out
instrucVons
_ Data
Path
_ Kernel
module
_ Licensed
under
GPLv2
_ Controller
_ Lives
in
userland
_ Licensed
under
Apache
2.0
10.04.14 OSDC 2014 30

31. Flow flow
_ Everything
is
a
flow
_ CombinaVon
of
input
port,
VLAN,
MAC,
IP,
TCP/UDP
port
10.04.14 OSDC 2014 31

32. OVS management
_ Command-­‐line
tools
_ Ovs-­‐vsctl
for
switch
management
_ Ovs-­‐ofctl
for
flow
management
_ Ovsdb-­‐tool
for
database
management
10.04.14 OSDC 2014 32

33. What‘s our angle here?
_ filoo
is
a
hoster.
_ We
host
VMs.
_ VMs
need
networking.
_ See
where
this
goes?
10.04.14 OSDC 2014 33

34. What we wanted
_ Internet-­‐facing
front-­‐net
interface
_ Private
LAN
for
VMs
_ VM
isolaVon
_ Firewalling
_ Traffic
shaping
_ Fine-­‐grained
accounVng
_ Live
migraVon
10.04.14 OSDC 2014 34

35. Overview - physical
10.04.14 OSDC 2014 35
Back-­‐end
switch
Front-­‐end
switch

36. Overview - virtual
10.04.14 OSDC 2014 36
Firewall
Firewall
Firewall

37. Overview – OVS stack
10.04.14 OSDC 2014 37
OVS
OVS
OVS

38. Let‘s get started
_ We
usually
compile
ovs
ourselves
_ There
are
also
packages
in
apt
_ Those
might
work
or
not
_ Download
&
compile
OVS
_ Latest
stable:
2.1.0,
latest
LTS:
1.9.3
_ ./boot.sh
&&
./configure
&&
make
&&
make
install
_ Kernel
module
from
3.3+
_ Enable
in
Kernel
Networking
-­‐>
OpVons
-­‐>
Open
Vswitch
_ modprobe
openvswitch
10.04.14 OSDC 2014 38

39. Let‘s get started 2
_ Set
up
ovs
db
_ Ovsdb-­‐tool
create
conf.db
vswitch.ovsschema
_ Conf.db
is
in
/usr/localetc/openvswitch
_ /usr/src/openvswitch-­‐1.9.3/vswitchd/vswitch.ovsschema
_ Make
sure
ovs-­‐vswitchd
and
ovsdb-­‐server
start
before
networking
_ Add
startup
entries
to
rc.local
_ Remove
networking
from
rc.d
_ start
networking
in
rc.local
10.04.14 OSDC 2014 39

40. Initial bridges
_ Front-­‐net
vlan:
199
_ Same
procedure
for
back-­‐net
VLAN
_ Add
bridges
_ ovs-­‐vsctl
add-­‐br
vmbr1
_ ovs-­‐vsctl
add-­‐port
vmbr1
vlan199
tag=199
_ ovs-­‐vsctl
set
interface
vlan199
type=internal
_ Log
in
via
IPMI
_ ovs-­‐vsctl
add-­‐port
vmbr1
eth1
_ Machine
is
offline
now
_ Modify
physical
switching
10.04.14 OSDC 2014 40

41. VM networking
_ We
use
KVM/QEMU
_ Add
the
TAP
interface
_ /sbin/ip
tuntap
add
dev
tap1i0d0
mode
tap
user
fcms
_ qemu-­‐system-­‐x86_64
...
-­‐device
rtl8139,mac=00:F1:70:00:00:10,netdev=vlan0d0
-­‐netdev
type=tap,id=vlan0d0,ifname=tap1i0d0
_ Bring
up
the
port
_ /usr/local/bin/ovs-­‐vsctl
add-­‐port
vmbr0
tap1i0d0
199
other_config:stp-­‐enable=false
10.04.14 OSDC 2014 41

42. From TAP to port to flow
_ We
have
a
tap
interface
tap1i0d0
_ Find
the
corresponding
bridge
port:
_ ovs-­‐ofctl
show
vmbr0
|
grep
tap1i0d0
_ 1820(tap1i0d0):
addr:fa:7a:67:e3:5d:€
_ Now
we
have
a
port
number:
1820
_ We
use
this
port
for
flow
management
10.04.14 OSDC 2014 42

43. Multiple interfaces
_ Add
more
TAP
interfaces
_ Assign
one
VLAN
per
customer
_ Internal
network
across
VMs
on
same
node
_ Make
VLAN
known
on
inter-­‐node
switches
_ Via
whatever
switch
automaVon
you
have
_ Cross-­‐node
internal
networking
_ VLAN
limits
apply
–
hard
cut
at
~4090
_ Overlay
networks
to
the
rescue
10.04.14 OSDC 2014 43

44. Prevent MAC spoofing
_ PORT=1820
_ ovs-­‐ofctl
add-­‐flow
vmbr0
"in_port="${PORT}"
arp
idle_Vmeout=0
priority=39500
acVon=resubmit("$
{PORT}",2)“
_ ovs-­‐ofctl
add-­‐flow
vmbr0
"in_port="${PORT}"
table=2
arp
priority=200
idle_Vmeout=0
arp_sha=00:F1:70:00:00:10
nw_src=192.168.1.1
acVon=normal"
_ ovs-­‐ofctl
add-­‐flow
vmbr0
"in_port="${PORT}"
table=2
priority=100
idle_Vmeout=0
acVon=drop"
10.04.14 OSDC 2014 44
We
know
this
MAC
because
we
control
the
hypervisor!
We
know
this
address
too!

45. Caveats for MAC/ARP
_ SomeVmes
you
want
customers
to
spoof
_ HA
soluVons
that
switch
„cluster
IP
addresses“
_ You
can
cater
for
this
in
case
you
know
the
corresponding
MACs
_ Assign
sequenVal
MACs
and
wildcard
_ Or
set
specific
rules
_ OpVonal
„HA
feature“
for
VMs
_ Never
allow
customers
to
wildcard
here!
10.04.14 OSDC 2014 45

46. Firewalling with flows
_ ovs-­‐ofctl
add-­‐flow
vmbr0
"in_port="${PORT}"
table=1
tcp
idle_Vmeout=0
nw_dst=192.168.12.13/32
nw_src=192.168.1.123/32
tp_dst="80"
priority=38000
acVon=drop“
_ From
192.168.1.123
_ To
192.168.12.13
_ Port
80
_ Drop
10.04.14 OSDC 2014 46

47. Port ranges
_ ovs-­‐ofctl
add-­‐flow
vmbr0
"in_port="${PORT}"
table=1
tcp
idle_Vmeout=0
nw_src=192.168.1.123/32
nw_dst=192.168.12.13/24
tp_src="0x05E8/0xFFFC"
priority=37960
acVon=drop“
_ Source
192.168.1.123
_ DesVnaVon
192.168.12.0/24
_ Source
port
=
0x05E8/0xFFFC
_ 0x05E8/0xFFFC
=
1512/65532
_ Port
1512
–
1516
_ OVS
1.11
supports
„Megaflows“,
i.e
universal
wildcarding
10.04.14 OSDC 2014 47

48. Default accept
_ ovs-­‐ofctl
add-­‐flow
vmbr0
"in_port="${PORT}"
table=1
priority=100
acVon=normal“
_ Fallthru
rule
_ Match
everything
else
10.04.14 OSDC 2014 48

49. Accounting
_ We
grab
interface
counters
from
the
tap
interfaces
_ You
can
also
use
Ne{low/sFlow
or
ipfix
_ We
didn‘t
go
there
yet,
experiences
welcome
10.04.14 OSDC 2014 49

50. Shaping
_ Simple
shaping:
_ ovs-­‐vsctl
set
Interface
tap0
ingress_policing_rate=100000
_ ovs-­‐vsctl
set
Interface
tap0
ingress_policing_burst=1000
_ QoS
policies:
_ ovs-­‐vsctl
set
port
eth1
qos=@newqos
id=@newqos
create
qos
type=linux-­‐htb
other-­‐config:max-­‐rate=200000000
queues=0=@q0,1=@q1
_ We
don‘t
do
QoS
policies,
shaping
works
mostly
as
intended
10.04.14 OSDC 2014 50

51. Live migration
_ We
don‘t
actually
do
OVS‘s
own
live
migraVon
_ Start
VM
on
target
host
in
suspend-­‐to-­‐RAM
mode
_ Stop
VM
on
losing
host;
down
interface
_ Resume
VM
on
target
host
_ There
are
live
migraVon
mechanisms
in
OVS
_ L2
based
_ Inter-­‐OVS
GRE
tunnel
_ Honestly,
I
have
no
clue.
10.04.14 OSDC 2014 51

52. Thank you
_ I
hope
you
learned
something
_ If
not,
I
hope
you
had
a
laugh
at
my
expense
_ If
neither,
I‘m
really
sorry.
Beer?
_ QuesVons?
10.04.14 OSDC 2014 52

53. Literature
_ [1]
hQp://openvswitch.org/slides/
OpenStack-­‐131107.pdf
–
OVS
Deep
Dive
_ OVS
IntroducVon:
hQp://horms.net/projects/
openvswitch/2010-­‐10/openvswitch.en.pdf
10.04.14 OSDC 2014 53