"Implementing an IPv6 Enabled Environment for a Public Cloud Tenant" case study I delivered in OpenStack Vancouver Summit (May, 2015) jointly with Anik and Sharmin from Cisco System.

1. Implementing an IPv6 Enabled Environment for a Public Cloud Tenant
Sharmin Choksey (Cisco), Anik Mazumder (Cisco), Shixiong Shang (Nephos6)
* OpenStack is a trademark of OpenStack Foundation

2. Introduction
2
Sharmin
Choksey
Technical
Leader
Cisco
Systems
Inc.
Email:
schoksey@cisco.com
Shixiong
Shang
Chief
Technology
Officer
Nephos6
(A
Cloud
and
IPv6
Company)
Twitter:
@shshang
Email:
shshang@nephos6.com
Anik
Mazumder
Architect
Cisco
Systems
Inc.
Email:
amazumde@cisco.com

3. Agenda
IPv6 in Public Cloud Overview
Architecture and Design
Challenges
Scalability and Performance
Lessons and Learns
Next Steps
3

4. IPv6 in Public Cloud
Overcome public IP (v4) exhaustion problem by moving to IPv6
Build a public cloud without the complexities of overlapping IP address space or NAT
Telcos and Mobile providers need IPv6 to cloud enable their services
Facilitate cloud adoption of highly distributed services like IOE/IOT
Facilitate adoption of IPv6 in public clouds
Allow tenants to embrace IPv6 for their business needs
Facilitate cloud adoption of network centric services
Increasing demand from Asia
4

5. IPv6 in Public Cloud
5

6. Logical Architecture – Icehouse OSP 5 on RHEL 7
6
M M M
Storage)Cluster
RBD)Computes)Nodes
Local)Storage
Local)Compute
Network)Nodes

7. Logical Scope
7
Neutron
Network
Node
Neutron
Network
Node
Open
vSwitch
(br-­‐int)
tap-­‐
interface
qdhcp
namespace
dhcp
agent
metadata
agent
ovs
agent Open
vSwitch
(br-­‐ex)
eth1
Nova
Compute
Node
Nova
Compute
Node
Dual
Stack
VM
vnic
(ipv4
and
ipv6)
Open
vSwitch
(br-­‐int)
Open
vSwitch
(br-­‐ex)
eth1
VLAN
Trunking
HSRP
ovs
agent
IPv6
Router
Advertisement:
IPv6
Prefix
Default
GW
LLA
A=1,
M=0,
O=1
(dhcpv6
stateless)
Linux
Bridge
(qbr)
DHCPv6
Info-­‐Request
DNSMASQ
dhcpv6
stateless
server
ipv4
dhcp
server
DHCPv6
Info-­‐Reply
(from
dnsmasq)
Provider
Networks

8. HSRP
Challenges - Security: Reconnaissance Attack
8
VLAN
Trunking
Nova
Compute
Node
eth1
Linux
Bridge
(qbr)
qbrd94e5c3e-­‐94:
flags=4163<UP,BROADCAST,RUNNING,MULTICAST>
mtu
1500
inet6
fe80::493:bff:fead:c085
prefixlen
64
scopeid
0x20<link>
ether
f2:de:90:e5:04:bf
txqueuelen
0
(Ethernet)
ping6
-­‐I
eth0
ff02::1
64
bytes
from
fe80::493:bff:fead:c085:
icmp_seq=1
ttl=64
time=0.046
ms
SLAAC
auto-­‐configures
various
ports
(qbr-­‐*,
qvb-­‐*,
qvo-­‐*,
int-­‐br-­‐ex,
phy-­‐br-­‐ex)
on
Compute
node
with
IPv6
Link
Local
Address
Open
vSwitch
(br-­‐int)
Open
vSwitch
(br-­‐ex)
Hello,
my
neighbors!
Hacker
can
try
to
gain
the
access
to
the
hypervisor
via
the
IPv6
Link
Local
Address
ssh
root@fe80::493:bff:fead:c085%eth0
echo
'net.ipv6.conf.default.disable_ipv6=1'
>>
/etc/sysctl.conf
echo
'net.ipv6.conf.all.disable_ipv6
=
1'
>>
/etc/sysctl.conf
sysctl
-­‐p

9. Challenges - Security: RA Guard
9
Neutron
Network
Node
Open
vSwitch
(br-­‐int)
tap-­‐
interface
qdhcp
namespace
DNSMASQ
dhcpv6
stateless
server
ipv4
dhcp
server
Nova
Compute
Node
Dual
Stack
VM
vnic
(ipv4
and
ipv6)
Open
vSwitch
(br-­‐int)
Open
vSwitch
(br-­‐ex)
VLAN
Trunking
Linux
Bridge
(qbr)
Nova
Compute
Node
Open
vSwitch
(br-­‐int)
Open
vSwitch
(br-­‐ex)
Linux
Bridge
(qbr)
Bogus
IPv6
RA
(i.e.
Blackhole
tenant
IPv6
traffic!)
Legitimate
IPv6
RA
(from
upstream
router)
eth1 eth1
Open
vSwitch
(br-­‐ex)
HSRP

10. Challenges - Security: DHCPv6 Guard
10
Neutron
Network
Node
qdhcp
namespace
eth1
Nova
Compute
Node
Dual
Stack
VM
vnic
(ipv4
and
ipv6)
eth1
VLAN
Trunking
Nova
Compute
Node
DHCPv6
Info-­‐Request
Bogus
DHCPv6
Info-­‐Reply
(i.e.
Poison
tenant
nameserver
entry)
DHCPv6
Info-­‐Reply
(from
dnsmasq)
ff02::1:2
udp/547
DNSMASQ
dhcpv6
stateless
server
ipv4
dhcp
server
Open
vSwitch
(br-­‐int)
tap-­‐
interface
Open
vSwitch
(br-­‐int)
Open
vSwitch
(br-­‐ex)
Linux
Bridge
(qbr)
Open
vSwitch
(br-­‐int)
Open
vSwitch
(br-­‐ex)
Linux
Bridge
(qbr)
Open
vSwitch
(br-­‐ex)
HSRP

11. Changes on Network Infrastructure
11
vlan
803
name
customer-­‐1
vlan
configuration
800
no
ip
igmp
snooping
optimise-­‐multicast-­‐flood
vpc
domain
5
peer-­‐switch
role
priority
1
system-­‐priority
100
peer-­‐keepalive
destination
1.1.1.2
source
1.1.1.1
vrf
vpc
peer-­‐gateway
auto-­‐recovery
ip
arp
synchronize
ipv6
nd
synchronize
interface
Vlan803
mtu
9216
vrf
member
customer
no
ip
redirects
ip
address
192.168.254.2/24
ipv6
address
2001:db8:cafe:b::2/64
ipv6
nd
other-­‐config-­‐flag
ipv6
nd
prefix
2001:db8:cafe:b::/64
no
ipv6
redirects
ip
arp
timeout
1740
hsrp
version
2
hsrp
803
…
hsrp
803
ipv6
authentication
md5
key-­‐chain
hsrp_auth
preempt
delay
minimum
600
priority
110
forwarding-­‐threshold
lower
0
upper
0
ip
autoconfig
track
1
decrement
20
no
shutdown

12. Operational Challenges
12
Image support (SLAAC + dhcpv6 Stateless)
– CentOS (6,7) RHEL (6,7) W2K8 Ubuntu(Trusty)
– SLAAC supported by all
– DHCP client behavior needed fix for CentOS,
RHEL and Ubuntu
IPv6 guest enablement criteria
– Incorrect usage of host level sysctl flags
– Gating criteria has no effect on IPv6 traffic
forwarding
IPv6 system of record inconsistencies
– Dual-stacking an existing IPv4-only network
– IPv6 SLAAC on the pre-existing IPv4 VMs
– SOR inconsistencies + missing IPv6 fire-walling
Subnet validation between v4 and v6
– force_gateway_on_subnet flag
– Incorrect validation for IP in Subnet of an IPv4
scheme against an IPv6 address
Compute host
– Stability failures e.g. tap interfaces DOWN
“Use
the
force,
Read
the
source”

13. Scale Testing Tools and Process
13
Objectives
– Test bed of 4K interfaces for IPv4 &IPv6 each
– Generate ICMP/ICMP6 traffic for the interfaces
– Test dhcp-agent (dnsmasq) resiliency
– Test metada-agent resiliency
– Test bed stability over a period of time
Scenarios
– 2K dual stack, dual vNic VMs
– 50 concurrent VM boots/reboots
– Total of 8K interfaces across 3 networks
performing dhcp offers/acks, dns-info, dhcp
renewals
– ICMP/ICMP6 across all 8K interfaces
Process/Tools
– iPerf scripts
– In-house scale test python package
– Supports concurrent operations
– VM boot, reboot, icmp/6, discovery, console_log
– verification for cloud-init customizations

14. Scale Testing Topology / Hardware Specs
14
OpenStack
Controllers
✓ 35
Controller
VMs
on
Service
Cloud
C240
servers
Compute
Capacity
(approx
single
core
vCPU
per
Node)
✓ 2
x
B200s
(2x10
Physical
CPU,
256G
Mem)
✓ 16
x
C220
(2x10
Physical
CPU,
256G
Mem)
✓ Oversubscription
ratios
(4.0x
cpu,
1.5x
ram)
Network
Nodes
✓ 4
x
B200M3
(all
neutron
agents)
✓ DHCP
agents
per
network
2
✓ 3
provider
vlan
networks
(IPv4,
IPv6)
Ceph
Cluster
(Shared)
✓ 3
x
C220
for
Mon
✓ 3
x
C220
Rados
GW
✓ 6
x
C240
OSDs
Test
VM
Configuration
✓ Flavor
Specs
(1
vCPU,
1G
RAM,
5G
Disk)
✓ Image
(cirros-­‐0.3.3-­‐x86_64-­‐disk)
✓ 2
vNics
per
Dual
Stack
VM

15. Scale Testing - VM distribution / Boot times
15

16. Performance and Scale - PING
16
IPv4 Min Response Time
ResponseTime(ms)
0
0.065
0.13
0.195
0.26
Number of VMs
0 25 50 75 100
IPv4 Avg Response Time
ResponseTime(ms)
0
0.15
0.3
0.45
0.6
Number of VMs
0 25 50 75 100
IPv4 Max Response Time
ResponseTime(ms)
0
2.5
5
7.5
10
Number of VMs
0 25 50 75 100
IPv6 Min Response Time
ResponseTime(ms)
0
0.065
0.13
0.195
0.26
Number of VMs
0 25 50 75 100
IPv6 Avg Response Time
ResponseTime(ms)
0
0.15
0.3
0.45
0.6
Number of VMs
0 25 50 75 100
IPv6 Max Response Time
ResponseTime(ms)
0
2.5
5
7.5
10
Number of VMs
0 25 50 75 100

17. IPv4 v.s. IPv6 Throughput Within A Compute Node
17
Note:
All
tests
were
run
for
200
secs
based
on
1470
payload
size
Average
IPv4
TCP
throughput*:
13.1
Gbits/sec
Average
IPv6
TCP
throughput*:
12.7
Gbits/sec
IPv4 UDP Throughput
Throughput(Mbits/sec)
0
175
350
525
700
Number of Samples
0 25 50 75 100
IPv6 UDP Throughput
Throughput(Mbits/sec)
0
175
350
525
700
Number of Samples
0 25 50 75 100
Average
IPv4
UDP
throughput*:
648
Mbits/sec
Average
IPv6
UDP
throughput*:
603
Mbits/sec
Note:
All
tests
were
run
for
200
secs
based
on
1450
payload
size
IPv6 TCP Throughput
Throughput(Gbits/sec)
0
4
8
12
16
Number of Samples
0 25 50 75 100
IPv4 TCP Throughput
Throughput(Gbits/sec)
0
4
8
12
16
Number of Samples
0 25 50 75 100

18. IPv4 v.s. IPv6 Throughput Between Two Compute Nodes
18
IPv4 TCP Throughput
Throughput(Gbits/sec)
0
2.5
5
7.5
10
Number of Samples
0 25 50 75 100
IPv6 TCP Throughput
Throughput(Gbits/sec)
0
2.5
5
7.5
10
Number of Samples
0 25 50 75 100
Note:
All
tests
were
run
for
200
secs
based
on
1470
payload
size
Average
IPv4
TCP
throughput*:
8.57
Gbits/sec
Average
IPv6
TCP
throughput*:
8.29
Gbits/sec
IPv4 UDP Throughput
Throughput(Mbits/sec)
0
200
400
600
800
Number of Samples
0 25 50 75 100
IPv6 UDP Throughput
Throughput(Mbits/sec)
0
200
400
600
800
Number of Samples
0 25 50 75 100
Average
IPv4
UDP
throughput*:
692
Mbits/sec
Average
IPv6
UDP
throughput*:
682
Mbits/sec
Note:
All
tests
were
run
for
200
secs
based
on
1450
payload
size

19. Value Adding to Icehouse Release
19
Filled Feature Gaps
– API validation on Neutron Server is not
adequate
– DHCPv6 Guard
Bridged Gaps on Unit Test
– Dnsmasq process launch for IPv6 subnet in
SLAAC mode was not included
– Security group and ip6table rules were not
verified for DHCPv6 Stateless mode
Fixed bugs
– DHCPv6 Stateful and DHCPv6 Stateless modes
are treated the same
– DHCP agent cannot reload dnsmasq process
properly during subnet addition/deletion
– IPv6 default route is still statically inserted
Enhanced IPv6 Testing Capability
– A total of 22 Tempest functional/API/negative
test cases
– A total of 14 Rally scalability/performance test
scenarios
We
will
contribute
back
to
the
community!

20. Lessons and Learns
Community provides good reference architecture to the adopter. However, customization
maybe be required
Security, performance/scalability and operations should be taken into the consideration as part
of the design
Process doesn’t always introduce overhead. Right SDLC process can provide the quality
assurance
We need think about how IPv6 solves a problem, NOT how to solve the problem of IPv6
We invite YOU to share YOUR lessons and learns, instead of features and functionalities, to
accelerate the adoption of IPv6
20

21. Next Steps
OpenStack tenant networking
Native L3 connectivity between tenant networks without the need for NAT
Direct routing to Internet from Tenant networks without the need for NAT
Allow tenant to choose between deploying IPv6 only, IPv4 only or Dual Stack networks
Allow cloud provider to centrally manage tenant IPv6 address space – we do not have the
problem of overlapping IP address space with IPv6
Allow multiple prefixes
Allow private interconnects between tenant network in cloud and tenant’s own enterprise
network
21

22. Thank
you!