This slide is presented in Dec., 2013 as part of Triangle OpenStack meet up sponsored by Cisco System in Raleigh-Durham area, North Carolina. We did proof of concept back in June, 2013 to evaluate IPv6 readiness of OpenStack as the initial step to make IPv6 and Cloud work together seamlessly. After 6-week of intensive efforts, we enabled OpenStack Grizzly release over IPv6. Later on, we also successfully launched dual-stack VM in Havana release. This slide summarized what problems we tried to tackle and how we resolved them. The presentation is based on the whitepaper we published at: http://www.nephos6.com/pdf/OpenStack-Havana-on-IPv6.pdf. The ideas captured in this slide will be leveraged by OpenStack Neutron IPv6 sub team to fulfill mid-term goals suggested by Neutron IPv6 roadmap. The target release is IceHouse in April, 2014. We will publish more white papers and slides when we reach next milestone. Stay tuned!

1. OpenStack
Havana
On
IPv6
Shixiong
Shang
Randy
Tuttle
Ciprian
Popoviciu
!
Version
1.9.3
© 2013 nephos6 and/or its affiliates. All rights reserved.

2. Agenda
§ Introduction
§ IPv6 and Cloud
§ IPv6 Refreshment
§ Proof of Concept
§ Proposed Blueprint
§ Next Steps
© 2013 nephos6 and/or its affiliates. All rights reserved.
2

3. Introduction
§ Nephos6
§ Ciprian Popoviciu
– Service assurance company
– Founder, CEO
– Founded in June, 2011
– IPv6 expert
– Twitter: @Nephos6
– Twitter: @Nephos6
– Web: http://www.nephos6.com
– Email: chip@nephos6.com
!
!
§ Shixiong Shang
§ Randy Tuttle
– Head of Engineering
– Network Consulting Engineer
– Twitter: @shshang
– Twitter: @randyttl
– Email: shshang@nephos6.com
– Email: rantuttl@cisco.com
© 2013 nephos6 and/or its affiliates. All rights reserved.
3

4. IP Comparison
IPv4
IPv6
Address
32-bit,
128-bit,
Network Address Translation Multiple Scopes
ICMP
ICMP
ICMPv6
Autoconfiguration
DHCP
SLAAC, DHCPv6, DHCP-PD
Routing
RIPv2, OSPFv2, ISIS, MPBGP, EIGRP
RIPng, OSPFv3, ISIS-ST/
MT, MP-BGP, EIGRPv6
IP Multicast
IGMP/PIM/Multicast BGP
MLD/PIM/Multicast BGP,
Scope Identifier
“IPv6
Is
an
Evolution,
Not
a
Revolution
of
the
Internet
Protocol”
© 2013 nephos6 and/or its affiliates. All rights reserved.
4

5. IPv6 and Cloud
IPv6 Strength
Business Value
Sufficient address space
Direct access to resources
Simplified Address Assignment
Native support of multicast and
flow label
New architectural models
}
}
Easier management and lower
operational cost
Great opportunity for innovation
“The
promise
of
Cloud
cannot
be
fully
met
without
IPv6”
© 2013 nephos6 and/or its affiliates. All rights reserved.
5

6. IPv6 Address Auto-Configuration
Our
focus
today!
Working
in
progress!
SLAAC*
DHCPv6
Address Assignment
(non-link-local)
By exchanging Router
Solicitation and Router
Advertisement messages with
neighboring routers.
From DHCPv6 server
Additional Information
None
From DHCPv6 server
Default Gateway
The only way to announce default route is using Router
Advertisement!
Pros
Plug and play
IPv4-like approach, but better
More control
Cons
Doesn’t provide Hostname,
DNS server, WINS, etc.
Operational overhead (extra
DHCP server, HA, etc.)
* StateLess Address AutoConfiguration
© 2013 nephos6 and/or its affiliates. All rights reserved.
6

7. SLAAC
§ RFC 4861 - “Neighbor Discovery for IP Version 6 (IPv6)” and RFC
4862 - “IPv6 Stateless Address Autoconfiguration”
§ Rely on ICMPv6 (IPv6 control plane!)
Host
Router
Solicitation
(RS)
Router
Advertisement
(RA)
subnet
prefix
lifetime
autoconfig
flag
Router Solicitation (RS)
Router Advertisement (RA)
ICMPv6 Type
133
ICMPv6 Type
134
IPv6 Source
A Link Local
IPv6 Source
A Link Local
IPv6 Destination
Link-local scope all-routers
address (FF02::2)
IPv6 Destination
Router
Link-local scope all-nodes
address (FF02::1)
§ VM sends Router Solicitation at boot
time to solicit Router Advertisement
§ Router sends RA to all-nodes address
periodically
§ Default route points to router’s link-local
address
§ Router can also unicast RA back to VM
upon receiving RS
© 2013 nephos6 and/or its affiliates. All rights reserved.
7

8. SLAAC Address Calculation
§ IPv6 SLAAC = network portion (i.e. /64 Prefix in RA) + interface id
(i.e. EUI64)
FA
MAC
Insert
0xFFFE
in
the
middle
FA
1111
EUI-­‐64
IPv6 address
73
83
D9
16
3E
FF
FE
73
83
D9
16
3E
FF
FE
73
83
D9
1000
F8
=
3E
1010
1111
Change
7th
bit
in
OUI
part
16
2001:7:10:180:F816:3EFF:FE73:83D9
© 2013 nephos6 and/or its affiliates. All rights reserved.
8

9. OpenStack IPv6 Readiness
OpenStack Havana
OpenStack Icehouse
Limited IPv6 support out of box
Neutron will support IPv6…
Neutron IPv6 roadmap is still in
preliminary stage
Blueprint: IPv6 Feature Parity
(working in progress…)
No clear IPv6 roadmap for other
OpenStack projects
Neutron-IPv6-Subteam
(ongoing)
Very limited documentation
Biggest risk of all: IPv4 way of thinking
© 2013 nephos6 and/or its affiliates. All rights reserved.
9

10. th
s wi nd
s
u c c e zz ly a
Proof Of Concept
S
i
h Gr a na!
b ot av
H
Mission Statement: To make these two inflection points, IPv6 and Cloud
work together seamlessly!
Motivation
Goals
We are believers
All OpenStack infrastructure nodes
should be able to communicate
with each other by IPv6
What it is v.s. What it should be
OpenStack should be able to spin
up dual-stack VMs in multi-tenant
environment
We are doers…but we are not
hackers, or developers :)
VMs should be able to gain
connectivity to external IPv6
network beyond OpenStack’s
control
© 2013 nephos6 and/or its affiliates. All rights reserved.
10

11. POC Architecture
Controller Node
nova-api
nova-scheduler
nova-consoleauth
Network Node
nova-novncproxy
neutron-dhcp-agent
Common Node
nova-cert
neutron-l3-agent
horizon
nova-conductor
neutron-metadataagent
keystone
cinder
openvswitch
nova-compute
mysql db
glance
neutronopenvswitch-agent
neutronopenvswitch-agent
rabbitmq
neutron-server
dnsmasq
openvswitch
eth0
eth0
7.10.180.101
2001:7:10:180::101
7.10.180.102
2001:7:10:180::102
Management and API network
7.10.180.0/24
2001:7:10:180::/64
Management
and
API
network
eth0
eth1
eth2
Compute Node
eth3
vlan 511
vlan 512
eth0
eth3
vlan 511
vlan 512
7.10.180.104
2001:7:10:180::104
7.10.180.103
2001:7:10:180::103
Tenant Data Networks
(Tenant 1: VLAN 511)
(Tenant 2: VLAN 512)
External
Network
Tenant 2 External Network
172.26.185.0/24
2001:172:26:185::/64
Tenant 1 External Network
172.26.184.0/24
2001:172:26:184::/64
Data
Network
Router
© 2013 nephos6 and/or its affiliates. All rights reserved.
11

12. 1.
All
OpenStack
infrastructure
n o d e s
s h o u l d
b e
a b l e
to
communicate
with
each
other
by
IPv6
-­‐
IT
IS
ALL
ABOUT
CONFIGURATION
© 2013 nephos6 and/or its affiliates. All rights reserved.
12

13. Enable IPv6 On Infrastructure
Nodes
Field
Value
Keystone
/etc/keystone/keystone.conf
bind_host
2001:7:10:180::101
MySQL DB
/etc/mysql/my.cnf
bind-address
::
Apache
/etc/apache2/ports.conf
Listen
80
my_ip
2001:7:10:180::102
use_ipv6
true
osapi_compute_listen
2001:7:10:180::102
metadata_listen
Common
Components Configuration Files
7.10.180.102
novncproxy_host
2001:7:10:180::102
bind_host
2001:7:10:180::102
registry_host
net-glance.sandbox.com
bind_host
2001:7:10:180::102
Nova
/etc/nova/nova.conf
Controller
/etc/glance/glance-api.conf
Glance
/etc/glance/glanceregistry.conf
© 2013 nephos6 and/or its affiliates. All rights reserved.
13

14. Enable IPv6 On Infrastructure
Nodes
/etc/cinder/cinder.conf
Value
2001:7:10:180::102
glance_host
2001:7:10:180::102
osapi_volume_listen
Cinder
Field
my_ip
Controller
Components Configuration Files
2001:7:10:180::102
Neutron
2001:7:10:180::102
Neutron
/etc/neutron/neutron.conf
bind_host
2001:7:10:180::103
2001:7:10:180::102
use_ipv6
Compute
bind_host
my_ip
Network
/etc/neutron/neutron.conf
true
osapi_compute_listen
2001:7:10:180::102
metadata_listen
7.10.180.102
novncproxy_host
2001:7:10:180::102
bind_host
2001:7:10:180::103
Nova
Neutron
/etc/nova/nova.conf
/etc/neutron/neutron.conf
© 2013 nephos6 and/or its affiliates. All rights reserved.
14

15. 2.
OpenStack
should
be
able
to
spin
up
dual-­‐stack
VMs
in
multi-­‐
tenant
environment
-­‐
IT
IS
ALL
ABOUT
IPV6
ADDRESS
ASSIGNMENT
© 2013 nephos6 and/or its affiliates. All rights reserved.
15

16. Neutron Tenant Network Provisioning
neutron router-create --tenant-id tenant2-id router2
!
neutron net-create --tenant-id tenant2-id net2_192_168_2 -provider:network_type vlan --provider:physical_network
physnet3 --provider:segmentation_id 512
!
IPv6
tenant
subnet
Specify
IP
version
6
neutron subnet-create --tenant-id tenant2-id --ip-version 4 -name sub2_192_168_2 net2_192_168_2 192.168.2.0/24
neutron subnet-create —tenant-id tenant2-id --ip-version 6 -name sub2_2001_192_168_2 net2_192_168_2
2001:192:168:2::/64
Port
is
associated
!
with
tenant
subnet
neutron router-interface-add router2 sub2_192_168_2
neutron router-interface-add router2 sub2_2001_192_168_2
© 2013 nephos6 and/or its affiliates. All rights reserved.
16

17. Neutron Tenant Network
dnsmasq
binding
interface
(ipv4)
2.
OpenStack
needs
to
know
this
self-­‐calculated
IPv6
SLAAC
address…
qdhcp
namespace
ns-­‐74f270ff-­‐01
(192.168.2.2)
3.
Need
dnsmasq
to
send
RA
from
default
gateway
interface
1.
Need
ip6tables
filter
rules
to
enable
ICMPv6
at
inbound
direction
VM
192.168.2.3
(ipv6
address)
tap-­‐intf
tap74f270ff-­‐01
RA
qrouter
namespace
br-­‐eth2
eth2
qr-­‐6dbfb73d-­‐89
(2001:192:168:2::1)
Default
Gateway
Interface
(ipv4)
To
External
Network
Default
Gateway
Interface
(ipv6)
br-­‐eth3
Compute
Node
qr-­‐2f573f07-­‐d9
(192.168.2.1)
Network
Node
br-­‐int
eth3
br-­‐int
br-­‐eth3
eth3
Tenant
2
Network
© 2013 nephos6 and/or its affiliates. All rights reserved.
17

18. Enable RA Within Router Namespace
§ Method “spawn_process” in neutron.agent.linux.dhcp.py on Network Node
Derive
router’s
namespace
and
gateway
interface
Enable
dnsmasq
with
RA
and
SLAAC
Specify
IPv6
DHCP
range.
Taken
from
CLI
Add
IP
version
check
Bind
to
IPv6
qr-­‐
interface
Launch
dnsmasq
in
router’s
namespace
© 2013 nephos6 and/or its affiliates. All rights reserved.
18

19. 3.
VMs
should
be
able
to
gain
connectivity
to
external
IPv6
network
beyond
OpenStack’s
control
-­‐
Support
dual-­‐stack
on
a
single
external
interface
-­‐
Utilize
existing
VLAN/Segmentation
ID
!
-­‐
Eliminate
NAT
and
GARP
for
IPv6
subnets
© 2013 nephos6 and/or its affiliates. All rights reserved.
19

20. Dual-Stack options
§ Option #1: Use next-hop RA and SLAAC to allow external GW
interface defined IPv6 address
§ Option #2: Statically assign IPv6 address to external GW interface
for the router
– neutron router-gateway-set router2 ext-net-185
© 2013 nephos6 and/or its affiliates. All rights reserved.
20

21. Neutron External Network
Need
ip6tables
filter
rules
to
enable
ICMPv6
at
inbound
direction
Namespace:
qdhcp-­‐bfc3d877-­‐
44b6-­‐4879-­‐a83e-­‐d37455e77f71
dnsmasq
binding
interface
(ipv4)
ns-­‐74f270ff-­‐01
(192.168.2.2)
dnsmasq
binding
interface
(ipv6)
VM
192.168.2.3
(2001:192:168:2::1)
br-­‐int
br-­‐int
qr-­‐2f573f07-­‐d9
(192.168.2.1)
qr-­‐6dbfb73d-­‐89
(2001:192:168:2::1)
qg-­‐3dac3be9-­‐1b
(172.26.185.70)
(SLAAC
or
statically
assigned)
br-­‐eth2
br-­‐eth3
Compute
Node
tap-­‐intf
Network
Node
tap74f270ff-­‐01
br-­‐eth3
Namespace:
qrouter-­‐94662c71-­‐
bf80-­‐4c2f-­‐9841-­‐09a2112e3f58
eth2
eth3
RA
To
External
Network
Disable
NAT
and
GARP
for
IPV6
eth3
Tenant
2
Network
© 2013 nephos6 and/or its affiliates. All rights reserved.
21

22. Dual-stack options
§ For Option #2, there exists a limitation on static IP address
assignment for dual-stack implementation.
§ The L3 (server and agent) only allows a single IP address per
network (VLAN) within the Linux namespace representing the
tenant's router.
§ This limitation precluded the possibility of a dual-stack
arrangement utilizing static assignments without code changes.
© 2013 nephos6 and/or its affiliates. All rights reserved.
22

23. Dual-stack solution
To
accomplish
a
static
dual-­‐stack
arrangement,
ip_version,
cidr,
ip_address
and
gateway_ip,
was
essential
for
L3
agent
to
build
dual-­‐
stack
interface
inside
router’s
namespace.
© 2013 nephos6 and/or its affiliates. All rights reserved.
23

24. Dual-stack configuration
§ For the tenant router, learn the default route from the upstream
router through RA. When adding an external gateway
– net.ipv6.conf.<gateway_interface>.accept_ra=2
– net.ipv6.conf.<gateway_interface>.forwarding=1
– net.ipv6.conf.<gateway_interface>.accept_ra_defrtr=1
§ Prevent learning a default route from RA from internal tenant
network
– net.ipv6.conf.<internal_interface>.accept_ra_defrtr=0
§ When the subnet assigned is an IPv6, don’t apply NAT
configuration or perform GARP.
© 2013 nephos6 and/or its affiliates. All rights reserved.
24

25. Summary
Findings
Fixes
RA is not sent to IPv6 enabled internal
tenant network by default
Enable RA on dnsmasq
DHCP process is bound to interface
other than default gateway of tenant
network
IPv6 address chosen by OpenStack is
not based on SLAAC standard
Launch dnsmasq process inside
router namespace
Neighbor Discovery packet is dropped
by ip6tables filter rules
Add ip6tables rules to allow ND
related ICMPv6 packets
NAT and GARP are turned on for IPv6
subnets. Not desirable!
Only perform NAT and GARP for
IPv4 subnets
Calculate VM’s IPv6 address based
on unique MAC address
Whitepaper:
http://www.nephos6.com/pdf/OpenStack-Havana-on-IPv6.pdf
© 2013 nephos6 and/or its affiliates. All rights reserved.
25

26. Proposed Blueprint
§ From openstack-dev mailer:
– Short term, my goal is to get provider networks up and running, where
instances can get RA's from an upstream router outside of OpenStack and
configure themselves.
– Medium term, we want to make dnsmasq configuration more flexible.
– More long term, I'd like to make it so that if there is an upstream router doing
RA's - Neutron should send a PD automatically on network creation, and
populate a subnet from the response given by the upstream router.
§ Service Provider focused; may not work entirely with L3 Agent
without revisions
§ Integrate this PoC work with Blueprint to address broader
OpenStack community and address L3 Agent
© 2013 nephos6 and/or its affiliates. All rights reserved.
26

27. Our Next Step
Tactical
Strategical
DHCPv6
IPv6 mindset
Migration Strategy
IPv6 understanding / education
SLAAC + DHCPv6
Participation in IPv6 + Cloud
efforts
Support for dual-stack infrastructure
Icehouse release validation
© 2013 nephos6 and/or its affiliates. All rights reserved.
27

28. © 2013 nephos6 and/or its affiliates. All rights reserved.