A First Look At VXLAN over Infiniband Network On Linux 3.7-rc7 & iproute2 29 Nov, 2012 SAKURA Internet Research Center. Senior Researcher / Naoto MATSUMOTO

1. 29 Nov, 2012
SAKURA Internet Research Center
Senior Researcher / Naoto MATSUMOTO

2. VXLAN TESTING Overview
1) Linux bridge-based VXLAN is developing by Stephen Hemminger & et al.
http://linux-network-plumber.blogspot.jp/2012/09/just-published-linux-kernel.html
+config VXLAN
+ tristate "Virtual eXtensible Local Area Network (VXLAN)"
+ depends on EXPERIMENTAL
+ ---help---
+ This allows one to create vxlan virtual interfaces that provide
+ Layer 2 Networks over Layer 3 Networks. VXLAN is often used
+ to tunnel virtual network infrastructure in virtualized environments.
+ For more information see:
+ http://tools.ietf.org/html/draft-mahalingam-dutt-dcops-vxlan-02
2) Infiniband Adapter is the fastest of all for datacenter networking.
Dual-port Infiniband FDR 56Gbit/s
Host Bus Adapter
VXLAN + IP over Infiniband Network is AWESOME!.

3. Buildup to the New Kernel & etc
1) Install OFED for Infiniband Networking
# wget https://www.openfabrics.org/downloads/OFED/ofed-1.5.4/OFED-1.5.4.1.tgz
# tar xzvf ./OFED-1.5.4.1.tgz
# ./OFED-1.5.4.1/install.pl /*Install OFED software */
# sync; sync; sync; reboot;
2) Linux Kernel Update
# cd /usr/local/src
# wget http://www.kernel.org/pub/linux/kernel/v3.0/testing/linux-3.7-rc7.tar.bz2
# tar xvfj linux-3.7-rc7.tar.bz2
# cd linux-3.7-rc7
# cp /boot/config-2.6.32-220.23.1.el6.x86_64 ./.config
# make oldconfig
# vi .config
CONFIG_VXLAN=m /* Add New Line */
# make; make modules_install ; make install
# sync; sync; sync; reboot
3) iproute2 Update
# yum install db4-devel.x86_64
# git clone git://git.kernel.org/pub/scm/linux/kernel/git/shemminger/iproute2.git
# cd iproute2; ./configure ; make ; make install
Source: SAKURA Internet Research Center. 11/2012: Project THORN

4. VXLAN Configuration
# uname -a
Linux VXLAN-A 3.7.0-rc7 #1 SMP .... 56Gbit/s IPoIB
# ip addr add 10.1.1.1/24 dev ib1 IP over Infiniband Network
# ip route add 224.0.0.0/4 dev ib1
# ip link set up dev ib1
# ip link add vxlan99 type vxlan id 5001 group 239.0.0.99 ttl 10 dev ib1 VXLAN Network
# ip addr add 192.168.99.1/24 dev vxlan99
# ip link set up dev vxlan99
# iperf -u -s -B 239.0.0.99 & /* IGMPv3 Join TIPS */
# uname -a
Linux VXLAN-B 3.7.0-rc7 #1 SMP ....
# ip addr add 10.1.1.2/24 dev ib1
# ip route add 224.0.0.0/4 dev ib1
# ip link set up dev ib1
# ip link add vxlan99 type vxlan id 5001 group 239.0.0.99 ttl 10 dev ib1
# ip addr add 192.168.99.2/24 dev vxlan99
# ip link set up dev vxlan99
# iperf -u -s -B 239.0.0.99 & /* IGMPv3 Join TIPS */
Source: SAKURA Internet Research Center. 11/2012: Project THORN

5. VXLAN performance check
VXLAN-A# ip link show dev vxlan99
5: vxlan99: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 65470 qdisc
56Gbit/s IPoIB
link/ether b2:cc:a5:b8:74:95 brd ff:ff:ff:ff:ff:ff IP over Infiniband Network
VXLAN-A# iperf -c 192.168.99.2 -d
: VXLAN Network
[ ID] Interval Transfer Bandwidth
[ 3] 0.0-10.0 sec 29.7 GBytes 25.5 Gbits/sec
VXLAN-B# ip link show dev vxlan99
5: vxlan99: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 65470 qdisc
link/ether 4a:a6:cb:85:fb:2b brd ff:ff:ff:ff:ff:ff
VXLAN-B# iperf -s & vnstat -l -i ib1
Monitoring ib1... (press CTRL-C to stop)
rx: 18.49 Gbit/s 53431 p/s tx: 22.82 Gbit/s 58258 p/s
Source: SAKURA Internet Research Center. 11/2012: Project THORN

6. Traffic Analysis using Wireshark
Outer Header
Inner Header
Source: SAKURA Internet Research Center. 11/2012: Project THORN

7. Debug TIPS for tshark
1) VXLAN Traffic Monitoring
# tshark -pni ib1
Capturing on ib1
0.000000 10.1.1.1 -> 239.0.0.99 UDP Source port: 38367 Destination port: 8472
0.001072 10.1.1.2 -> 10.1.1.1 UDP Source port: 39994 Destination port: 8472
2) IGMPv3 Join/Releave Monitoring
# tshark -i ib1 -VV
Frame 1 (56 bytes on wire, 56 bytes captured)
Internet Protocol, Src: 10.1.1.1 (10.1.1.1), Dst: 224.0.0.22 (224.0.0.22)
Internet Group Management Protocol
[IGMP Version: 3]
Type: Membership Report (0x22)
Header checksum: 0xea9a [correct]
Num Group Records: 1
Group Record : 239.0.0.99 Change To Exclude Mode
Record Type: Change To Exclude Mode (4)
Aux Data Len: 0
Num Src: 0
Multicast Address: 239.0.0.99 (239.0.0.99)
Source: SAKURA Internet Research Center. 11/2012: Project THORN

8. Debug TIPS for iproute2
1) VXLAN Link status
# ip -d link show dev vxlan99
5: vxlan99: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 65470 qdisc noqueue state UNKNOWN
link/ether 4a:a6:cb:85:fb:2b brd ff:ff:ff:ff:ff:ff
vxlan id 5001 group 239.0.0.99 dev ib1 port 32768 61000 ageing 300
2) VXLAN IP Address information
# ip -d addr show dev vxlan99
5: vxlan99: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 65470 qdisc noqueue state UNKNOWN
link/ether 4a:a6:cb:85:fb:2b brd ff:ff:ff:ff:ff:ff
inet 192.168.99.1/24 scope global vxlan99
inet6 fe80::48a6:cbff:fe85:fb2b/64 scope link
valid_lft forever preferred_lft forever
3) VXLAN Network Table Lookup
# netstat -nr
Kernel IP routing table
Destination Gateway Genmask Flags MSS Window irtt Iface
10.1.1.0 0.0.0.0 255.255.255.0 U 00 0 ib1
192.168.99.0 0.0.0.0 255.255.255.0 U 00 0 vxlan99
224.0.0.0 0.0.0.0 240.0.0.0 U 00 0 ib1
Source: SAKURA Internet Research Center. 11/2012: Project THORN

9. Thank you for your interest.