Server-side Intelligent Switching using vyatta. (10 oct, 2012) for Japan Vyatta Users Meeting 2012 Autumn in Tokyo. SAKURA Internet Research Center. Senior Researcher / Naoto MATSUMOTO

1. 10 Oct, 2012
SAKURA Internet Research Center
Senior Researcher / Naoto MATSUMOTO

2. Inteligent Switching Overview
Process Process Process Process Process Process
Vyatta VM Vyatta VM
IaaS-B IaaS-C
Process Process Process
Tunnel Tunnel Tunnel
Vyatta VM
Bridge + IEEE 802.1w(Rapid STP)
IaaS-D
Virtual Ethernet NIC
Process Process Process
Vyatta VM
IaaS-A Source: SAKURA Internet Research Center. 08/2012: Project THORN

3. How does it works?
Tunnel Interface + Linux Bridge+IEEE 802.1w (Rapid STP) Daemon =
Server-side Intelligent Switching
vyattaVM# sudo rstpctl showport br0
Tunnel Tunnel Tunnel
* vtun1 8001 Fwd 8000-027a0b4f3269 8...3269 8001 D
Bridge + IEEE 802.1w(Rapid STP) * vtun2 8001 Fwd 8000-027a0b4f3269 8...3269 8002 D
:
Virtual Ethernet NIC
veth1 8004 Fwd 8000-027a0b4f3269 8...3269 8004 D
Process Process Process
Vyatta VM
Source: SAKURA Internet Research Center. 08/2012: Project THORN

4. Packages for Inteligent Switching
Tunnel Tunnel Tunnel - Tunneling Protocol:
OpenVPN,GRETAP,L2TPv3 or VXLAN ..etc
Bridge + IEEE 802.1w - Bridging Daemon: bridge-utils
(Rapid STP) - Rapid STP Daemon: rstpd
http://git.kernel.org/?p=linux/kernel/git/shemminger/rstp.git
Virtual Ethernet NIC - Virtual Interface: iproute2
http://kernel.org/pub/linux/utils/net/iproute2/
Process Process Process
VyattaCore 6.4
Source: SAKURA Internet Research Center. 08/2012: Project THORN

5. How to Build Dev-Env on Vyatta
configure
delete system package repository community
set system package repository oxnard components main
set system package repository oxnard distribution oxnard
set system package repository oxnard url http://packages.vyatta.com/vyatta-dev/oxnard/unstable
commit
save
reboot
sudo full-upgrade -k
configure
set system package repository squeeze components main
set system package repository squeeze distribution squeeze
set system package repository squeeze url http://cdn.debian.net/debian
commit
save
sudo apt-get update
sudo aptitude install module-assistant
sudo aptitude install linux-vyatta-kbuild
cd /lib/modules/3.0.23-1-586-vyatta-virt/
sudo ln -s /usr/src/linux-image/debian/build/build-i386-none-586-vyatta-virt/ build
sudo aptitude install rpm zlib1g-dev zlib1g-dbg byacc bison flex
sudo module-assistant prepare

6. Install rstpd and switch-setup
- Downloading latest master from git
http://git.kernel.org/?p=linux/kernel/git/shemminger/rstp.git
cd rstp
sudo make
sudo make install
- L2TPv3 pseudo-wire Config
sudo ip l2tp add tunnel tunnel_id 1 peer_tunnel_id 1 udp_sport 5001 udp_dport 5001 ¥
encap udp local X.X.X.X remote Y.Y.Y.Y
sudo ip l2tp add session tunnel_id 1 session_id 1 peer_session_id 1
:
sudo ifconfig l2tpeth0 0
sudo ifconfig l2tpeth0 up up
- Virtual Ehternet Config Tunnel Tunnel Tunnel
ip link add type veth
sudo ifconfig veth0 up up ; sudo ifconfig veth1 up up Bridge + IEEE 802.1w(Rapid STP)
- Bridge/Switch Config Virtual Ethernet NIC
sudo brctl addbr br0
sudo brctl addif br0 veth1 Process Process Process
sudo brctl addif br0 l2tpeth0
sudo rstpd Vyatta VM
sudo rstpctl rstp br0 on

7. Linux Performance Tuning
- Linux kernel parameter Tuning
su
sysctl -w net.core.netdev_max_backlog=500000
sysctl -w net.core.rmem_max=67108864
sysctl -w net.core.wmem_max=67108864
sysctl -w net.core.rmem_default=67108864
sysctl -w net.core.wmem_default=67108864
sysctl -w net.ipv4.tcp_mem="67108864 67108864 67108864″
echo 1000000000 > /proc/sys/kernel/shmmax
echo 400 > /proc/sys/vm/nr_hugepages
- Linux RPS(Recive Packet Steering) Tuning
echo 65535 >/proc/sys/net/core/rps_sock_flow_entries
echo 002 > /sys/class/net/eth0/queues/rx-0/rps_cpus
echo 200 > /sys/class/net/l2tpeth0/queues/rx-0/rps_cpus
:
etc
echo 65535 > /sys/class/net/eth0/queues/rx-0/rps_flow_cnt
echo 65535 > /sys/class/net/l2tpeth0/queues/rx-0/rps_flow_cnt
:
etc

8. Bridge/RSTP Aging Timer
- Linux Bridge STP Timer Config
brctl sethello br0 1
brctl setfd br0 4 Tunnel Tunnel Tunnel
brctl setmaxage br0 6
brctl setageing br0 10
Bridge + IEEE 802.1w
- Linux RSTP Timer Config
rstpctl sethello br0 1 (Rapid STP)
rstpctl setmaxage br0 6
rstpctl setfdelay br0 4
Virtual Ethernet NIC
Process Process Process
VyattaCore 6.4

9. Performance Benchmark HowTo
- Linux in-kernel packet generator tool
su
modprobe pktgen
echo "rem_device_all" > /proc/net/pktgen/kpktgend_0
echo "add_device eth0" > /proc/net/pktgen/kpktgend_0
echo "max_before_softirq 50000" > /proc/net/pktgen/kpktgend_0
echo "count 0" > /proc/net/pktgen/eth0
echo "clone_skb 1000000" > /proc/net/pktgen/eth0
echo "pkt_size 64" > /proc/net/pktgen/eth0
echo "delay 400" > /proc/net/pktgen/eth0
echo "dst X.X.X.X" > /proc/net/pktgen/eth0
echo "dst_mac aa:18:39:6e:fc:3c" > /proc/net/pktgen/eth0
echo "start" > /proc/net/pktgen/pgctrl
Tunnel Tunnel Tunnel
Bridge + IEEE 802.1w(Rapid STP)
Virtual Ethernet NIC
Process Process Process
Vyatta VM

10. Tunneling Overhead Analysis
High is Faster Low is Better
Mellanox ConnectX3 FDR10-HCA (IPoIB) 29.56 Gbit/s** 61Kpps (MTU 65,520)
Mellanox ConnectX3 40GbE-NIC (IPoEth) 27.45 Gbit/s* 2.28Mpps (MTU 1,500)
L2TPv3 on FDR10-HCA (IPoIB) 10.65 Gbit/s* 882Kpps (MTU 1,500)
Mellanox ConnectX3 10GbE-NIC (IPoEth) 9.93 Gbit/s* 820Kpps (MTU 1,500)
GRETAP on FDR10-HCA (IPoIB) 5.41 Gbit/s* N/A
VXLAN on UNKOWN-DEVICE (IPoEth) 4.79 Gbit/s** N/A
OpenVPN on FDR10-HCA (IPoIB) 0.24 Gbit/s* N/A
(Gbit/sec) (Packet/sec)
[System: Intel(R) Core(TM) i7-3930K CPU @ 3.20GHz / 32GB DDR3-DIMM]
[Traffic Generater: *pktgen, **iperf. Traffic Monitor: vnstat] Sourece: SAKURA Internet Research Center 07/2012

11. Thank you for your interest
(C)Copyright 1996-2010 SAKURA Internet Inc.