Routed Fabrics For Ceph

http://intrbiz.comchris@intrbiz.com
Routed Fabrics For Ceph
Chris Ellis - @intrbiz
Fast & Effective Networking For Ceph
Ceph Day London 2019

Hello!
● I’m Chris
○ IT jack of all trades
● Mostly a PostgreSQL Consultant
○ Full stack:
■ from electronic design to web dev
● Very much into Open Source
○ Started a monitoring system project a few years ago
○ Big openSUSE and PostgreSQL fan
● Been using and playing with Ceph for a couple of years
○ Build a small VM farm with Ceph for shared storage

Routed Fabrics, Huh?

Routed Fabrics, Huh?
● Essentially we make servers participate in routing
○ Every network link the server has is active / active utilised
○ Every server takes part in the routing protocol
○ Routing protocol deals with device and link failures
■ Data just takes another path in the event of a fault
● Equal Cost Multi Path (ECMP) is used to efficiently move traffic
○ IP packets are routed over all available links
○ TCP streams don’t get split across more than one path
■ Single stream is still limited to the bandwidth of your links
○ IE: with 4x 10Gbe NICs we can push 40Gb/s of traffic in aggregate
■ An individual TCP stream maxes at 10Gb/s

The Build
● My setup is about as small as you can go
● I've my R&D setup
● It's only two switches
● But it's about showing that these approaches work even at small scale
○ All traffic is still routed
○ We still get all benefits of a Routed Fabric
○ We can use cheap commodity switching
○ You don't need super high end kit to get efficiency and speed
● Yes, it's not a real Clos topology, you need a bigger problem domain for that
● This is about thinking about different ways of doing things

What You’ll Need

Connecting Things

A Cunning Plan - Network Assignments
● Switch 1: 172.31.1.0/24
○ Port 1: 172.31.1.0/30
○ Port 2: 172.31.1.4/30
○ …
○ Port 24: 172.31.1.92/30
● Inter-switch: 172.31.3.0/24
○ Link 1: 172.31.3.0/30
○ Link 2: 172.31.3.4/30
○ …
○ Link 8: 172.31.3.28/30
● Switch 2: 172.31.2.0/24
○ Port 1: 172.31.2.0/30
○ Port 2: 172.31.2.4/30
○ …
○ Port 24: 172.31.2.92/30
● Ceph: 172.28.0.0/24
○ Node 1: 172.28.0.1/32
○ Node 2: 172.28.0.2/32
○ …
○ Node 12: 172.28.0.12/32

Configuring Your Switches - Turn On Routing
ip routing
router ospf
router-id 172.26.1.210
network 172.31.1.0 255.255.255.0 area 0.0.0.0
network 172.31.3.0 255.255.255.0 area 0.0.0.0
redistribute connected
redistribute static
exit

Configuring Your Switches - Server Interface
interface 0/1
mtu 9018
routing
ip address 172.31.1.1 255.255.255.252
ip ospf area 0.0.0.0
exit

interface 0/2
mtu 9018
routing
ip address 172.31.1.5 255.255.255.252
exit

interface 0/24
mtu 9018
routing
ip address 172.31.1.93 255.255.255.252
exit

Configuring Your Switches - Inter Switch Interface
interface 0/28
mtu 9018
routing
ip address 172.31.3.1 255.255.255.252
exit

Configuring Your Ceph Server - Interfaces
$> cat ifcfg-eth4
BOOTPROTO='static'
IPADDR='172.31.1.2/30'
MTU='9000'
$> cat ifcfg-eth6
BOOTPROTO='static'
IPADDR='172.31.1.6/30'
MTU='9000'
$> cat ifcfg-eth5
BOOTPROTO='static'
IPADDR='172.31.2.2/30'
MTU='9000'
$> cat ifcfg-eth7
BOOTPROTO='static'
IPADDR='172.31.2.6/30'
MTU='9000'

Configuring Your Ceph Server - Dummy Interface
$> cat ifcfg-dummy0
BOOTPROTO='static'
IPADDR='172.28.0.1/32'
MTU='9000'

Configuring Your Ceph Server - Quagga & OSPFd
$> zypper in quagga ospfd

Configuring Your Ceph Server - Quagga
$> cat zebra.conf
hostname ceph1
!
interface eth4
ip address 172.31.1.2/30
interface eth5
ip address 172.31.2.2/30
interface eth6
ip address 172.31.1.6/30
interface eth7
ip address 172.31.2.6/30
!

Configuring Your Ceph Server - OSPFd
$> cat ospfd.conf
hostname ceph1
!
interface eth4
interface eth5
interface eth6
interface eth7
router ospf
ospf router-id 172.26.1.1
network 172.28.0.1/32 area 0
network 172.31.1.2/30 area 0
network 172.31.1.6/30 area 0
network 172.31.2.2/30 area 0
network 172.31.2.6/30 area 0
!

Configuring Your Ceph Server - Kernel
$> cat sysctl.conf
# Enable IP routing
net.ipv4.ip_forward = 1
# Tweak ECMP policy
net.ipv4.fib_multipath_hash_policy = 1
net.ipv4.fib_multipath_use_neigh = 1
# Disable reverse path filtering
net.ipv4.conf.all.rp_filter = 0
# Enable reverse path filtering on normal NICs
net.ipv4.conf.bond1.rp_filter = 1

Configuring Your Ceph Server - Ceph
$> cat ceph.conf
[global]
public_network = 172.28.0.0/24

Et Volia
$> ip route
172.26.28.2 proto zebra metric 20
nexthop via 172.31.1.10 dev eth7 weight 1
172.26.28.3 proto zebra metric 20
...

Caveats
● Make sure that MTUs are configured correctly and match
○ OSPF is a custom IP type, if your MTU is mismatched packets get corrupted
● Label your cables
○ Swapping cables around will break things
● Quagga will only set a default route if no default route is already defined
○ OSPFd needs: `default-information originate metric-type 1`

Further Reading
● Intro to Clos networks
○ https://en.wikipedia.org/wiki/Clos_network
● Google white paper on their CLOS topologies
○ https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/43837.pdf
● Cumulus on Clos and ECMP:
○ https://cumulusnetworks.com/blog/celebrating-ecmp-part-one/
● Benefits of ditching layer 2
○ https://thenewstack.io/ditch-pitfalls-layer-2-networks-modern-data-center-design/

Routed Fabrics For Ceph

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Routed Fabrics For Ceph

Similar to Routed Fabrics For Ceph (20)

More from ShapeBlue

More from ShapeBlue (20)

Recently uploaded

Recently uploaded (20)

Routed Fabrics For Ceph