Deploying flash storage for Ceph without compromising performance

Ceph Day LA – July 16, 2015
Deploying Flash Storage For Ceph

© 2015 Mellanox Technologies 2- Mellanox Confidential -
Leading Supplier of End-to-End Interconnect Solutions
Virtual Protocol Interconnect
Storage
Front / Back-EndServer / Compute Switch / Gateway
56G IB & FCoIB 56G InfiniBand
10/40/56GbE & FCoE 10/40/56GbE
Virtual Protocol Interconnect
Host/Fabric SoftwareICs Switches/GatewaysAdapter Cards Cables/Modules
Comprehensive End-to-End InfiniBand and Ethernet Portfolio
Metro / WAN

Scale-Out Architecture Requires A Fast Network
§ Scale-out grows capacity and performance in parallel
§ Requires fast network for replication, sharing, and metadata (file)
•  Throughput requires bandwidth
•  IOPS requires low latency
§ Proven in HPC, storage appliances, cloud, and now… Ceph
Interconnect Capabilities Determine Scale Out Performance

Solid State Storage Technology Evolution – Lower Latency
Advanced Networking and Protocol Offloads Required to Match Storage Media Performance
0.1
10
1000
HD SSD NVM
Access
Time
(micro-‐Sec)
Storage
Media
Technology
50%
100%
Networked
Storage
Storage Protocol
(SW) Network
Storage Media
Network
HW & SW
Hard
Drives
NAND
Flash
Next Gen
NVM

Ceph and Networks
§ High performance networks enable maximum cluster availability
•  Clients, OSD, Monitors and Metadata servers communicate over multiple network layers
•  Real-time requirements for heartbeat, replication, recovery and re-balancing
§ Cluster (“backend”) network performance dictates cluster’s performance and scalability
•  “Network load between Ceph OSD Daemons easily dwarfs the network load between Ceph Clients
and the Ceph Storage Cluster” (Ceph Documentation)

Ceph Deployment Using 10GbE and 40GbE
§ Cluster (Private) Network @ 40/56GbE
•  Smooth HA, unblocked heartbeats, efficient data balancing
§ Throughput Clients @ 40/56GbE
•  Guaranties line rate for high ingress/egress clients
§ IOPs Clients @ 10GbE or 40/56GbE
•  100K+ IOPs/Client @4K blocks
2.5x Higher Throughput , 15% Higher IOPs with 40Gb Ethernet vs. 10GbE!
(http://www.mellanox.com/related-docs/whitepapers/WP_Deploying_Ceph_over_High_Performance_Networks.pdf)
Throughput Testing results based on fio benchmark, 8MB block, 20GB file,128 parallel jobs, RBD Kernel Driver with Linux Kernel 3.13.3 RHEL 6.3, Ceph 0.72.2
IOPs Testing results based on fio benchmark, 4KB block, 20GB file,128 parallel jobs, RBD Kernel Driver with Linux Kernel 3.13.3 RHEL 6.3, Ceph 0.72.2
Cluster
Network
Admin
Node
40GbE
Public
Network
10GbE/40GBE
Ceph
Nodes
(Monitors,
OSDs,
MDS)
Client
Nodes
10GbE/40GbE

Ceph Is Accelerated by A Faster Network – Optimized at 56GbE
4,300 4,350
5,475 5,495
-
1,000
2,000
3,000
4,000
5,000
6,000
Ceph fio_rbd 64K random read Ceph fio_rbd 256K random read
40 Gb/s (MB/s) 56 Gb/s (MB/s)
27% More Throughput On Random Reads

Ceph Reference Architectures Using Disk

Optimizing Ceph For Throughput and Price/Throughput
§ Red Hat, Supermicro, Seagate, Mellanox, Intel
§ Extensive Performance Testing: Disk, Flash, Network, CPU, OS, Ceph
§ Reference Architecture Published Soon
10GbE Network Setup 40GbE Network Setup

Testing 12 to 72 Disks Per Node, 2x10GbE vs. 1x40GbE
§ Key Test Results
•  More disks = more MB/s per server, less/OSD
•  More flash is faster (usually)
•  All-flash 2 SSDs as fast as many disks
§ 40GbE Advantages
•  Up to 2x read throughput per server
•  Up to 50% decrease in latency
•  Easier than bonding multiple 10GbE links

Cisco UCS Reference Architecture for Ceph
§ Cisco Test Setup
•  UCS C3160 servers, Nexus 9396PX switch
•  28 or 56 6TB SAS disks; Replication or EC
•  4x10GbE per server, bonded
§ Results
•  One node read: 3700 MB/s rep, 1100 MB/s EC
•  One node write: 860 MB/s rep, 1050 MB/s EC
•  3 nodes read: 9,700 MB/s rep, 7,700 MB/s EC
•  8 nodes read: 20,000 MB/s rep, 10,000 MB/s EC

Optimizing Ceph for Flash

Ceph Flash Optimization
Highlights Compared to Stock Ceph
•  Read performance up to 8x better
•  Write performance up to 2x better with tuning
Optimizations
•  All-flash storage for OSDs
•  Enhanced parallelism and lock optimization
•  Optimization for reads from flash
•  Improvements to Ceph messenger
Test Configuration
•  InfiniFlash Storage with IFOS 1.0 EAP3
•  Up to 4 RBDs
•  2 Ceph OSD nodes, connected to InfiniFlash
•  40GbE NICs from Mellanox
SanDisk InfiniFlash

SanDisk InfiniFlash, Maximizing Ceph Random Read IOPS
Random Read IOPs Random Read Latency (ms)
0
20000
40000
60000
80000
100000
120000
140000
160000
180000
200000
25% Read 50% Read 75% Read 100% Read
8KB Random Read, QD=16
Stock Ceph IF-500
0
2
4
6
8
10
12
14
25% Read 50% Read 75% Read 100% Read
8KB Random Read, QD=16
Stock Ceph IF-500

SanDisk Ceph Optimizations for Flash
Setup SanDisk
InfiniFlash
Scalable
Informatics
Supermicro Mellanox
OSD Servers Dell R720 SI Unison Supermicro Supermicro
OSD Nodes 2 2 3 2
Flash 1 InfiniFlash
64x8TB = 512TB
24 SATA SSDs
per node
2x PCIe SSDs
per node
12x SAS SSDs
per node
Cluster Network 40GbE 100GbE 40GbE 56GbE
Total Read
Throughput
71.6 Gb/s 70 Gb/s 43 Gb/s 44 Gb/s

XioMessenger
Adding RDMA To Ceph

RDMA Enables Efficient Data Movement
§ Hardware Network Acceleration à Higher bandwidth, Lower latency
§ Highest CPU efficiency à more CPU Power To Run Applications
Efficient Data Movement
With RDMA
Higher Bandwidth
Lower Latency
More CPU Power For
Applications

RDMA Enables Efficient Data Movement At 100Gb/s
§ Without RDMA
•  5.7 GB/s throughput
•  20-26% CPU utilization
•  4 cores 100% consumed by moving data
§ With Hardware RDMA
•  11.1 GB/s throughput at half the latency
•  13-14% CPU utilization
•  More CPU power for applications, better ROI
x x x x
100GbE With CPU Onload 100 GbE With Network Offload
CPU Onload Penalties
•  Half the Throughput
•  Twice the Latency
•  Higher CPU Consumption
2X Better Bandwidth
Half the Latency
33% Lower CPU
See the demo: https://www.youtube.com/watch?v=u8ZYhUjSUoI

Adding RDMA to Ceph
§ RDMA Beta Included in Hammer
•  Mellanox, Red Hat, CohortFS, and Community collaboration
•  Full RDMA expected in Infernalis
§ Refactoring of Ceph Messaging Layer
•  New RDMA messenger layer called XioMessenger
•  New class hierarchy allowing multiple transports (simple one is TCP)
•  Async design that leverages Accelio
•  Reduced locks; Reduced number of threads
§ XioMessenger built on top of Accelio (RDMA abstraction layer)
•  Integrated into all CEPH user space components: daemons and clients
•  Both “public network” and “cloud network”

§ Open source!
•  https://github.com/accelio/accelio/ && www.accelio.org
§ Faster RDMA integration to application
§ Asynchronous
§ Maximize msg and CPU parallelism
•  Enable >10GB/s from single node
•  Enable <10usec latency under load
§ In Giant and Hammer
•  http://wiki.ceph.com/Planning/Blueprints/Giant/Accelio_RDMA_Messenger
Accelio, High-Performance Reliable Messaging and RPC Library

Ceph 4KB Read IOPS: 40Gb TCP vs. 40Gb RDMA
0
50
100
150
200
250
300
350
400
450
2 OSDs, 4 clients 4 OSDs, 4 clients
ThousandsofIOPS
40Gb TCP
40Gb RDMA34coresinOSD
24coresinclient
38coresinOSD
24coresinclient
38coresinOSD
30coresinclient
38coresinOSD
24coresinclient
RDMATCP
RDMATCP

Ceph-Powered Solutions
Deployment Examples

Ceph For Large Scale Storage– Fujitsu Eternus CD10000
§ Hyperscale Storage
•  4 to 224 nodes
•  Up to 56 PB raw capacity
§ Runs Ceph with Enhancements
•  3 different storage nodes
•  Object, block, and file storage
§ Mellanox InfiniBand Cluster Network
•  40Gb InfiniBand cluster network
•  10Gb Ethernet front end network

Media & Entertainment Storage – StorageFoundry Nautilus
§ Turnkey Object Storage
•  Built on Ceph
•  Pre-configured for rapid deployment
•  Mellanox 10/40GbE networking
§ High-Capacity Configuration
•  6-8TB Helium-filled drives
•  Up to 2PB in 18U
§ High-Performance Configuration
•  Single client read 2.2 GB/s
•  SSD caching + Hard Drives
•  Supports Ethernet, IB, FC, FCoE front-end ports
§ More information: www.storagefoundry.net

SanDisk InfiniFlash
§ Flash Storage System
•  Announced March 2015
•  512 TB (raw) in one 3U enclosure
•  Tested with 40GbE networking
§ High Throughput
•  8 SAS ports, up to 7GB/s
•  Connect to 2 or 4 OSD nodes
•  Up to 1M IOPS with two nodes
§ More information:
•  http://bigdataflash.sandisk.com/infiniflash

More Ceph Solutions
§ Cloud – OnyxCCS ElectraStack
•  Turnkey IaaS
•  Multi-tenant computing system
•  5x faster Node/Data restoration
•  https://www.onyxccs.com/products/8-series
§ Flextronics CloudLabs
•  OpenStack on CloudX design
•  2SSD + 20HDD per node
•  Mix of 1Gb/40GbE network
•  http://www.flextronics.com/
§ ISS Storage Supercore
•  Healthcare solution
•  82,000 IOPS on 512B reads
•  74,000 IOPS on 4KB reads
•  1.1GB/s on 256KB reads
•  http://www.iss-integration.com/supercore.html
§ Scalable Informatics Unison
•  High availability cluster
•  60 HDD in 4U
•  Tier 1 performance at archive cost
•  https://scalableinformatics.com/unison.html

Even More Ceph Solutions
§ Keeper Technology – keeperSAFE
•  Ceph appliance
•  For US Government
•  File Gateway for NFS, SMB, & StorNext
•  Mellanox Switches
§ Monash University -- Melbourne, Australia
•  3 Ceph Clusters, >6PB total storage
•  8, 17 (27), and 37 nodes
•  OpenStack Cinder and S3/Swift Object Storage
•  Mellanox networking, 10GbE nodes, 56GbE ISLs

Summary
§ Ceph scalability and performance benefit from high performance networks
•  Especially with lots of disk
§ Ceph being optimized for flash storage
§ End-to-end 40/56 Gb/s transport accelerates Ceph today
•  100Gb/s testing has begun!
•  Available in various Ceph solutions and appliances
§ RDMA is next to optimize flash performance—beta in Hammer

SanDisk IF-500 topology on a single 512 TB IF-100
Flash Memory Summit 2015
Santa Clara, CA 30
-‐

IF-‐100
BW
is
~8.5GB/s
(with
6Gb
SAS,
12
Gb
is
coming
EOY)
and
~1.5M
4K
IOPS

-‐  We
saw
that
Ceph
is
very
resource
hungry,
so,
need
at
least
2
physical
nodes
on
top
of
IF-‐100

-‐  We
need
to
connect
all
8
ports
of
an
HBA
to
saturate
IF-‐100
for
bigger
block
size

SanDisk Ceph-InfiniFlash Setup Details
Flash Memory Summit 2015
Santa Clara, CA 31
Performance
Config

-‐
IF-‐500

2
Node
Cluster
(
32
drives
shared
to
each
OSD
node)

Node

2
Servers

(Dell
R720)
2x
E5-‐2680
12C
2.8GHz

4x
16GB
RDIMM,
dual
rank
x4
(64GB)

1x
Mellanox
X3
Dual
40GbE

1x
LSI
9207
HBA
card

RBD
Client

4
Servers

(Dell
R620)

1
x
E5-‐2680
10C
2.8GHz

2
x
16GB
RDIMM,
dual
rank
x4
(32
GB)

1x
Mellanox
X3

Dual
40GbE

Storage
–
IF-‐100
with
64
Icechips
in
A2
Config

IF-‐100
IF-‐100
is
connected
64
x
1YX2
Icechips
in
A2
topology.
Total
storage
-‐
64
*
8
tb
=
512tb

Network
Details

40G
Switch
NA

OS
Details

OS

Ubuntu
14.04
LTS
64bit
3.13.0-‐32

LSI
card/
driver
SAS2308(9207)
mpt2sas

Mellanox
40gbps
nw
card
MT27500
[ConnectX-‐3]
mlx4_en

-‐

2.2-‐1
(Feb
2014)

Cluster
Configura[on

CEPH
Version
sndk-‐ifos-‐1.0.0.04
0.86.rc.eap2

Replicadon
(Default)

2

[Host]

Note:
-‐
Host
level
replicadon.

Number
of
Pools,
PGs
&
RBDs

pool
=
4

;PG
=

2048
per
pool

2
RBDs
from
each
pool

RBD
size
2TB

Number
of
Monitors
1

Number
of
OSD
Nodes
2

Number
of
OSDs
per
Node
32
total
OSDs

=
32
*
2
=

64

SanDisk: 8K Random - 2 RBD/Client with File System
IOPS: 2 LUNs /Client (Total 4 Clients)
0

50000

100000

150000

200000

250000

300000

1
2
4
8
16
32
1
2
4
8
16
32
1
2
4
8
16
32
1
2
4
8
16
32
1
2
4
8
16
32

0
25
50
75
100

Stock
Ceph
IFOS
1.0

Lat(ms): 2 LUNs/Client (Total 4 Clients)
[Queue Depth]
Read Percent
0

20

40

60

80

100

120

1
2
4
8
16
32
1
2
4
8
16
32
1
2
4
8
16
32
1
2
4
8
16
32
1
2
4
8
16
32

0
25
50
75
100

IOPS Latency
(ms)

SanDisk: 64K Random -2 RBD/Client with File System
IOPS: 2 LUNs/Client (Total 4 Clients) Lat(ms): 2 LUNs/Client (Total 4 Clients)
[Queue Depth]
Read Percent
0

20000

40000

60000

80000

100000

120000

140000

160000

1
2
4
8
16
32
1
2
4
8
16
32
1
2
4
8
16
32
1
2
4
8
16
32
1
2
4
8
16
32

0
25
50
75
100

Stock
Ceph

IFOS
1.0

0

20

40

60

80

100

120

140

160

180

1
2
4
8
16
32
1
2
4
8
16
32
1
2
4
8
16
32
1
2
4
8
16
32
1
2
4
8
16
32

0
25
50
75
100

IOPS Latency
(ms)

Deploying flash storage for Ceph without compromising performance

Empfohlen

Empfohlen

Weitere ähnliche Inhalte

Was ist angesagt?

Was ist angesagt? (20)

Andere mochten auch

Andere mochten auch (9)

Ähnlich wie Deploying flash storage for Ceph without compromising performance

Ähnlich wie Deploying flash storage for Ceph without compromising performance (20)

Kürzlich hochgeladen

Kürzlich hochgeladen (20)

Deploying flash storage for Ceph without compromising performance