Ahmad Ali, Supermicro

1. © Supermicro 2015
Supermicro Ceph Solutions
Open Solutions Defined by Workload
Ahmad Ali
Director, Storage Solutions
AhmadAli@supermicro.com
Chicago, 8/18/2015

2. Objectives
 Generate well-balanced
solutions without
compromising on availability,
performance, capacity and cost
 Derive configurations from
empirical data during the
testing.
 Provide reference architectures
to prime opportunities and
accelerate the sales cycle

3. Proof of Concept - Objective
Usable Capacity XS - 100TB S - 500TB M- 1PB L - 2PB
IOPS-Optimized
3x replicated data,
tiered over EC X X X X
Throughput
Optimized*
3x replicated data X X X X
Capacity Optimized*
6+2 EC protected data
X X X X
ScaleWorkload
The Ceph Performance Bingo Card

4. 36-Bay Server Flexibility
Framework to support multiple
configurations in the same box
 Dual Socket Motherboard
 Enabling Single and Dual CPU
Configurations
 Dual Backplanes
 x8 lane SAS Connectivity
 Dual IT mode controllers
 2x Available PCI slots for NVMe
 Improving drive to SSD ratio
 2x Dual port 10G cards
 Separate Cluster and Client Networks

5. POC Hardware Description

6. POC Hardware Description

7. Write Journals for OSDs
can be stored on SSD
Media for optimal OSD
write performance.
The Ratio of SSD to HDD
allows Ceph to be tuned for
use in a broad spectrum of
applications
X9 Ceph Drive Configuration
5x OSDs 5x OSDs
SSD
SSD
OSD Node
5:1 Ratio
X9 generation

8. Write Journals for OSDs
can be stored on SSD
Media for optimal OSD
write performance.
The Ratio of SSD to HDD
allows Ceph to be tuned for
use in a broad spectrum of
applications
For 12-36 Bay OSD Nodes
the use of PCI-E flash /
NVMe instead of SAS
based SSD increases
available I/O bandwidth to
the media, enabling greater
HDD:SSD ratios to be used
X10 Ceph Drive Configuration
12 Bay Node (12+1)
12:1 Ratio
NVMe
NVMe
18:1 Ratio
NVMe
36 Bay Node (36+2)
X10 generation

9. 0
500
1000
1500
2000
2500
4 64 4096
MB/sec
Object Size (KB)
12+1
10G+10G
18+1
10G+10G
18+0
10G+10G
36+2
10G+10G
36+0
10G+10G
36+2
40G (shared)
60+12
40G (shared)
72+0
40G (shared)
Sequential Read Throughput per Server
3x replication, librados

10. Sequential Write Throughput per Server
3x replication, librados
0
100
200
300
400
500
600
700
4 64 4096
MB/sec
Object Size (KB)
12+1
10G+10G
18+1
10G+10G
18+0
10G+10G
36+2
10G+10G
36+0
10G+10G
36+2
40G (shared)
60+12
40G (shared)
72+0
40G (shared)

11. Sequential Read Throughput per Server
3+2 Erasure Coding, librados
0
300
600
900
1200
1500
1800
4 64 4096
MB/sec
Object Size (KB)
12+1
10G+10G
12+0
10G+10G
18+1
10G+10G
18+0
10G+10G
36+2
10G+10G
36+0
10G+10G
36+2
40G (shared)
60+12
40G (shared)
72+0
40G (shared)

12. Sequential Write Throughput per Server
3+2 Erasure Coding, librados
0
200
400
600
800
1000
1200
4 64 4096
MB/sec
Object Size (KB)
12+1
10G+10G
12+0
10G+10G
18+1
10G+10G
18+0
10G+10G
36+2
10G+10G
36+0
10G+10G
36+2
40G (shared)
60+12
40G (shared)
72+0
40G (shared)

13. Red Hat Ceph Reference Architecture
http://www.redhat.com/en/resources/red-hat-ceph-storage-clusters-supermicro-storage-servers

14. Proof of Concept - Objective
Usable Capacity XS - 100TB S - 500TB M- 1PB L - 2PB
IOPS-Optimized
3x replicated data,
tiered over EC X X X X
Throughput
Optimized*
3x replicated data X X X X
Capacity Optimized*
6+2 EC protected data
X X X X
ScaleWorkload
The Ceph Performance Bingo Card

15. Ceph Workload Positioning
Usable Capacity XS - 100TB S - 500TB M- 1PB L - 2PB
IOPS-Optimized
3x replicated data,
tiered over EC
Planned for Winter 2015
Throughput Optimized*
3x replicated data 6 node
• Read: 5,300 MB/s
• Write: 1,400 MB/s
32 node
• Read: 28,000 MB/s
• Write: 9,500 MB/s
63 node
• Read: 55,000 MB/s
• Write: 19,000 MB/s
125 node
• Read: 110,000 MB/s
• Write: 37,000 MB/s
Capacity Optimized*
6+2 EC protected data 10 node
• Read: 8,000 MB/s
• Write: 2,000 MB/s
8 node
• Read: 7,000 MB/s
• Write: 3,400 MB/s
13 node
• Read: 11,000 MB/s
• Write: 5,000 MB/s
* Projected performance based on node performance in 10 node cluster
(12 x 4TB + 1NVMe) (12 x 4TB + 1NVMe) (12 x 4TB + 1NVMe) (12 x 4TB + 1NVMe)
(12 x 6TB) (36 x 6TB) (72 x 6TB)

16. Price/Performance Comparisons

17. © Supermicro 2015
X10 SuperStorage
X10 Optimized
Configurations for Ceph

18. X10 and X9 Platforms for Ceph
X10 Models CPU/Mem
Drive
Config X9 Models CPU/Mem
Drive
Config
SSG-6018R-MON2
Dual Intel Xeon
E5-2630 v3
64GB
Ix 800GB PCI-flash /
NVMe
SSG-6017R-MON1
Dual Intel Xeon
E5-2630 v2
64GB
NA
SSG-F618H-OSD288P
Single Intel Xeon
E5-2620 v3
64GB
(12+1)
12x 6TB HDD + 1x
NVMe
NA NA NA
SSG-6028R-OSD072
Single Intel Xeon
E5-2620 v3
64GB
(12+0)
12x 6TB HDD
NA NA NA
SSG-6028R-OSD072P
Single Intel Xeon
E5-2620 v3
64GB
(12+1)
12x 6TB HDD + 1x
NVMe
SSG-6027R-OSD040H
Single Intel Xeon
E5-2630 v2
64GB
(10+2)
10x 4TB HDD +
2x SSD
SSG-6048R-OSD216
Dual Intel Xeon
E5-2630 v3
128GB
(36+0)
36x 6TB HDD
NA NA NA
SSG-6048R-OSD216P
Dual Intel Xeon
E5-2630 v3
128GB
(36+2)
36x 6TB HDD + 2x
NVMe
SSG-6047R-OSD120H
Dual Intel Xeon
E5-2630 v2
128GB
(30+6)
30x 4TB HDD +
6x SSD
SSG-6048R-OSD432
Dual Intel Xeon
E5-2690 v3
256GB
(72+0)
72x 6TB HDD
NA NA NA
SSG-6048R-OSD360P
Dual Intel Xeon
E5-2690 v3
256GB
(60+12)
60x 6TB HDD + 12x
SSD
SSG-6047R-OSD320H
Dual Intel Xeon
E5-2670 v2
(E5-2697
recommended)
128GB
(60+12)
60x 4TB HDD +
12x SSD

19. Conclusion
 Single CPU 12+1 config seemed to be optimal for
throughput-optimized clusters.
 SSD write journaling helps greatly. PCIe flash allows
greater HDD ratios for both throughput (replication)
and capacity (EC) optimized configs.
 A lot more work is still needed
 Identify IO optimized reference architecture
 Study all flash and tiered performance
 Understand OSD performance and implications of
non-uniform memory access (NUMA) systems
 Proc-core to OSD ratios with/without hyper-threading

20. © Supermicro 2015
Thank You
AhmadAli@supermicro.com

21. 189 187
261
250
262
228
0
50
100
150
200
250
300
GB per Watt
X10 Ceph Node Power Consumption*
(12+0)
SSG-6028R-
OSD072
(12+1)
SSG-6028R-
OSD072P
(36+0)
SSG-6048R-
OSD216
(36+2)
SSG-6048R-
OSD216P
(72+0)
SSG-6048R-
0SD432
(60+12)
SSG-6048R-
0SD360P
AC input /
Watts@240V= 374 379 812 850 1621 1555
Total
BTU/hour= 1279 1294 2773 2903 5533 5309
VA= 390 395 846 886 1688 1620
72-Bay36-Bay12-Bay
* Paper
estimate based
on maximum
loading of all
components,
real world is
typically lower

22. Rack Power / Density Guidance
42U rack-level sizing projected from OSD node paper power budgets
Supermicro Models
5KVA 8.5 KVA 10KVA 16KVA
Nodes
Rack
Units HDDs Nodes
Rack
Units HDDs Nodes
Rack
Units HDDs Nodes
Rack
Units HDDs
SSG-F618H-OSD288P 12 13 144 20 22 240 24 26 288 36 40 432
SSG-6028R-OSD072
SSG-6028R-OSD072P 13 26 156 20 40 240
SSG-6048R-OSD216
SSG-6048R-OSD216P 5 20 180 9 36 324
SSG-6048R-OSD432 3 12 216 5 20 360 6 24 432 9 36 648
SSG-6048R-OSD360P 3 12 180 5 20 300 6 24 360 9 36 540
1U/12-bay nodes
2U/12-bay nodes
4U/36-bay nodes
4U/72-bay nodes
SRS-42E112-CEPH-02
SRS-42E112-CEPH-03
SRS-42E136-CEPH-02
SRS-42E136-CEPH-03
SRS-42E172-CEPH-02
SRS-42E172-CEPH-03
SRS-14E412-CEPH-01
SRS-42E412-CEPH-01TBD
TBD
TBD
TBD