1. Kyle Bader
kyle.bader@dreamhost.com
@mmgaggle
DreamHost.com
A Storage Story

2. About Me
Dad, husband, technologist.
Sr. Systems Engineer
@DreamHost
free software linux internals storage
networking security monitoring
distributed systems automation
DreamHost

3. Outline
DreamHost Storage History
Anatomy of Ceph
Automating Storage
DreamHost

4. Destro
- DreamHost's first web server
- Pentium 100
- SCSI storage
- Shared T1 line
DreamHost

5. DH2DreamHost

6. NetApp
NetApp Fabric Attached Storage
- 15k Fiber channel drives
- Filer heads serve NFS
- Fast failover
- Large failure domains
- Expensive
- Low density
DreamHost

7. Coraid
Coraid SAN
- Shelves carry SATA devices, provide AoE volumes
- Head units mount AoE volumes, XFS, NFS shares
- Linux!
- Fast failover
- Large failure domains (single L2 segment)
DreamHost

8. Thumper
Sun Sunfire X4500
- 45 Drives in a 4U chassis
- Legendary hardware
- Fast failover
- High density
- Large failure domains
- SATA
- Heavy
DreamHost

9. DreamHost Solution: Hybrid

10. BlueArc "Titanic"
- Switched fiber channel
- Head units serve NFS
- Tiered storage, FC/SATA
- Fast failover
- Larger failure domain (than NetApp)
- Software bugs :(
- Tiering: find -atime
DreamHost BlueArc

11. DreamHost UNLIMITED

12. DreamHost Thoughting..

13. Mixed Strategy
- Separate email and web storage
- Email IO is heavy random, lots of small files
- Web storage needs to be dense
- FC NAS for email
- SATA RAID for web storage
- SATA ZRAID for backups
DreamHost

14. Local RAID
Local RAID
- RAID6, RAID10, RAID6
- SATA, SAS disks
- ext3, XFS
- Shrink failure domain
- Great density
- Slower failover
- RAID Controllers..
DreamHost

15. SighDreamHost

16. Ceph
Ceph
- Open source
- Build with COTS hardware
- Distributed and replicated
- No single point of failure
- Consist
- Self healing and self managing
DreamHost

17. Building Blocks
Monitors:
- Maintain cluster map
- Provide consensus for distributed decision making
- Must have an odd number
- These do not serve stored objects to clients
OSDs:
- One per disk (recommended)
- Serve stored objects to clients
- Intelligently peer to perform replication tasks
- Supports object classes
DreamHost

18. Building Blocks
OSD States
Up available and ready
Down not available
In current member of cluster
Out not member of cluster
DreamHost

19. CephstoreDreamHost
XFS
BTRFS
EXT4

20. ClusterDreamHost

21. Ceph ConsumersDreamHost

22. Creating a MapDreamHost

23. CRUSHDreamHost
CRUSH
- Pseudo-random placement algorithm
- Ensures statistically even distribution
- Repeatable, deterministic
- Rule based configuration
- Replica count
- Infrastructure Topology
- Weighting

24. CRUSHDreamHost

25. OSD DOWN!DreamHost

26. Remap and BackfillDreamHost

27. Ceph AnatomyDreamHost

28. RESTful Storage ServiceDreamHost

29. DreamObjects
DreamObjects:
- Ceph Storage Cluster
- Ubuntu Linux (12.04)
- Managed by Opscode Chef
- S3 and Swift RESTful interfaces
- Highly durable (8 nines)
- 2+ PB raw capacity
DreamHost

30. DreamCompute
DreamCompute:
- Ceph Storage Cluster (RDB)
- Ubuntu Linux (12.04)
- Managed by Opscode Chef
- OpenStack
- Virtualized L2 and L3 networking
- Highly durable (8 nines)
- 3+ PB raw capacity
DreamHost

31. AutomateDreamHost
- Bootstrap cluster
- Packages and configuration
- Creates, Destroys and Encrypts OSDS
- Roles map to pdsh genders
- User and SSH key management
- Push monitoring configurations

32. Hard StuffDreamHost
Key management
Leader election

33. What we useDreamHost
- Attributes
- Environments
- Search
- No databags

34. EnvironmentsDreamHost
- Ceph package versions
- VIPs for API endpoints
- Package repository URI
- Ceph configuration data driven by attributes

35. Gated EnvironmentsDreamHost
Development
Staging
Production

36. Operational Feedback
- Continuous functional testing
- Metrics, metrics, metrics
- Dashboards
DreamHost

37. Chef Infrastructure
- Chef cluster per datacenter
- Private Chef from Opscode
- Erchef is awesome
- Migrating legacy automation
DreamHost

38. Chef for Ceph
- Prototyped Ceph cluster automation with Chef
- Adapted Ceph to ease configuration
- Pushed some automation down into Ceph
- Move towards being CM agnostic
- Simplify Chef recipes
DreamHost

39. Chef the network
- DreamCompute utilizes ODM switches
- Cumulus Networks provides Linux based OS
- Custom Chef omnibus builds for PPC
- Ohai networking!
DreamHost

40. Resiliency EngineeringDreamHost
- Amazon (GameDay)
- Etsy (GameDay)
- Google (DiRT)

41. Infra. a CodeDreamHost
- Bare metal servers configured by code
- Network devices configured by code
- Block storage configured by code
- Virtual networking configured by code

42. ThanksDreamHost
DreamHost
Sage Weil and Inktank
OpenStack Developers
Opscode

43. DreamObjectsDreamHost
Questions?
kyle.bader@dreamhost.com
@mmgaggle