Dr. Wolfgang Schulze, Director Global Storage Consulting Practice, Red Hat

1. WHEN THE CEPH HITS THE FAN
Dr. Wolfgang Schulze
Director Global Storage Consulting
Practice
Red Hat
October 20, 2016

2. CAN THE CEPH EVEN HIT THE FAN?
2
• A"er all…
• Architecture has no single point of failure
• Code base is very solid and had many years to mature
• Designed from the ground up to accommodate for failures
• Supposed to be self-healing and self-managing
• It simplifies day-to-day data center opera?ons

3. WHAT IS “HITTING THE FAN”, ANYWAYS?
3
• Example scenarios:
• Heavy storm takes out data center, cluster fails to restart automa?cally
• Increased work load makes cluster unstable
• Performance is fine when cluster is empty to moderately filled, but when
when geHng close physical capacity, write performance drops
• Nearly full cluster has become unresponsive and denies writes
• Bulk dele?on of objects takes so long that the client applica?on ?mes out
• Rebalancing a"er a par?al electric outage impacts clients with slow/
blocked requests
• Result in each case: customer files
• Sev 1: Produc?on is down
• Sev 2: Produc?on is impacted

4. TICKET QUEUE IN RED HAT SUPPORT
4
Real screenshot, dated 2016-10-19
Customer names removed
Many of these ,ckets could have
been avoided if best prac,ces had
been followed

5. A SAD, BUT TRUE STORY
5
• Customer bought Red Hat Ceph Storage subscrip?ons
• They were sure they had enough experience on their team and specifically
declined offers for training and consul?ng
• They designed and deployed Ceph cluster without guidance
• Originally for feasibility study, but everything seemed to work fine,
so they put it into produc?on
• Nobody no?ced that the journal size was configured to only 100MB instead
of best prac?ce size of 5GB
• A couple of months later a"er a power failure, the Ceph cluster failed to recover
• Support ?cket went on for several weeks, at the end some permanent data loss
• End result: Par?al data loss, unhappy management, unhappy customers

6. SOME COMMON MISCONCEPTIONS
6
• The new tools make Ceph easy to set up
• You don’t need detailed planning or architecture design
• Ceph works on any hardware, and you can mix & match hardware
• Storage infrastructure people will know how to handle the product
• Server people will know how to handle the product
• Ceph community bits are just fine (“We use a stable release”)
• Using community bits is more “cuHng edge”

7. COMMON TROUBLE #1
UPSTREAM BITS FOR PRODUCTION SYSTEMS
7
Observa,on
• User is running upstream bits
• This happens even with users who are paying for a Red Hat Support subscrip?on
• People misinterpret the phrase “stable release” in community release notes
Problem
• Red Hat Support won’t be able to help
• Red Hat only supports long term stable releases
• What could be a safe and fully documented upgrade to a newer LTS version
suddenly becomes a “migra?on” with risks and piealls
Mi,ga,on
• Use supported bits, stay informed about roadmap, get involved

8. COMMON TROUBLE #2
USE OF UNSUPPORTED FEATURES
8
Observa,on
• User deploys system into produc?on using features which are not (yet) supported
• Examples: CephFS, BlueStore
Problem
• Red Hat Support won’t be able to help
• Unless you have a support excep?on, the conversa?on may end quickly
• Red Hat Engineering will not build hot fixes for you
Mi,ga,on
• Try to get a support excep?on from Red Hat
• Don’t use the feature

9. COMMON TROUBLE #3
USE OF UNSUPPORTED CONFIGURATIONS
9
Observa,on
• User deploy Ceph in a way that is not approved and has not been tested
• Examples:
• Running Ceph on unsupported Opera?ng System versions (e.g. GenToo, Debian)
• Deploying
Problem
• Red Hat Support won’t be able to help
• Unless you have a support excep?on, the conversa?on may end quickly
• Red Hat Engineering will not build hot fixes for you
Mi,ga,on
• Read documenta?on, consider health check before go-live

10. COMMON TROUBLE #4
POORLY MANAGED CLUSTER GROWTH
10
Observa,on
• Adding disks (or even en?re nodes) to clusters of rela?vely small total capacity
• Backfill/recovery starves client I/O
Problem
• In older versions of Ceph, default configura?on values are not ideal for this
(osd_max_backfills, osd_recovery_max_ac?ve, osd_recovery_op_priority)
• If you fail to adjust these before you change the physical configura?on, you will
indeed have huge impact
Mi,ga,on
• Know your stuff, think ahead, es?mate impact, gradually weigh in

11. COMMON TROUBLE #5
POOR SKILLS AND OPERATIONAL PRACTICES
11
Observa,ons
• Subject majer experts who brought Ceph to the organiza?on were hired guns,
or employees who have since le"
• Team that ends up managing cluster considers it some sort of black art
Problem
• Operators who don’t know what they are doing put your data at risk
• The built-in safety/durability may be compromised
Mi,ga,on
• Make sure users receive proper training, and avoid staff SPOF
• Conduct controlled emergency drills to prac?ce for outages
• Maintain separate cluster with same version for experiments and dry run,
or learn how to do it with a cloud based environment

12. COMMON TROUBLE #6
RISKY CONFIGURATION CHOICES
12
Observa,ons
• Users read somewhere that moun?ng XFS OSD’s with the ‘nobarrier’ op?on
will result in performance gains
Problem
• While the performance gets no?ceably bejer, you are introducing a risk for
data corrup?on during power outages
• The built-in safety/durability may be compromised
Mi,ga,on
• Do not use ‘nobarrier’ mount op?on unless you understand fully what
hardware you have, and unless you know what you are doing

13. COMMON TROUBLE #7
POOR NETWORK CONFIGURATION
13
Observa,ons
• Users don’t pay enough ajen?on to network configura?on
• Network inconsistencies (e.g. Jumbo Frames) and bojlenecks go undetected
…un?l Ceph performs poorly.
Problem
• Troubleshoo?ng networking issues is difficult and experts hard to find
• Ceph heavily relies on proper configura?on
Mi,ga,on
• Invest in your team and network maintenance skills

14. WHAT TO DO WHEN THINGS WENT WRONG
14
1. Stay calm and don’t make it worse!
• Poorly skilled operators may turn a problem into a catastrophe
2. Contact Red Hat Support immediately
• Sev 1 and Sev 2 issues are handled with top priority
• Chances are that they will be able to help right away and get your cluster
humming again
3. Contact your trusted Red Hat Services or Sales contacts
• If problems persist or you feel you need extra help, you might want to get a
Ceph expert from Red Hat Professional Services

15. GOOD PRACTICES TO AVOID PROBLEMS
15
1. Don’t stumble into implementa?on/deployment without careful planning
• Capture and document requirements, do a POC, do an actual design
• Engage experts early to help with cluster design and hardware choices
2. Unless you love to take risks, use supported bits
3. Stay close to the recommended reference architectures from Red Hat partners
4. Make sure your staff receives proper training
• Red Hat Global Learning provides excellent training for Gluster and Ceph
5. Plan for growth
6. Don’t let things linger. Ceph does not like it when the cluster is 90% full
7. Have an expert perform regular Storage Health Checks to detect problems while
they are s?ll small

16. STORAGE DESIGN CONSULTING
16
• Specialists from Red Hat Consul?ng will help planning your Ceph
deployment
• Start: Storage Discovery Session
• We can help discover requirements and design a storage solu?on
that matches
• You will receive a detailed Storage Solu,on architecture document
which will ar?culate design choices and lay out a step-by-step plan
for implementa?on

17. STORAGE HEALTH CHECKS
17
• Standard 3-day engagement done by Red Hat storage experts
• Comprehensive top-to-bojom analysis of your so"ware-defined storage plaeorm
• Six focus areas
1. Life cycle
2. Configura?on
3. Organiza?on
4. Use Case
5. Hardware
6. Opera?onal
• Clear read-out of issues
• Ac?onable recommenda?ons

18. POSITIVE NOTE
18
• I asked my consultants for feedback on this presenta?on.
Here is one comment

19. 19
WHERE TO GO NEXT
RED HAT
SUBSCRIPTIONS
hjps://access.redhat.com/subscrip?on-value
Evalua?on, Pre-produc?on, and Produc?on subscrip?ons available
CONSULTING hjp://www.redhat.com/en/services/consul?ng/storage
TRAINING hjps://www.redhat.com/en/services/training
TEST DRIVE hjp://red.ht/cephtestdrive
To engage a Territory Service Manager in your area, ask for a local Red Hat Storage sales professional at:
NORTH AMERICA: 1 (888) REDHAT-1; LATIN AMERICA: 54 (11) 4329-7300; EMEA: 00800 7334 2835
APJ: 65 6490 4200; Brazil: 55 (11) 3529-6000,; Australia: 1800 733 428; New Zealand: 0800 733 428