Empfohlen
Empfohlen
Weitere ähnliche Inhalte
Was ist angesagt?
Was ist angesagt? (19)
Andere mochten auch
Andere mochten auch (8)
Ähnlich wie Preparing for CDN failure: Why and how
Ähnlich wie Preparing for CDN failure: Why and how (20)
Kürzlich hochgeladen
Kürzlich hochgeladen (20)
Preparing for CDN failure: Why and how
- 1. Preparing for CDN Failure: Why and How Aaron Peters TurboBytes @aaronpeters & @ksgyoung Kyle Young Mobify
- 2. Examples of CDN Failure https://www.flickr.com/photos/phobia/2308371224/
- 3. Multi-continent outage for 17 minutes @aaronpeters & @ksgyoung 100% broken in North-America and Europe.
- 4. Up and down for days, some ASNs only @aaronpeters & @ksgyoung
- 5. Zoomed in on May 4 and May 5 … @aaronpeters & @ksgyoung OMG!
- 6. Japan, all networks: 5 hours of bad Three days later, it happened again … @aaronpeters & @ksgyoung
- 7. #1 eyeballs network in Germany Single country, single ASN perf degradations are not uncommon in CDN land … @aaronpeters & @ksgyoung
- 8. “System Maintenance” @aaronpeters & @ksgyoung
- 9. Take-Aways @aaronpeters & @ksgyoung Expect your CDN to fail on you They won’t tell you about it
- 11. Synthetic monitoring @aaronpeters & @ksgyoung Pingdom, cron+curl, ThousandEyes, Interns, etc.
- 12. Synthetic monitoring @aaronpeters & @ksgyoung The Pros Very Specific Known Intervals Controlled Locations Lots of Information
- 13. Synthetic monitoring @aaronpeters & @ksgyoung The Cons Limited Locations Datacenters Hyper-idealized Client Model
- 14. Synthetic monitoring @aaronpeters & @ksgyoung Still better than nothing.
- 15. RUM for Page Load Time @aaronpeters & @ksgyoung CDN must serve HTML + resources Third party content adds noise Can’t capture CDN failing Not very useful
- 16. RUM with Test Object @aaronpeters & @ksgyoung Fetch small object from CDN(s) after onload Beacon timings, or a Fail Use Nav Timing for best insight Challenge get your JS on other sites too, to capture CDN failing
- 17. RUM for Page Resources @aaronpeters & @ksgyoung Resource Timing API Send assets with TAO header Onload => beacon timings Easy, right? Not so fast …
- 18. Starting Points @aaronpeters & @ksgyoung Did the asset come from CDN? Did it load fast enough? Was it a good response (200/304)? Fetched from network? if (total time > 20 ms) { // from network } DOES NOT WORK !
- 19. RT API has many quirks @aaronpeters & @ksgyoung DNS time and Connect time always zero in IE No data for 4xx/5xx responses, except in IE Nothing in RT API until asset fully loaded, except in IE FF doesn’t tell the truth
- 20. How measure CDN perf with RT API @aaronpeters & @ksgyoung main.css inline JS, high in HEAD, exec only if window.chrome setTimeOut(checkRTAPI, 5000); at onload or when timer ended: if ( main.css in RT API && connectTime > 3 ) { loaded fine from network } else { meh } Too slow: Fail
- 21. transferSize attribute, FTW! @aaronpeters & @ksgyoung transferSize = byte size that came over the wire if ( transferSize != 0 ) { // from network } Status: no browser is implementing this yet Background/discussion: https://github.com/w3c/navigation-timing/issues/3 Future
- 22. Take-Aways @aaronpeters & @ksgyoung Measure Fail Ratio too, not just Speed Use RUM for real-world performance insight And Synthetic monitoring for deep visibility Beware of the many bugs in Res Timing API
- 24. Selecting your CDNs Feature & Behaviour Parity Performance Costs @aaronpeters & @ksgyoung
- 25. Doing Multi-CDN @aaronpeters & @ksgyoung Perf data Decision making Targeting Time-to-Switch High volume, high quality Make good sense of the data, quickly Very granular, very accurate Asap!
- 26. Where switch CDNs? @aaronpeters & @ksgyoung cs109.wac.edgecastcdn.net. cds.z4b9c4e6.hwcdn.net //cdn1.mydomain.com/main.css //cdn2.mydomain.com/main.css in DNS in HTML OR
- 27. Traffic management in DNS @aaronpeters & @ksgyoung resolver authoritativeclient I see the request comes from NL, based on resolver IP address … … so I’ll handout the CNAME to the CDN configured for NL CDN BCDN A
- 28. Static Geo @aaronpeters & @ksgyoung Always route to that CDN in that geo Easy: no need to monitor perf But what if CDN has boo boo?
- 29. Dynamic Geo @aaronpeters & @ksgyoung Always route to best CDN per geo Needs solid perf data (RUM !) Geo targeting accuracy important
- 30. Dynamic Geo + ASN @aaronpeters & @ksgyoung Holy grail: gives best results Really needs RUM data, and lots of it Targeting accuracy even more important
- 31. Geo targeting gone wrong @aaronpeters & @ksgyoung 8.8.8.8 authoritativeclient I see the request comes from MY, based on resolver IP address … … so I’ll handout the CNAME to the CDN configured for MY CDN B - best in India !CDN A
- 32. EDNS0 to the rescue ! @aaronpeters & @ksgyoung 8.8.8.8 authoritativeclient I see the request comes from IN, based on client IP address /24 … … so I’ll handout the CNAME to the CDN configured for IN CDN B - best in India !CDN A
- 33. Decision Making @aaronpeters & @ksgyoung Look at everything, not just ‘Response Time’ Use multiple statistics, not just median Make your ‘decider’ sensitive to Fail Ratio ! Tuning your logic takes time
- 34. Coping with low volume data @aaronpeters & @ksgyoung Don’t make changes Make decisions with lower confidence Have a dynamic targeting granularity
- 35. Experiment: do a Pat Meenan ! @aaronpeters & @ksgyoung http://www.slideshare.net/patrickmeenan/service-workers-for-performance Hi, I’m Pat
- 36. Example CDN Perf Program http://bit.ly/1KppCfo @aaronpeters & @ksgyoung
- 37. Example CDN Perf Program @aaronpeters & @ksgyoung
- 38. Example CDN Perf Program Limitations Not Practical for Monitoring Humans are Required Misses Important Factors (EG SSL) Hard to Commit to Bandwidth
- 39. War Story
- 40. Seattle Down @aaronpeters & @ksgyoung or How we Started Caring About MultiCDN
- 42. @aaronpeters & @ksgyoung Yay, Monitoring!
- 43. @aaronpeters & @ksgyoung Support Lines - A Tale of Lowered Expectations IT Crowd - Fremantle Media Batman - 20th Century Fox
- 44. @aaronpeters & @ksgyoung SLA Reminder 99.9% 8 hours 99.99% 52 minutes 99.999% 5 minutes
- 45. @aaronpeters & @ksgyoung Best Effort Switch over to an alternate CDN for the entire service, across the globe, as per the “Holy S#!t Handbook”.
- 46. @aaronpeters & @ksgyoung This Isn’t Pretty Cold Cache Backend Thrashing 3 to 4 hours of Intermittent Failure
- 47. @aaronpeters & @ksgyoung Lessons Learned Hot Standby Geographic DNS Control, and Optimization More Monitoring
- 48. @aaronpeters & @ksgyoung Thank you. @aaronpeters - aaron@turbobytes.com @ksgyoung - kyoung@mobify.com Questions?