Diese Präsentation wurde erfolgreich gemeldet.

Weitere Verwandte Inhalte

Ähnliche Bücher

Kostenlos mit einer 14-tägigen Testversion von Scribd

Alle anzeigen

Ähnliche Hörbücher

Kostenlos mit einer 14-tägigen Testversion von Scribd

Alle anzeigen

Days In Green : Forecasting the Life of a Healthy Service @Twitter

  1. 1. Days In Green (DIG): Forecasting the life of a healthy service Vibhav Garg, Arun Kejariwal (@ativilambit, @arun_kejariwal) Capacity and Performance Engineering @ Twitter June 2014
  2. 2. #WorldCup [1] http://www.newsweek.com/twitter-us-soccer-what-radio-was-baseball-256336 (June 2014) [1]
  3. 3. [1] http://www.telegraph.co.uk/technology/twitter/10912738/Wimbledon-goes-digital-with-Twitter.html (June 2014) [1] #Wimbledon2014
  4. 4. Internet trends • Mobile-first  25% of total web usage [1]  Mobile data traffic: 81%, accelerating growth [1] • Real-time [1] http://www.kpcb.com/file/kpcb-internet-trends-2014 (May 2014) VG, AK 4 #Selfie
  5. 5. Capacity & Performance • Organic growth  Over 255M monthly active users [1] • Evolving product landscape • Handle Peak Traffic  Mobile Busy Hour Is 66% Higher Than Average Hour in 2013, 83% by 2018 [2]  Events [1] https://investor.twitterinc.com/releasedetail.cfm?releaseid=843245 [2] http://www.cisco.com/c/en/us/solutions/collateral/service-provider/visual-networking-index-vni/white_paper_c11-520862.html VG, AK 5
  6. 6. Systematic Capacity Planning • Objectives  Check under-allocation  Performance, Availability o Adversely impact user experience  Check over-allocation  Operational efficiency o Adversely impacts bottom line  Check poor scalability • Approaches  Reactive  Adversely impact user experience  Proactive PoorUX Underutilization VG, AK 6
  7. 7. Systematic Capacity Planning (contd.) • Non-trivial  Rapidly evolving product landscape  Changes services’ performance profile  Organic growth • Scalable Approach  Service Oriented Architecture  100s of services  Millions of metrics [1,2]  Automated [1] http://strata.oreilly.com/2013/09/how-twitter-monitors-millions-of-time-series.html [2] http://strataconf.com/strata2014/public/schedule/detail/32431 VG, AK 7
  8. 8. DIG: Days in Green • Objective  Statistically determine the # of days for which a service is expected to stay healthy • Methodology  Determine driving resource  Determine capacity threshold T  Generate a time series and forecast  DIG - # days before the service is expected to exceed T VG, AK 8 Time DrivingResource DIG T
  9. 9. DIG (contd.) • Determining Capacity Thresholds  Service specific  Driving resource differs  Load Test  Canaries  Replay production traffic  Examples  CPU at 70%  Disk utilization at, 80%  RPS at X requests/sec VG, AK 9 SLA T CPU Latency
  10. 10. DIG (contd.) • Time Series Analysis  Data collection  Granularity o Daily • Long term forecast o Which value? • Close to the daily peak but low standard deviation (σ) o Assume 7 day seasonality  Duration o 30-90 days  Model fitting  Forecast VG, AK 10 Percentile Duration Mean σ 100 (Max) 57.7 3.29 99 14.4 mins 54.7 2.49 95 72 mins 53.1 2.4
  11. 11. DIG (contd.) • Model fitting  Linear  Captures trend well  Does not fit well for seasonal time series  No weightage to recent data VG, AK 11 R2 = 0.56
  12. 12. DIG (contd.) • Model fitting  Polynomial  Fits better than linear, not good for forecasting  Seasonality unaware VG, AK 12 R2 = 0.62
  13. 13. DIG (contd.) • Model fitting  Splines  Widely used for curve fitting  Tend to overfit data  Not suitable for forecasting  Triple Exponential Smoothing (Holt Winters)  Good for fit and forecasting  Trend and seasonality modeled implicitly • ARIMA VG, AK 13
  14. 14. ARIMA • Auto-Regressive Integrated Moving Average  (p, d , q)  Explicitly models seasonality and trend  Applicable to non-stationary time series  Worst Case degenerates to linear fit Autoregressive component Moving Average component Moving Average order Integrated order Autoregressive order VG, AK 14
  15. 15. DIG (contd.) • Model Fitting  ARIMA in action  Captures underlying trend  Captures seasonality  Are we good? Not quite! VG, AK 15 Forecast
  16. 16. • Time Series Characteristics  Anomalies  Positive  Negative VG, AK 16 Anomalies DIG (contd.)
  17. 17. Breakout • Time series characteristics  Breakout  Flavors o Mean shift o Ramp up  Direction o Positive, Negative DIG (contd.) VG, AK 17
  18. 18. • Time series characteristics  Seasonality breaks  Various reasons (but not limited to)  Daily deployments  Changes in traffic  Collection issues Seasonality Breaks VG, AK 18 DIG (contd.)
  19. 19. VG, AK 19 • Curve fitting with ARIMA  Trend and seasonality aware  What does the DIG forecast look like? Trend 1 Trend 2 DIG (contd.) Anomaly T Breakout
  20. 20. DIG (contd.) • ARIMA Forecast  Not a good forecast because of multiple trends and anomalies  Wide confidence band  40 Days In Green with Confidence band of 10-40 VG, AK 20 95%confidenceband T DIG
  21. 21. • ARIMA Forecast with breakout(s) eliminated  35 Days In Green with a Confidence Band of 2-40  Limitations o Wide confidence band o Susceptible to anomalies VG, AK 21 DIG (contd.) T DIG
  22. 22. • ARIMA Forecast with Breakout and Anomaly eliminated  25 Days In Green with a Confidence Band of 2-40  Narrow confidence band  Improved Accuracy VG, AK 22 DIG (contd.) T DIG
  23. 23. • DIG Comparison  With breakout and anomaly detection DIG (contd.) VG, AK 23 DIG T Raw Raw - BO Raw – BO- Anomaly
  24. 24. DIG (contd.) VG, AK 24 • Discussion  Boundary conditions  False seasonality T
  25. 25. DIG (contd.) • Limitations  “Quality” of data: Poor forecasts VG, AK 25 T
  26. 26. • Limitations  Idiosyncratic patterns: Poor forecasts  Ongoing work! VG, AK 26 DIG (contd.) T
  27. 27. DIG (contd.) VG, AK 27 • Current Status – Deployed in Production  Hundreds of services  Fully automated for CPU, extending to other metrics  DR Compliance  Combine data from multiple datacenters  Detect services that are close to DR threshold • Future Work  Utilization Based Allocation
  28. 28. DIG (contd.) VG, AK 28 • Anomaly Detection  Algorithm developed in-house  Presented at USENIX HotCloud’14[1] [1] https://www.usenix.org/conference/hotcloud14/workshop-program/presentation/vallis
  29. 29. Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466%Mean Shift: 105.466% DIG (contd.) VG, AK 29 • Breakout Detection  Algorithm developed in-house  To be presented at Velocity New York’14 [1] [1] http://velocityconf.com/velocityny2014/public/schedule/detail/35485
  30. 30. Wrapping up & Lessons learned • DIG: Days In Green  Proactively assess future health of a service  Modeling and forecasting: ARIMA  Anomaly and Breakout removal • Modeling  Hard to get a stable time series  Organic growth, New products, Behavioral aspect  Exploring advanced data cleansing techniques  Improve Breakout and Anomaly Detection VG, AK 30
  31. 31. Acknowledgements • Piyush Kumar, Capacity Engineer • Winston Lee, Capacity Engineer • Owen Vallis Jr & Jordan Hochenbaum, Ex Interns • Nicholas James, Intern • Management team VG, AK 31
  32. 32. Join the Flock • We are hiring!!  https://twitter.com/JoinTheFlock  https://twitter.com/jobs  Contact us: @ativilambit, @arun_kejariwal Like problem solving? Like challenges? Be at cutting Edge Make an impact VG, AK 32

Notizen

  • Thank you Jon. Good morning everybody. Thank you for coming to our talk. I realize that that this is the last talk before lunch; so, we shall keep it short and leave 5 mins for Q&A. My colleague Vibhav and I will be presenting our methodology behind forecasting the life of a healthy service. If you have any questions, please feel free to ask at the end of presentation or reach out to us afterwards or tweet us at any time – our tweet handles are mentioned on the slide. Both Vibhav and I are part of the Capacity and Performance Engineering team at Twitter. The team is chartered to forecast capacity demand, to create performance and scalability models for our services, and to come up with capacity budget. In this regard, we work with a various teams such as supply, finance etc.

    I shall kick off the talk with motivation behind the metric Days in Green and then Vibhav shall walk you through the details of the methodology behind DIG. Finally, I shall wrap up with the lessons learned.
  • Last month Mary Meeker from KPCB posted her annual report on Internet trends. Two highlights of the report were that mobile now accounts for 25% of the total web usage. Also, mobile data traffic growth has been accelerating. There are several drivers such as video behind 81% growth. A key aspect of mobile internet usage is real-time content consumption. Be it a selfie as was the case during Oscars earlier this year, or be the elections in India, the largest democracy in the world or be it events such as the Asiana plane crash or the unfortunate mudslide in Afghanistan.
  • Capacity planning plays a key role at Twitter for a variety of reasons (which I believe also hold true in general). First of all, we need to carry out capacity planning to support organic growth. In a recent SEC filing, we reported that Twitter has over 255M monthly active users. Further, we need to plan to support growing user engagement. Also, in an agile product development such as at Twitter, new features and new products are A/B tested or rolled out at a rapid pace. To support this, a systematic capacity planning methodology is needed. Last but not the least, capacity planning is needed to support peak traffic. In a recent report, Cisco report that the mobile busy hour was 66% higher than the average hour and is expected to climb to 81% by 2018. It is well know that people flock to Twitter during events such as the Superbowl, Oscars, UEFA Champions League and the ongoing FIFA World Cup; these events induce significant traffic peaks.
  • A straightforward approach towards capacity planning would be to throw hardware at the problem. Operationally, this approach is clearly not desirable.

    Broadly speaking, the objectives of capacity planning are two fold: Check under allocation so as to check any impact on performance and availability of a given service. Both poor performance and availability in turn adversely impact end-user user experience. Second, check over-allocation as it directly relates to operational efficiency. The plot here exemplifies underutilization wherein the red line corresponds to the max utilization for a given SLA and the green line corresponds to actual utilization.

    There are two approaches to capacity planning. Under reactive approach additional capacity on a need basis. However, this may result in poor user experience which would in turn adversely impact the bottomline. Clearly, Reactive approach is undesirable. At Twitter, we employ a proactive approach to capacity planning. Further, we use statistical models to determine capacity ask on a per-service basis. Depending on the context, we business metrics such as TPS, Photos per sec or system metrics for forecasting.
  • Capacity planning is non-trivial due to a variety of other factors such as rapidly evolving product landscape. This in turn changes the performance profile of a given service. Also, organic growth exacts new capacity asks.

    Also, an approach should be scalable. At Twitter, we have a service oriented architecture with hundreds of services. We monitor millions of time series on a daily basis. Last but not the least, the methodology needs to be automated. Vibhav shall now walk you through the details of our approach to forecast the life of a healthy service.
  • One of the most fundamental questions that we deal with in capacity planning is to understand when a service is expected go over its capacity limit. Most approaches to this are reactive in nature, like when you run into performance problems because the service is not able to handle the traffic that is being thrown at it. So we have come up with the idea of DIG number associated with each service, which tells us the # of days left before the service goes out of capacity.

    Methodology

    Determine the Driving Resource – The driving resource is the metric that constrains the service, e.g., CPU, Disk etc…
    Determine Capacity Threshold T – Determine the maximum value of the core driver at which the service will remain healthy
    Generate a time series for the metric and forecast its value into the future.
    So here we setup the notion of the Green Zone and the Red Zone. The Green Zone is the region in the graph before the driving resource exceeds threshold T, and the Red Zone is the region after.
    We then calculate DIG, Days In Green, as the distance between the last point in the raw time series and the start of the RED zone, expressed as the number of day for the service in the green zone,
  • As already stated before, the first step in determining DIG is to determine the driving resource and its capacity threshold for the service.


    Load Test –

    In production using canaries by manipulating the amount of traffic served to it (more accurate)
    Or could also be a in a load test environment where we replay production traffic.

    I would like to point out all graphs that we show from this point onwards are generated from real production data.

    The graph shows what is commonly called a hockey stick curve. We plot the CPU on the x axis and Latency on the Y axis from a real production load test, and we observe that until CPU reaches threshold T, the relationship between CPU and Latency is linear, and after this point the latency goes into non-linear degradation, so we choose T as are CPU threshold

    So depending on the service characteristics we may choose different metric thresholds, e.g. CPU at 70%, Disk at 80%. If we want to use an application metric we can use RPS too, but keep in mind that the RPS metric threshold can change based on the underlying code.
  • Once we have the threshold for the driving resource, the next step in determining DIG is to generate and analyze the time series for that resource. In order to generate suitable time series, we need to answer two important questions.

    Granularity – Should we use daily, hourly, minutely data. Since we want to a long term forecast we choose daily data. Then we need to choose a daily value that is likely to give us a stable time series as well as be as accurate as possible. Because of the nature of twitter’s traffic daily peaks are not very stable and are prone to spikes. We therefore want to choose a value that is close to the daily peak. and has low standard deviation. If you look at this table, I have compared the different percentile values of a metric for a certain number of days. As you can see the mean of the daily maxes is 57.7 and the standard deviation is 3.29. The mean of the daily p99 is 54.7 and standard deviation is 2.49. The values for p95 are 53.1 and 2.4 respectively. There are 1440 mins in a day and if we are only removing 14.4 mins of the day from consideration, against 72 mins for p95. So in this case p99 is a better metric to use. We also assume 7 day seasonality since we have different patterns on different days of the week.

    Duration – How long should the time series be? It has been observed that to generate a statistically significant model, we need a minimum of 30 data points. Also, a time series of 90 days give us enough data points to discern a reasonable trend in the data. Due to the frequent changes in service profile anything more than 90 days would probably end up being stale data.

    Model Fitting - Once we have built the time series we try to fit a statistical model on it an then forecast it to generate the DIG number
  • The most simple form of model fitting is linear regression which I am sure everyone has used at some point in time. It is very good for capturing trend but does not fit well for a time series which shows seasonality. As you can see from the graph, the trend is decently captured but the R2 value , which tells us how well the model fits with dataset with values between 0 and 1, with 1 being the perfect fit, is low at 0.56, which doesn’t give us good confidence in our forecast. This model also doesn’t give any weightage to recent data, which would give us a better idea of recent trend changes.
  • The other model that is quite common is the polynomial model, which fits better than a linear model, with a R2 value of 0.62, but as you can see does a poor job of forecasting. It is also seasonality unaware like the linear model.
  • There are other techniques like Splines which are really good for fitting but not good for forecasting because they tend to overfit the data.

    Holt Winters is widely used to model time series with seasonality and trend but they are modeled implicitly and the parameters are hard to control and automate.

    And then there is ARIMA, which we use to model our time series to generate DIG
  • So ARIMA stands for Auto-Regressive Integrated Moving Average. The model is made of three parameters, p,d, and q where , d is Integrated Order, p is the AR order and q is the MA order. The equation shown here are for the AR and MA components of the model. The I component is evaluated by differencing the time series to de-trend it. I will not go into the details of the math as it is beyond the scope of this talk

    The most interesting properties of ARIMA are its ability to explicitly model the seasonality and trend based on the parameters we pass it. And as we said, the “d” parameter allows modeling of time series that are not stationary in nature by detrending the time series by differencing

    One of the interesting properties of ARIMA is that if is not able to detect seasonality it degenerate into a linear fit, which can still be useful.
  • So here we see the an ARIMA model that we fit onto a time series generated by a service. With a nice seasonality and a noticeable trend, ARIMA does a very good job of fitting the time series as shown by the blue dashed line, and is able to forecast the same pattern as shown by the green dashed lined at the end

    So its should be pretty easy to take an off the shelf ARIMA package and fit any time series right? Not quite
  • What we showed in the previous case was ideal case of time series. Most time series that we see are not stable enough to model easily.

    We would therefore now like to discuss some of the characteristics of the time series that we notice in our data.

    As can be seen in the graph, time series can have anomalies, which are data points with values that are much higher than values around them and show up as spikes on the graph. Anomalies can be both positive and negative, and can have a big impact on the model. Depending on where the anomalies are located, they can skew the model. A really spikey graph is likely to produce really bad forecasts.
  • Due to changes in the service profile, new features etc, we can also have breakouts, where there is a sudden jump or decrease in metric over a sustained period of time. We observe different types of breakout. For example, if there is a code deployment which makes the service is more expensive and increases the cpu utilization from 40-50% we observe a mean shift. If in another case there is a new feature which is seeing increased adoption but has not stabilized yet, we might see a ramp up

    Breakouts make it hard for ARIMA to detect trends and produce a stationary time series for forecasting.

    One of the interesting things that you notice in this graph is the change in the shape of the seasonality after the breakout. If we consider the time series after the breakout, it provides for a much better ARIMA model than if we choose the entire time series
  • For some time series we also see weak seasonality and also different seasonality patterns throughout the time series . In presence of weak seasonality , anomalies can have a strong effect on the model.

    As I said before the patterns that we see in the time series can be due to multiple reasons , not limited to the multiple deployments we do throughout the day, changes in traffic pattern, collection issues etc…
  • We would now like to walk you through the process of fitting the ARIMA model in light of the issues described previously. We take an example time series shown here.

    As we can see, this time series is a very interesting use. We observe a breakout in the beginning of the graph, three different trend lines, a lot of a anomalies, and weak seasonality patterns throughout the time series

    We can also see that ARIMA seems to fit the time series and seems aware of the trend and seasonality patterns. This is shown by blue dashed line which plots the fitted data.

    Lets see what the forecast look like?
  • As you can see, not very good. Because of the multiple trends and anomalies, the model is not able to successfully de-trend the time series, and gives us an flat linear forecast, which basically means its not able to forecast this time series.
    At this point I would also like to introduce the idea of confidence bands, which is the space between the top blue dotted line and bottom brown dotted line. The green dotted line is the mean of two. Since we are using 95% confidence interval, the forecast tells us that there is a 95% probability of the forecasted value lying between these two bands,
    As we can see here, if the confidence band is very big, so its hard to make a good judgment on the quality of forecast. The lower confidence band actually shows a downward trend here

    So how many Days In Green does this service have according to the forecast? It is the length of the mean forecast, with a confidence band of where the upper band hits threshold T and the length of the lower confidence band. If we assume each box in the grid is about 5 days, the DIG is 40, with a confidence band 10 – 40 days, where 40 is the the maximum forecast days
  • So in order to improve our model, we first apply our breakout algorithm to the raw time series. As we can see, the algorithm is able to detect a mean shift in the series , and we remove that data from the fit, as can be seen by the dashed fit line.

    As we can see this makes a huge difference in the forecast. The time series has enough stability that allows ARIMA to de-trend it and create a stationary time series. As we can see the seasonality is also visible in the forecast of the confidence bands. The lower confidence band also shows an upward trend, unlike the previous case.
    So in this case the service has 35 Days In Green, with a confidence band of 2-45


    However we still see the limitations of the model fitting. It is still susceptible to anomalies , like the big anomaly we see at the start of the fitted time series. The size of the anomaly actually flattens the forecast a little bit, and we also see a relatively wide confidence band.
  • We next apply are anomaly detection algorithm over the time series generated by the breakout and we observe that we have been able to eliminate the big anomaly. This results in the reduced weight of that data point on the model and gives more weightage to the later trend in the time series. This reduces the width of the confidence band, resulting in improved accuracy of the forecast. In this case the service has 25 Days In Green with a confidence band of 2-40
  • We would now like to compare DIG number for the 3 time series discussed earlier. The purple line at the bottom Is the forecast for the RAW time series, The magenta line in the middle is the forecast for time series with breakout detection and the dark blue line is the forecast for time series with breakout and anomaly detection.

    So the DIG number is the number forecast by the dark blue line, since it has smallest confidence band, and we can see it has significantly improved the forecast from other two time series. Based on the numbers shown previously, the service has 25 Days In Green, which is improvement of 10 days over DIG for time series with just the Breakout algorithm and 15 days over the raw time series.

    From a capacity perspective, In this case it would lead us to add capacity early , thus saving us from potential performance issues.

    In other situations, if we report a larger DIG number, then it can lead to potential cost savings, since we know that we can add capacity at a later date.

    And that in my view Is the significance of DIG.
  • So next we would like to look at some of the boundary conditions and limitations of this approach. One of the major boundary conditions we have found with the use of ARIMA is the condition of false seasonality at the end of a time series, where ARIMA tries to assume the change in pattern to be seasonality and tries to forecast that. As we can see , the confidence band widens pretty quickly.
  • As we have mentioned before the time series we see are not the most stable and we do a best effort to remove breakouts and anomalies, however the time series left may not provide a strong sense of seasonality or trend to allow the model to generate a good forecast. In these cases we may need to look at a longer time series.
  • Again some time series are not stable enough to forecast and we may not be able to use breakout or anomaly detection to make them forecastable. We will need to look at alternative methods of forecasting here and we are looking at them.
  • We have already deployed DIG in production and are using it to monitor hundreds of services in a fully automated fashion.
    Our current algorithm uses CPU data from our services but will be extending to other metrics as well.
    We are currently using to monitor DR compliance across multiple data centers and to detect services that get close to DR threshold, and allocate capacity accordingly

    One of the interesting outcomes of this works has been the idea of Utilization Based Allocation where we are trying to forecast when and how much capacity. The idea is to forecast the value of metric at a certain time in the future and use that forecast value to calculate expected capacity
  • To summarize the lessons we have learned: It’s of paramount importance to check for data fidelity w.r.t presence of anomalies, absence of seasonality. Further, no forecasting model is perfect. So, it’s critical to assess the forecasting error and continuously refine the model, particularly when the incoming data is dynamic in nature as is the case at Twitter.
  • ×