Sydney Continuous Delivery Meetup May 2014
•Download as PPTX, PDF•
0 likes•1,203 views
I gave this presentation at the Sydney Continuous Delivery Meetup Group. The main goal was to talk about Performance Metrics that you should monitor along the pipeline. I examples in 4 different areas where deployments failed and how metrics would have helped preventing these problems
Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Viewers also liked
Viewers also liked (20)
Similar to Sydney Continuous Delivery Meetup May 2014
Similar to Sydney Continuous Delivery Meetup May 2014 (20)
More from Andreas Grabner
More from Andreas Grabner (17)
Recently uploaded
Recently uploaded (20)
Sydney Continuous Delivery Meetup May 2014
- 1. Performance Quality Metrics along CD Pipeline @grabnerandi http://apmblog.compuware.com
- 6. And just very recently @ Euro Song Contest
- 8. What you should check out …
- 9. Commit Stage • Compile • Execute Unit Test • Code Analysis • Build installers Automated Acceptance Testing Automated Capacity Testing Manual testing • Key showcases • Exploratory testing Release
- 11. Who is doing it? How many successful deployments can they do? 300 Deployments / Year 50-60 Deployments / Day 10+ Deployments / Day Every 11.6 seconds
- 12. More on Amazons Story 75% fewer outages since 2006 90% fewer outage minutes ~0.001% of deployments cause a problem Instantaneous automatic rollback Deploying every 11.6s
- 16. Testing is Important – and gives Confidence
- 17. But are we ready for “The Real” world?
- 18. Measure Performance during the game Ball Possession: 40 : 60 Fouls: 0 : 0 Score: 0 : 0 Minute 1 - 5
- 19. Measure Performance during the game Minute 6 - 35 Ball Possession: 80 : 20 Fouls: 2 : 12 Score: 0 : 0
- 21. Options “To Fix” the situation
- 22. Not always a happy ending Minute 90 Ball Possession: 80 : 20 Fouls: 4 : 25 Score: 3 : 0
- 24. How does that relate to Software?
- 25. From Deploy to … Deploy Promotion/Event Problems Ops Playbook War Room Timeline
- 26. The “War Room” – back then 'Houston, we have a problem‘ NASA Mission Control Center, Apollo 13, 1970
- 27. The “War Room” – NOW Facebook – December 2012
- 28. 4 Situations on WHY this happens, HOW to avoid it
- 31. Requirement: We need a report
- 32. Using Hibernate results in 4k+ SQL Statements to display 3 items! Hibernate Executes 4k+ Statements Individual Execution VERY FAST But Total SUM takes 6s
- 33. Requirement: We need a fancy UI
- 34. Using Telerik Controls Results in 9s for Data- Binding of UI Controls #1: Slow Stored Procedure Depending on Request execution time of this SP varies between 1 and 7.5s #2: 240! Similar SQL Statements Most of these 240! Statements are not prepared and just differ in things like Column Names
- 35. Metrics: # Total SQLs # SQLs / Web Request # Same SQLs / Request Transferred Rows Test: With realistic Data Dev: “Learn” Frameworks
- 38. Business Impact requires Action!
- 39. Solution: Cache to the RESCUE!!
- 40. Implementation and Rollout Implemented InMemory Cache Worked well in Load Testing
- 41. Result: Out of Memory Crashes!! Still crashes Problem fixed!Fixed Version Deployed
- 42. Metrics: Heap Size, # Objects Allocated, # Objects in Cache Cache Hit Ratio Test: With realistic Data
- 45. Adonair Load Spike resulted in Unavailability
- 46. Alternative: “GoDaddy goes DevOps” 1h before SuperBowl KickOff 1h after Game ended
- 48. Metrics: Availability Page Size, # Objects # Hosts, # Connections DevOps: “Feature” Switches
- 51. Mobile Landing Page of Super Bowl Ad 434 Resources in total on that page: 230 JPEGs, 75 PNGs, 50 GIFs, … Total size of ~ 20MB
- 52. m.store.com redirects to www.store.com ALL CSS and JS files are redirected to the www domain This is a lot of time “wasted” especially on high latency mobile connections
- 53. Metrics: Load Time, # Resources (Images, …), # HTTP 3xx, 4xx, 5xx Dev: Build for Mobile Test: Test on Mobile Ops: Monitor Mobile
- 55. # of Requests / User # of Log Messages # of Exceptions # Objects Allocated # Objects In Cache Cache Hit Ratio # of Images # of SQLs # SQLs per RequestAvailability # HTTP 3xx, 4xx Page Size
- 56. Commit Stage • Compile • Execute Unit Test • Code Analysis • Build installers Automated Acceptance Testing Automated Capacity Testing Manual testing • Key showcases • Exploratory testing Release Unit & Integration Tests Functional Tests Performance Tests Production Monitoring Functional Tests
- 57. 57
- 59. If we do all that
- 60. Which gives you more time for the real important things in life …
- 61. Want MORE of these and more details? http://apmblog.compuware.com
- 62. FREE Products & More Info • dynaTrace Enterprise – Full End-to-End Visibility in your Java, .NET, PHP Apps – Sign up for a 15 Days Free Trial on http://compuwareapm.com • dynaTrace AJAX Edition – Browser Diagnostics for IE + FF – Download @ http://ajax.dynatrace.com • Our Blog: http://apmblog.compuware.com
Editor's Notes
- MEASURE!!
- Teams are currently competing in the qualifications to compete in Brazil 2014
- This is “my” austrian national team soccer team. Their GOAL is to qualify for Brazil 2014. After the many failed attempts in the past we hired a new coach who’s goal is to form a new team that PERFORMs good enough to qualify
- In order to get there the team competed in many test games. Which gaves them a lot of confidence because they played against teams that were “easier” to beat. At the end of these tests we even started in the qualification with some wins against teams that we were expecting to win So – at the end of these “test and easy qualification games” we thought: “ALL GOOD – THE ROAD IS OPEN FOR 2014 – NOT ONLY WILL WE QUALIFY BUT WE ALSO BELIEVE WE HAVE SUCH A STRONG TEAM THAT WILL ALSO DO WELL AT THE WORLDCUP”
- Then reality kicked in when we had our first “real competitor” – it was the first qualification against a team whos quality level is at a level that we have to expect at the world cup. The competing team was Germany – and – based on these images you can see how the game went
- The coach is responsible to watch the game and see how things are going. Like in other sports – soccer has a couple of Key Performance Indicators such as Ball Possession, Fouls and the actual score The first 5 minutes actually didn’t look too bad
- After the first 5 minutes the game changes – with germany taking over the game in their typical way. The KPIs make this very clear The coach is responsible to react based on these values and how the game wents
- The coach should use more data for detailed analysis on what is going wrong in the game
- One of his options is to substitute players – or even change tactics Does this succeed based on the KPIs that we have seen before?
- Well – not always. Just replacing players – putting some in that are faster in chasing the ball doesn’t always help
- Story New Build Deployed on Thursday Evening Everything runs smooth on Friday Daytime An Ad Campaign hits the Air Friday Night The site crashes under load -> ALERTS GO OFF Restarting Server -> SERVER DOESN’T START Adding more Servers-> PROBLEM REMAINS Calling in the “App Experts” and Pizza Delivery!