Customer experience is a top priority for Verizon, so they turned to DevOps best practices to address technical issues. The result was tremendous – 3x faster build times and a 50% drop in reported bugs – all within the first six months!
This success evolved to an automated delivery pipeline approach that leveraged cloud and container technology, and teams were able to deploy new features faster, directly into production.
With faster releases came vast technical complexity. Using Artificial Intelligence (AI)-powered monitoring, Verizon was able to transcend this problem, and today easily manages complex, web-scale DevOps in the cloud.
• Learn the DevOps “shift-left” quality model Verizon embraced to find issues before they reached production.
• Discover best practices that accelerated Verizon’s build and test cycle times by 3x in just six months.
• Understand what AI-powered technology is, and how it can help you master the complexity of DevOps cloud environments.
• Learn how Verizon uses DevOps and cloud monitoring as part of one integrated application delivery chain.
You’ll gain best practices and insights you can use immediately!
4. What have we done and achieved so far?
•3xfaster build and test cycles
•50% faster deployment
(every two weeks, rather than once a month)
•33% faster revenue realization
•50% reduction in issues
#1 Deployment & Pipeline #2 Feedback Loops #3 Cultural Change
AWS Webinar: https://pages.awscloud.com/devops_dynatrace_june_2017.html
17. Improve MTTR: Automate Mitigate with AI Data
Auto Mitigate!
1 CPU Exhausted? Add a new service instance!
3 Issue with BLUE only? Switch back to GREEN!
?Escalate at 2AM?
2 High Garbage Collection? Adjust/Revert Memory Settings!
4 Hung threads? Restart Service!
5 Still ongoing? Initiate Rollback!
Escalate
? Still ongoing?5
1
2
3
4
Mark Bad Commits
Update Dev Tickets
…
…
Impact Mitigated??
?
20. Actionable Feedback Loops: Business
Success Criteria
Labels become Key User Action
Live Data Queries
New Requirement Definition
21. Dev Perf/Test Ops Biz
Shift-Left: Break Pipeline Earlier
Improve MTTR: Auto-Mitigation
Shift-Right: Tags, Deploys, Events
Actionable Feedback Loops
22. DevOps is about … … creating more value …
… making money faster …
… by also making your life
easier!
23. Questions?
Give this a try and see for yourself!
Free Trial: http://bit.ly/dtsaastrial
Andreas GrabnerAnil Chintalapudi
Director, Verizon Enterprise Solutions
grabnerandi
DevOps Activist, Dynatrace
@grabnerandianilchintalapudi
24. Connect with us!
Participate in our Forum :
community.dynatrace.com
Like us on Facebook :
facebook.com/dynatrace
Follow us on LinkedIn :
linkedin.com/company/dynatrace
Follow us on Twitter :
twitter.com/dynatrace
Watch our Videos & Demos :
youtube.com/dynatrace
Read our Blog :
https://www.dynatrace.com/blog/
Lets start with what Verizon has already achieved. If you want to learn more – we talked about this during our AWS Webinar
At PERFORM I remember you talking a lot about how much you invested in your Pipeline, Automated Deployments as well as in your whole DevOps Tool chain
I also remember you using Hygieia and actually extending it to pull in Dynatrace UEM Data
I was also fascinated about how you changed culturally by allowing development teams to learn what it means to do “Test Driven Development”, “Deploying into the Cloud” and “Operating an Application”
We have seen many new technologies on the horizon. And with DevOps promoting “Choose the best technology for your problem” you quickly end up with a lot of moving pieces
The problem is that most monitoring tools simply aren’t good enough anymore.
That’s not to say they have zero value, they’re just not good enough. Why is that?
Despite expensive investments in multiple monitoring tools, we still see companies doing this bad behavior… war rooms, reactive, finger pointing...
You can’t afford to have people watching dashboards all the time.
You can’t afford to have 50 people in a war room or on bridge call responding to a problem.
If you’re doing this, you’re not innovating. And if you’re not innovating, you will become digital prey.
DevOps Operations Teams
Have a lot of “Feedback Loop data” but not the expertise to understand and act on it on their own
Feedback Loop is not Actionable!
Alert Drowning!
Results in Large Meetings during “Crisis Management”
Actual Numbers from Verizon on # of People & Hours involved in Root Cause Analysis and Remediating Actions
Explain the typical “MANUAL ACTIONS” of a “Crisis Management Situation”
Fault Domain Isolation: App, Infra or Configuration Problem
Get evidence on potential root cause, e.g: look up log files, code or config changes …
After consulting with “domain experts” decide on whether to
A) FIX
B) ROLLBACK
Now lets learn from some of those companies that made a transformation and those where DevOps was always in their DNA
Better Root Cause Information
Linking all the evidence from Code Change to Impact to Problem Ticket
“One Click” Remediation Action
Automating manual Remediation Actions to be executed by DevOps Operations Team
“One Click” follow up actions
Open Tickets in JIRA, …
Post Mortems in Confluence, …
Quickly Explain the approach Verizon itself took
And how that has inspired Dynatrace to build our own AI to support exactly these use cases for better production monitoring
But there is more than production! There is more we can do throughout the whole DevOps Toolchain
But there is more than production! There is more we can do throughout the whole DevOps Toolchain