SlideShare ist ein Scribd-Unternehmen logo
1 von 18
Downloaden Sie, um offline zu lesen
The inaugural report describing the on-call triumphs 
and challenges of the average IT professional.
Motivation 
Methodology 
This report, the 
first of its kind, is 
designed to capture 
the challenges of 
those who do the 
hard job of being 
on-call. 
Over the last 20 years, software has slowly eaten the world. 
Businesses that historically operated on 8-hour business days 
are now 24x7, navigating a global economy that never sleeps. 
The internet enablement of everything has thrust what was 
originally an important support role into the lifeline of business. 
Then, a decade ago, Agile software development forced 
more pressure into the IT system by deploying software at 
an ever-increasing rate to the point it’s at today - where 
innovative companies can deploy new software multiple 
times a day. DevOps is at the forefront of addressing the 
question of how IT deals with these problems but like Agile 
before it, there’s a learning curve. 
All participants were 18+ and located in North America. We 
received 500 responses that were acquired through a variety of 
partners and channels. The statistical relevance of this survey is 
based on a 95% confidence and a +/- 5% margin of error.
Who took this survey? Industry Breakdown 
COMPANIES 
Companies taking part were equally balanced 
between SMB and Enterprise. 
= Company Size 
50% 28% 22% 
4% 
4% 
5% 
5% 
DOERS: RESPONDENTS WERE THE PRIMARY PEOPLE DEALING WITH THE ISSUES. 
Role: System Administrator, Operations, IT, Developer, VP/Director, Programmer 
19% 
36% 
11% 16% 
501+ 51-500 1-50 
Half of all those who responded are from internet-forward 
businesses (internet services & software), meaning they 
deliver their value via the internet. 
Internet Service 
Software 
Professional Services 
Financial Services 
Media/Entertainment 
Healthcare 
Manufacturing 
Other
On-call is not getting any better and many 
believe there are huge inefficiencies in the 
current process. 
Over 70% of respondents are doing something 
with DevOps and consider themselves Agile. 
It takes most companies an average of 10-30 
minutes to resolve an incident with an average 
of 5 people involved in the resolution. 
Alert fatigue is a real thing (over 60% agree) and 
respondents believe that almost a quarter of all 
alerts are false alarms. 
Most teams believe that post-mortems (aka 
retrospectives) are an important piece of the 
process but are not doing them all of the time. 
TL;DR - Summary of Data
Being on-call sucks. 
BIGGEST PROBLEMS NOTED: 
What respondents said: 
“I get stressed, and that causes tension which then affects 
my marriage.” 
“I always have to warn my family when it’s my week on-call. 
Making sure I have my laptop with me 
at all times when I’m on-call is a hassle. Having to plan 
family activities around when I’m on-call.” 
“I have had to leave movies, cut dates short, affected 
Thanksgiving dinner (now a family joke on what will break 
on Thanksgiving Day) - can be fairly disruptive. Also affects 
planning functions during on-call times.” 
“It affects my health due to complications of tension, and 
anxiety over missing family events.” 
OVER 60% BELIEVE THAT THE 
PROBLEMS AROUND BEING 
ON-CALL ARE ONLY “SORT OF” 
GETTING BETTER OR ARE 
ACTUALLY GETTING WORSE. 
Not enough people sharing the load (burnout) 
People not responding to calls of help (and lack of accountability) 
Not revisiting incidents to reduce noise and adjust thresholds 
(lack of reporting & incident follow-up) 
Lack of documentation that is easily available 
VICTOROPS INTEL: 
The average on-call rotation of our customers is 7.6 days.
But it’s not all bad. 
We were amazed that in the face of 
adversity while being on-call, there 
were many people who reported that 
they enjoyed the responsibility of the role. 
The best part of the job? 
What respondents said: 
“You make an impact and truly help the customer.” 
“It only lasts a week.” 
“Being able to truly help someone out quickly when 
they are stressed and think it’s the end of the world/ 
can’t be fixed.” 
“I get to work on so many different situations so its never boring.” 
“Working with great engineers.” 
Want to make on-call suck less? You should invest in tools & processes that remove the pain.
Organizationally... 
HARDWARE IS STILL KING. 
While respondents stated that most of their infrastructure is still 
physical, more and more are moving to the cloud every year. 
The infrastructure breakdown: 
RESPONDENTS USE A NUMBER OF MONITORING 
SYSTEMS, EVEN WITHIN THE SAME COMPANY, 
BUT THE MAJORITY OF ANSWERS WERE: 
cloud 
36% 
metal VS 
63% 
Nagios/Icinga, New Relic and Other (Pingdom, Splunk, 
Cacti, Cloudwatch, Solarwinds, Microsoft Network Monitor, 
Zabbix, Zenoss, Sensu, CA Nimsoft, AppDynamics, Loggly, 
Datadog, Sentry, BMC Remedy, Dotcom-Monitor, Circonus, 
Crittercism, LogicMonitor) 
VICTOROPS INTEL: 
We see that most customers are using as many 
as 5 monitoring services.
DevOps means automation and 
solving the problem faster. 
AGILE PROCESSES ARE MATURE AND 
DEVOPS IS NOT FAR BEHIND. 
60% of respondents consider themselves Agile 
52% of teams have a year or more of DevOps experience. Most 
have heard of it but the adoption of processes is still a work in 
progress. 
58% are using infrastructure 
automation tools 
Of those, 75% agree that 
these automation tools have 
made being on-call easier 
If you haven’t looked at automation tools like Puppet or Chef yet, you probably should. Many of the respondents use one or the other.
The Job of On-Call | The Basics 
ON-CALL RESPONSIBILITIES ARE SHARED 
AMONGST A WIDE VARIETY OF ROLES. 
One of the difficult aspects of being on-call is that the role is 
inherently multidisciplinary. From issue to issue, the problem 
can be completely different. 
TODAY’S ON-CALL TEAMS ARE MADE UP OF 
MEMBERS FROM IT, SUPPORT, OPERATIONS, 
DEVOPS, AND DEVELOPMENT. 
Average duration of an on-call rotation: 
Less than a week 
One week 
Two weeks 
More than two weeks 
20% 
52% 
22% 
6% 
VICTOROPS INTEL: 
Across the VictorOps customer base, we see most 
companies resolve through collaborative problem 
solving, averaging 4-5 people in the ultimate resolution.
Managing On-call 
What respondents said when we asked them about their 
homegrown solutions: 
“(Do) anything other than grow it yourself.” 
“Pros are that it is built specifically for our company and how we 
work to support our clients. Cons is that it’s now 7 years old, hasn’t 
been given the dev time to evolve with the company’s needs so is 
no longer really fit for purpose.” 
While the vast majority of respondents had homebuilt 
systems, they were not happy with them. The relatively 
new availability of tools like VictorOps, PagerDuty and 
OpsGenie are moving those numbers and we expect to 
see more people turning to SaaS solutions. 
TWO BIGGEST PROBLEMS WITH 
CURRENT SOLUTION: 
ALMOST 50% HAVE NO IDEA WHAT 
THEY ARE PAYING FOR THEIR 
CURRENT SOLUTION. 
ABOUT 70% USE HOMEGROWN 
TOOLS OR PROCESSES TO SOLVE 
ON-CALL PROBLEMS. 
The scheduling and communications of who is 
on-call and the ability to change that easily 
Sharing information and getting the right person 
involved to solve the problem faster 
This is largely due to the fact that most are using homebuilt tools. 
Internally-developed solutions are typically very expensive to 
build and support, even if they are quite simple. Additionally, 
most companies have not conducted an analysis of what 
downtime actually costs them.
Alerts, oh my! 
63% REPORT ALERT FATIGUE AS AN ISSUE. 
This is what we expected to see. We believe that using social 
toolsets and conducting retrospectives is helping, resulting in 
alarms being effectively tuned and noise being removed. 
VICTOROPS INTEL: 
The average VictorOps customer sends us over 123 monitoring 
system notifications a day. As a result, the average customer is 
notified of a critical issue 20.9 times a day. 
STEPS TAKEN TO REDUCE THE NOISE: 
Adjust alert thresholds 
Regularly evaluate and delete superfluous alerts 
Route incidents to specific people or teams 
Best practice: Consider having retrospectives once a week, when doing the 
on-call handoff, in order to address alert thresholds and how to improve them. 
VICTOROPS MATH: 
Out of our respondents, 64% believe that up 
to a quarter of all alerts are false alarms. 
Each false alarm could conservatively cost an 
organization approximately $72. 
Avg of 3 devs working to resolve each incident 
Avg time to resolve an incident is 30 minutes 
VictorOps data shows a company average of 
7,716 alerts per year 
IF 1/4 of all those are assumed to be false, the 
total cost to an organization could conservatively 
be $138,888.
EMAIL IS STILL THE NUMBER ONE WAY 
TO KNOW ABOUT PROBLEMS. SADLY. 
The when and how of alerting: 
OUTAGES WILL HAPPEN WHEN THEY HAPPEN. 
When asked about when the majority of incidents take 
place, the responses were divided pretty equally between 
daytime and nighttime. Incidents are just as likely to 
occur anytime. 
There is a plethora of reasons why this is ineffective (low signal-to-noise 
ratio, chance of email getting lost, no sense of urgency) but 82% report 
that email is how they are alerted to issues. 
Following that, the most popular means of alert delivery are 
(respondents chose all that applied): 
82% Email 
57% SMS 
46% Telephone Call 
37 % Push Notification 
31% Dashboards 
17% Paging Service 
14% NOC 
3% Other 
Learn how to NOT have alert details lost in your email inbox by using push notifications via VictorOps mobile apps.
During the firefight... 
PEOPLE FIX PROBLEMS. 
Over 80% said that 1-5 people are 
normally involved in resolving a problem. 
TEAMS OFTEN UNDERESTIMATE 
The top 3 things that people are using to solve problems all 
involve communication and collaboration between people. 
The list of tools used during a firefight HOW LONG THE FIX WILL TAKE. 
involve (respondents chose all that applied): 
44% said that it takes 10-30 minutes, on average, to 
resolve the problem while 33% said that resolution 
takes a bit longer, from 30 minutes to an hour. 
72% Chat Platform 
65% 1-to-1 Phone Call 
50% Conference Call 
33% Wiki Articles 
30% Graph Tools 
24% Video Conferencing 
23% Google Hangout 
23% Runbooks 
33% 
44%
Other remediation 
CHATOPS IS A GROWING TREND. 
Only 28% are currently practicing ChatOps. In order to facilitate 
that process, most are using HipChat and before that, Campfire. 
Some are reporting the move to Slack, which shows that chat 
seems to be a fairly fluid capability inside companies. 
INTERNAL WIKIS ARE THE MOST USED 
MEANS OF ACCESSING INFORMATION 
TO SOLVE PROBLEMS. 
Most teams keep good wikis but often access to the right data 
is still a problem. The top reported issue to problem remediation 
was effective surfacing of correct internal wikis. 
Learn more about ChatOps & see how to attach runbooks to your alerts using annotations.
After the firefight... 
Half of all companies conduct 
post-mortems, or retrospectives, 
but sadly, the majority of those (75%) 
only do it after significant outages. 
Of those that do post-mortems or 
retrospectives, 65% strive for them 
to be blameless. 
The purpose of the post-mortem is to learn, not point fingers 
or call anyone out. Most of the teams doing post-mortems 
(retrospectives) are using them primarily to make their teams 
smarter, but also to share findings with the executive team. 
VICTOROPS INTEL: 
Our system data suggests that companies that use our post-mortem/ 
retrospective tools have less false alarms and less downtime.
Solving the problem faster. 
MOST TEAMS ARE USING COMMUNICATION 
AS A METRIC OF SUCCESS. 
AUDIO STILL DOMINATES. 
Ways to improve your remediation process 
(respondents chose all that applied): 
56% are using audio/video conferencing as part of remediation but 
the majority of those only use it when the outage necessitates it. 
When asked about the most valuable thing done to 
solve problems faster, the majority of those surveyed 
answered … 
RUNBOOKS AND 
COLLABORATION. 
56% Documentation 
55% Reporting 
50% Post-Mortems 
44% Alert Accuracy 
15% MTTR 
8% MTBF
In Summary 
DevOps is a growing trend from enterprise IT to SaaS solution providers. 
On-call support is no longer the exception but an ever-growing reality of 
what business has become. 
IT no longer operates under the doctrine of the “check engine” light, which 
used to alert to a generic problem that could be handled out-of-band. 
Today, with the emergence of DevOps, what was a support function inside of 
companies is now a critical part of business. The people, that make up these 
on-call teams, work real-time 24/7 to solve IT’s biggest challenges and are 
quickly becoming the backbone of what business is today. 
Full-stack applications involving multiple disciplines have changed the 
equation of what being on-call means. Teams now have 5 monitoring 
systems or more that are capable of providing very specific insights and data. 
Alongside this newfound power of visibility and measurement across systems, 
also comes the pain of alert fatigue and added complexity in problem solving. 
A new breed of tools, technologies and philosophies are being developed to 
help teams deal with this stress. 
CLEAR TRENDS INCLUDE: 
Widening the responsibilities of being on-call. Respondents reported that a 
wider array of disciplines are now on-call to support system problems. Alert 
fatigue is being managed with larger on-call teams. 
Investigation of new platform tools to manage and inform team members 
to problems. Most companies in the survey were using homebuilt systems 
to manage on-call people but the majority planned to move off those tools 
citing the lack of improvements in those tools compared to new off-the-shelf 
tools and services. 
Collaboration is essential to solving problems quickly. The top three ways 
of dealing with system problems were collaborative in nature, showing that 
problems can no longer be solved by just a few people, as systems have 
grown too complex. Scalable ways to bring other people into the resolution 
process are key to quick resolution. 
Ways to enable integrated documentation is a trend that is gaining speed. 
Many respondents stated it was easier to just solve the problem (again) than 
find the resolution in their documentation. Documentation needs to be 
delivered alongside the problem with annotation technology. 
The 2014 State of On-call report was the first of its kind opportunity to capture this sea-change 
in technology and how it is supported. We intend to continue to track this dynamic part of the 
technology problem and update this report annually to see how the story changes year to year.
Many thanks to the following partners 
who helped us with this report: 
This survey was brought to you by your friends at VictorOps. We make being on-call suck less. 
SEE HOW HERE

Weitere ähnliche Inhalte

Was ist angesagt?

Brighttalk brining it all together - final
Brighttalk   brining it all together - finalBrighttalk   brining it all together - final
Brighttalk brining it all together - finalAndrew White
 
Trustwave: 7 Experts on Transforming Your Threat Detection & Response Strategy
Trustwave: 7 Experts on Transforming Your Threat Detection & Response StrategyTrustwave: 7 Experts on Transforming Your Threat Detection & Response Strategy
Trustwave: 7 Experts on Transforming Your Threat Detection & Response StrategyMighty Guides, Inc.
 
Brighttalk high scale low touch and other bedtime stories - final
Brighttalk   high scale low touch and other bedtime stories - finalBrighttalk   high scale low touch and other bedtime stories - final
Brighttalk high scale low touch and other bedtime stories - finalAndrew White
 
Survey: Business Impact of IT Incident Communications
Survey: Business Impact of IT Incident CommunicationsSurvey: Business Impact of IT Incident Communications
Survey: Business Impact of IT Incident CommunicationsxMatters Inc
 
HP Education Solutions
HP Education SolutionsHP Education Solutions
HP Education SolutionsTejas Paldhe
 
World-Class Incident Response Management
World-Class Incident Response ManagementWorld-Class Incident Response Management
World-Class Incident Response ManagementKeith Smith
 
Blameless Post-mortems: Everything You Ever Wanted to Know
Blameless Post-mortems: Everything You Ever Wanted to KnowBlameless Post-mortems: Everything You Ever Wanted to Know
Blameless Post-mortems: Everything You Ever Wanted to KnowVictorOps
 
Mastering disaster e book Telehouse
Mastering disaster e book TelehouseMastering disaster e book Telehouse
Mastering disaster e book TelehouseTelehouse
 
2011 09 18 United "Platitudes, reality and promise"
2011 09 18 United "Platitudes, reality and promise"2011 09 18 United "Platitudes, reality and promise"
2011 09 18 United "Platitudes, reality and promise"Gene Kim
 
2014 Top 10 Predictions for BC/DR by Dr. Steven B Goldman
2014 Top 10 Predictions for BC/DR by Dr. Steven B Goldman2014 Top 10 Predictions for BC/DR by Dr. Steven B Goldman
2014 Top 10 Predictions for BC/DR by Dr. Steven B GoldmanxMattersMarketing
 
How to improve your system monitoring
How to improve your system monitoringHow to improve your system monitoring
How to improve your system monitoringAndrew White
 
Work from Home - Are You Prepared for the Future of Work?
Work from Home - Are You Prepared for the Future of Work?Work from Home - Are You Prepared for the Future of Work?
Work from Home - Are You Prepared for the Future of Work?Northwest Executive Education
 
Software and Tear
Software and TearSoftware and Tear
Software and TearJosh Howell
 
IT Survey: UK and Germany SMEs
IT Survey: UK and Germany SMEsIT Survey: UK and Germany SMEs
IT Survey: UK and Germany SMEsSolarWinds
 
7 key problems Water Industry need to solve
7 key problems Water Industry need to solve7 key problems Water Industry need to solve
7 key problems Water Industry need to solveDaniel Cardelús
 
Netadmin and Sysadmin Survey Results - US
Netadmin and Sysadmin Survey Results - USNetadmin and Sysadmin Survey Results - US
Netadmin and Sysadmin Survey Results - USSolarWinds
 
Software delivery perfomance duncan ham
Software delivery perfomance duncan hamSoftware delivery perfomance duncan ham
Software delivery perfomance duncan hamDuncan Ham
 
Security Differently - DevSecOps Days Austin 2019
Security Differently - DevSecOps Days Austin 2019Security Differently - DevSecOps Days Austin 2019
Security Differently - DevSecOps Days Austin 2019Aaron Rinehart
 

Was ist angesagt? (20)

Brighttalk brining it all together - final
Brighttalk   brining it all together - finalBrighttalk   brining it all together - final
Brighttalk brining it all together - final
 
Trustwave: 7 Experts on Transforming Your Threat Detection & Response Strategy
Trustwave: 7 Experts on Transforming Your Threat Detection & Response StrategyTrustwave: 7 Experts on Transforming Your Threat Detection & Response Strategy
Trustwave: 7 Experts on Transforming Your Threat Detection & Response Strategy
 
Brighttalk high scale low touch and other bedtime stories - final
Brighttalk   high scale low touch and other bedtime stories - finalBrighttalk   high scale low touch and other bedtime stories - final
Brighttalk high scale low touch and other bedtime stories - final
 
Survey: Business Impact of IT Incident Communications
Survey: Business Impact of IT Incident CommunicationsSurvey: Business Impact of IT Incident Communications
Survey: Business Impact of IT Incident Communications
 
HP Education Solutions
HP Education SolutionsHP Education Solutions
HP Education Solutions
 
World-Class Incident Response Management
World-Class Incident Response ManagementWorld-Class Incident Response Management
World-Class Incident Response Management
 
Blameless Post-mortems: Everything You Ever Wanted to Know
Blameless Post-mortems: Everything You Ever Wanted to KnowBlameless Post-mortems: Everything You Ever Wanted to Know
Blameless Post-mortems: Everything You Ever Wanted to Know
 
Mastering disaster e book Telehouse
Mastering disaster e book TelehouseMastering disaster e book Telehouse
Mastering disaster e book Telehouse
 
Dit yvol3iss41
Dit yvol3iss41Dit yvol3iss41
Dit yvol3iss41
 
2011 09 18 United "Platitudes, reality and promise"
2011 09 18 United "Platitudes, reality and promise"2011 09 18 United "Platitudes, reality and promise"
2011 09 18 United "Platitudes, reality and promise"
 
232 a7d01
232 a7d01232 a7d01
232 a7d01
 
2014 Top 10 Predictions for BC/DR by Dr. Steven B Goldman
2014 Top 10 Predictions for BC/DR by Dr. Steven B Goldman2014 Top 10 Predictions for BC/DR by Dr. Steven B Goldman
2014 Top 10 Predictions for BC/DR by Dr. Steven B Goldman
 
How to improve your system monitoring
How to improve your system monitoringHow to improve your system monitoring
How to improve your system monitoring
 
Work from Home - Are You Prepared for the Future of Work?
Work from Home - Are You Prepared for the Future of Work?Work from Home - Are You Prepared for the Future of Work?
Work from Home - Are You Prepared for the Future of Work?
 
Software and Tear
Software and TearSoftware and Tear
Software and Tear
 
IT Survey: UK and Germany SMEs
IT Survey: UK and Germany SMEsIT Survey: UK and Germany SMEs
IT Survey: UK and Germany SMEs
 
7 key problems Water Industry need to solve
7 key problems Water Industry need to solve7 key problems Water Industry need to solve
7 key problems Water Industry need to solve
 
Netadmin and Sysadmin Survey Results - US
Netadmin and Sysadmin Survey Results - USNetadmin and Sysadmin Survey Results - US
Netadmin and Sysadmin Survey Results - US
 
Software delivery perfomance duncan ham
Software delivery perfomance duncan hamSoftware delivery perfomance duncan ham
Software delivery perfomance duncan ham
 
Security Differently - DevSecOps Days Austin 2019
Security Differently - DevSecOps Days Austin 2019Security Differently - DevSecOps Days Austin 2019
Security Differently - DevSecOps Days Austin 2019
 

Andere mochten auch

OOP - Kelas abstrak dan Komposisi
OOP - Kelas abstrak dan KomposisiOOP - Kelas abstrak dan Komposisi
OOP - Kelas abstrak dan KomposisiKuliahKita
 
Especial dia das mães (2)
Especial dia das mães (2)Especial dia das mães (2)
Especial dia das mães (2)Meio & Mensagem
 
Especial namorados no vila mulher
Especial namorados no vila mulherEspecial namorados no vila mulher
Especial namorados no vila mulherMeio & Mensagem
 
Novabrasi lno teatro_pedromariano
Novabrasi lno teatro_pedromarianoNovabrasi lno teatro_pedromariano
Novabrasi lno teatro_pedromarianoMeio & Mensagem
 
Body po 2 kole ZBS 2014_15
Body po 2 kole ZBS 2014_15Body po 2 kole ZBS 2014_15
Body po 2 kole ZBS 2014_15Peter II
 
Especial final de ano 2014 30.10
Especial final de ano 2014 30.10Especial final de ano 2014 30.10
Especial final de ano 2014 30.10Meio & Mensagem
 
Manual 55030091 edited
Manual 55030091 editedManual 55030091 edited
Manual 55030091 editedPetite Ploy
 
положение предм. кафедры
положение предм. кафедрыположение предм. кафедры
положение предм. кафедрыpkgpkg
 
Information Evolution Dec 2
Information Evolution Dec 2Information Evolution Dec 2
Information Evolution Dec 2Mainak Roy
 
Diálogos capitais infraestrutura
Diálogos capitais infraestruturaDiálogos capitais infraestrutura
Diálogos capitais infraestruturaMeio & Mensagem
 

Andere mochten auch (20)

OOP - Kelas abstrak dan Komposisi
OOP - Kelas abstrak dan KomposisiOOP - Kelas abstrak dan Komposisi
OOP - Kelas abstrak dan Komposisi
 
Diálogos capitais pme
Diálogos capitais   pmeDiálogos capitais   pme
Diálogos capitais pme
 
Especial dia das mães (2)
Especial dia das mães (2)Especial dia das mães (2)
Especial dia das mães (2)
 
Psychologist ref
Psychologist refPsychologist ref
Psychologist ref
 
Especial namorados no vila mulher
Especial namorados no vila mulherEspecial namorados no vila mulher
Especial namorados no vila mulher
 
Arq219564.pdf correio
Arq219564.pdf  correioArq219564.pdf  correio
Arq219564.pdf correio
 
Novabrasi lno teatro_pedromariano
Novabrasi lno teatro_pedromarianoNovabrasi lno teatro_pedromariano
Novabrasi lno teatro_pedromariano
 
Body po 2 kole ZBS 2014_15
Body po 2 kole ZBS 2014_15Body po 2 kole ZBS 2014_15
Body po 2 kole ZBS 2014_15
 
Banner-Produtos-copy.jpg
Banner-Produtos-copy.jpgBanner-Produtos-copy.jpg
Banner-Produtos-copy.jpg
 
Content marketing 101
Content marketing 101Content marketing 101
Content marketing 101
 
№14
 №14 №14
№14
 
Mobile info money 10.12
Mobile info money 10.12Mobile info money 10.12
Mobile info money 10.12
 
Especial final de ano 2014 30.10
Especial final de ano 2014 30.10Especial final de ano 2014 30.10
Especial final de ano 2014 30.10
 
Manual 55030091 edited
Manual 55030091 editedManual 55030091 edited
Manual 55030091 edited
 
положение предм. кафедры
положение предм. кафедрыположение предм. кафедры
положение предм. кафедры
 
Sobre rodas 12.11
Sobre rodas 12.11Sobre rodas 12.11
Sobre rodas 12.11
 
En büyük göller
En büyük göllerEn büyük göller
En büyük göller
 
Information Evolution Dec 2
Information Evolution Dec 2Information Evolution Dec 2
Information Evolution Dec 2
 
Pod
PodPod
Pod
 
Diálogos capitais infraestrutura
Diálogos capitais infraestruturaDiálogos capitais infraestrutura
Diálogos capitais infraestrutura
 

Ähnlich wie State of on call report 2014

Continuing Education Conferance
Continuing Education ConferanceContinuing Education Conferance
Continuing Education ConferanceTommy Riggins
 
Impacts cloud remote_workforce
Impacts cloud remote_workforceImpacts cloud remote_workforce
Impacts cloud remote_workforceRodrigo Varas
 
VIPRE --Responding to Cyberattacks
VIPRE --Responding to CyberattacksVIPRE --Responding to Cyberattacks
VIPRE --Responding to CyberattacksAbhishek Sood
 
Ryan ArcherTopic Panic AttacksSpecific Purpose To inform my.docx
Ryan ArcherTopic Panic AttacksSpecific Purpose To inform my.docxRyan ArcherTopic Panic AttacksSpecific Purpose To inform my.docx
Ryan ArcherTopic Panic AttacksSpecific Purpose To inform my.docxjeffsrosalyn
 
Ryan ArcherTopic Panic AttacksSpecific Purpose To inform my.docx
Ryan ArcherTopic Panic AttacksSpecific Purpose To inform my.docxRyan ArcherTopic Panic AttacksSpecific Purpose To inform my.docx
Ryan ArcherTopic Panic AttacksSpecific Purpose To inform my.docxrtodd599
 
BCS ELITE: Organisational change survey
BCS ELITE: Organisational change surveyBCS ELITE: Organisational change survey
BCS ELITE: Organisational change surveybcselite
 
Idge dell reignite2014 qp #2
Idge dell reignite2014 qp #2Idge dell reignite2014 qp #2
Idge dell reignite2014 qp #2jmariani14
 
The Seceret to Implementing New Technology
The Seceret to Implementing New TechnologyThe Seceret to Implementing New Technology
The Seceret to Implementing New TechnologyGeoff Talbot
 
5 Tips for Implementing New Technology
5 Tips for Implementing New Technology5 Tips for Implementing New Technology
5 Tips for Implementing New TechnologyKelly Burlingham
 
State of endpoint risk v3
State of endpoint risk v3State of endpoint risk v3
State of endpoint risk v3Lumension
 
State of endpoint risk v3
State of endpoint risk v3State of endpoint risk v3
State of endpoint risk v3Lumension
 
State of endpoint risk v3
State of endpoint risk v3State of endpoint risk v3
State of endpoint risk v3Lumension
 
React Faster and Better: New Approaches for Advanced Incident Response
React Faster and Better: New Approaches for Advanced Incident ResponseReact Faster and Better: New Approaches for Advanced Incident Response
React Faster and Better: New Approaches for Advanced Incident ResponseSilvioPappalardo
 
The Impacts of COVID-19 on Enterprise IT
The Impacts of COVID-19 on Enterprise ITThe Impacts of COVID-19 on Enterprise IT
The Impacts of COVID-19 on Enterprise ITInsight
 
EndpointSecurityConcerns2014
EndpointSecurityConcerns2014EndpointSecurityConcerns2014
EndpointSecurityConcerns2014Peggy Lawless
 

Ähnlich wie State of on call report 2014 (20)

Continuing Education Conferance
Continuing Education ConferanceContinuing Education Conferance
Continuing Education Conferance
 
Executive Breach Response Playbook
Executive Breach Response PlaybookExecutive Breach Response Playbook
Executive Breach Response Playbook
 
Impacts cloud remote_workforce
Impacts cloud remote_workforceImpacts cloud remote_workforce
Impacts cloud remote_workforce
 
VIPRE --Responding to Cyberattacks
VIPRE --Responding to CyberattacksVIPRE --Responding to Cyberattacks
VIPRE --Responding to Cyberattacks
 
Ryan ArcherTopic Panic AttacksSpecific Purpose To inform my.docx
Ryan ArcherTopic Panic AttacksSpecific Purpose To inform my.docxRyan ArcherTopic Panic AttacksSpecific Purpose To inform my.docx
Ryan ArcherTopic Panic AttacksSpecific Purpose To inform my.docx
 
Ryan ArcherTopic Panic AttacksSpecific Purpose To inform my.docx
Ryan ArcherTopic Panic AttacksSpecific Purpose To inform my.docxRyan ArcherTopic Panic AttacksSpecific Purpose To inform my.docx
Ryan ArcherTopic Panic AttacksSpecific Purpose To inform my.docx
 
Breaches Are Bad for Business. How Will You Detect and Respond to Your Next C...
Breaches Are Bad for Business. How Will You Detect and Respond to Your Next C...Breaches Are Bad for Business. How Will You Detect and Respond to Your Next C...
Breaches Are Bad for Business. How Will You Detect and Respond to Your Next C...
 
BCS ELITE: Organisational change survey
BCS ELITE: Organisational change surveyBCS ELITE: Organisational change survey
BCS ELITE: Organisational change survey
 
Clear These 3 User Adoption Hurdles
Clear These 3 User Adoption HurdlesClear These 3 User Adoption Hurdles
Clear These 3 User Adoption Hurdles
 
It’s a world of bugs after all
It’s a world of bugs after allIt’s a world of bugs after all
It’s a world of bugs after all
 
Idge dell reignite2014 qp #2
Idge dell reignite2014 qp #2Idge dell reignite2014 qp #2
Idge dell reignite2014 qp #2
 
The Seceret to Implementing New Technology
The Seceret to Implementing New TechnologyThe Seceret to Implementing New Technology
The Seceret to Implementing New Technology
 
5 Tips for Implementing New Technology
5 Tips for Implementing New Technology5 Tips for Implementing New Technology
5 Tips for Implementing New Technology
 
State of endpoint risk v3
State of endpoint risk v3State of endpoint risk v3
State of endpoint risk v3
 
State of endpoint risk v3
State of endpoint risk v3State of endpoint risk v3
State of endpoint risk v3
 
State of endpoint risk v3
State of endpoint risk v3State of endpoint risk v3
State of endpoint risk v3
 
React Faster and Better: New Approaches for Advanced Incident Response
React Faster and Better: New Approaches for Advanced Incident ResponseReact Faster and Better: New Approaches for Advanced Incident Response
React Faster and Better: New Approaches for Advanced Incident Response
 
The Impacts of COVID-19 on Enterprise IT
The Impacts of COVID-19 on Enterprise ITThe Impacts of COVID-19 on Enterprise IT
The Impacts of COVID-19 on Enterprise IT
 
EndpointSecurityConcerns2014
EndpointSecurityConcerns2014EndpointSecurityConcerns2014
EndpointSecurityConcerns2014
 
Productivity 3.0
Productivity 3.0Productivity 3.0
Productivity 3.0
 

Kürzlich hochgeladen

React 19: Revolutionizing Web Development
React 19: Revolutionizing Web DevelopmentReact 19: Revolutionizing Web Development
React 19: Revolutionizing Web DevelopmentBOSC Tech Labs
 
How to Improve the Employee Experience? - HRMS Software
How to Improve the Employee Experience? - HRMS SoftwareHow to Improve the Employee Experience? - HRMS Software
How to Improve the Employee Experience? - HRMS SoftwareNYGGS Automation Suite
 
Big Data Bellevue Meetup | Enhancing Python Data Loading in the Cloud for AI/ML
Big Data Bellevue Meetup | Enhancing Python Data Loading in the Cloud for AI/MLBig Data Bellevue Meetup | Enhancing Python Data Loading in the Cloud for AI/ML
Big Data Bellevue Meetup | Enhancing Python Data Loading in the Cloud for AI/MLAlluxio, Inc.
 
New ThousandEyes Product Features and Release Highlights: March 2024
New ThousandEyes Product Features and Release Highlights: March 2024New ThousandEyes Product Features and Release Highlights: March 2024
New ThousandEyes Product Features and Release Highlights: March 2024ThousandEyes
 
Sales Territory Management: A Definitive Guide to Expand Sales Coverage
Sales Territory Management: A Definitive Guide to Expand Sales CoverageSales Territory Management: A Definitive Guide to Expand Sales Coverage
Sales Territory Management: A Definitive Guide to Expand Sales CoverageDista
 
Cybersecurity Challenges with Generative AI - for Good and Bad
Cybersecurity Challenges with Generative AI - for Good and BadCybersecurity Challenges with Generative AI - for Good and Bad
Cybersecurity Challenges with Generative AI - for Good and BadIvo Andreev
 
Vectors are the new JSON in PostgreSQL (SCaLE 21x)
Vectors are the new JSON in PostgreSQL (SCaLE 21x)Vectors are the new JSON in PostgreSQL (SCaLE 21x)
Vectors are the new JSON in PostgreSQL (SCaLE 21x)Jonathan Katz
 
Generative AI for Cybersecurity - EC-Council
Generative AI for Cybersecurity - EC-CouncilGenerative AI for Cybersecurity - EC-Council
Generative AI for Cybersecurity - EC-CouncilVICTOR MAESTRE RAMIREZ
 
eAuditor Audits & Inspections - conduct field inspections
eAuditor Audits & Inspections - conduct field inspectionseAuditor Audits & Inspections - conduct field inspections
eAuditor Audits & Inspections - conduct field inspectionsNirav Modi
 
Fields in Java and Kotlin and what to expect.pptx
Fields in Java and Kotlin and what to expect.pptxFields in Java and Kotlin and what to expect.pptx
Fields in Java and Kotlin and what to expect.pptxJoão Esperancinha
 
Webinar_050417_LeClair12345666777889.ppt
Webinar_050417_LeClair12345666777889.pptWebinar_050417_LeClair12345666777889.ppt
Webinar_050417_LeClair12345666777889.pptkinjal48
 
Enterprise Document Management System - Qualityze Inc
Enterprise Document Management System - Qualityze IncEnterprise Document Management System - Qualityze Inc
Enterprise Document Management System - Qualityze Incrobinwilliams8624
 
Deep Learning for Images with PyTorch - Datacamp
Deep Learning for Images with PyTorch - DatacampDeep Learning for Images with PyTorch - Datacamp
Deep Learning for Images with PyTorch - DatacampVICTOR MAESTRE RAMIREZ
 
Understanding Native Mobile App Development
Understanding Native Mobile App DevelopmentUnderstanding Native Mobile App Development
Understanding Native Mobile App DevelopmentMobulous Technologies
 
ERP For Electrical and Electronics manufecturing.pptx
ERP For Electrical and Electronics manufecturing.pptxERP For Electrical and Electronics manufecturing.pptx
ERP For Electrical and Electronics manufecturing.pptxAutus Cyber Tech
 
20240319 Car Simulator Plan.pptx . Plan for a JavaScript Car Driving Simulator.
20240319 Car Simulator Plan.pptx . Plan for a JavaScript Car Driving Simulator.20240319 Car Simulator Plan.pptx . Plan for a JavaScript Car Driving Simulator.
20240319 Car Simulator Plan.pptx . Plan for a JavaScript Car Driving Simulator.Sharon Liu
 
Optimizing Business Potential: A Guide to Outsourcing Engineering Services in...
Optimizing Business Potential: A Guide to Outsourcing Engineering Services in...Optimizing Business Potential: A Guide to Outsourcing Engineering Services in...
Optimizing Business Potential: A Guide to Outsourcing Engineering Services in...Jaydeep Chhasatia
 
online pdf editor software solutions.pdf
online pdf editor software solutions.pdfonline pdf editor software solutions.pdf
online pdf editor software solutions.pdfMeon Technology
 
Your Vision, Our Expertise: TECUNIQUE's Tailored Software Teams
Your Vision, Our Expertise: TECUNIQUE's Tailored Software TeamsYour Vision, Our Expertise: TECUNIQUE's Tailored Software Teams
Your Vision, Our Expertise: TECUNIQUE's Tailored Software TeamsJaydeep Chhasatia
 
Top Software Development Trends in 2024
Top Software Development Trends in  2024Top Software Development Trends in  2024
Top Software Development Trends in 2024Mind IT Systems
 

Kürzlich hochgeladen (20)

React 19: Revolutionizing Web Development
React 19: Revolutionizing Web DevelopmentReact 19: Revolutionizing Web Development
React 19: Revolutionizing Web Development
 
How to Improve the Employee Experience? - HRMS Software
How to Improve the Employee Experience? - HRMS SoftwareHow to Improve the Employee Experience? - HRMS Software
How to Improve the Employee Experience? - HRMS Software
 
Big Data Bellevue Meetup | Enhancing Python Data Loading in the Cloud for AI/ML
Big Data Bellevue Meetup | Enhancing Python Data Loading in the Cloud for AI/MLBig Data Bellevue Meetup | Enhancing Python Data Loading in the Cloud for AI/ML
Big Data Bellevue Meetup | Enhancing Python Data Loading in the Cloud for AI/ML
 
New ThousandEyes Product Features and Release Highlights: March 2024
New ThousandEyes Product Features and Release Highlights: March 2024New ThousandEyes Product Features and Release Highlights: March 2024
New ThousandEyes Product Features and Release Highlights: March 2024
 
Sales Territory Management: A Definitive Guide to Expand Sales Coverage
Sales Territory Management: A Definitive Guide to Expand Sales CoverageSales Territory Management: A Definitive Guide to Expand Sales Coverage
Sales Territory Management: A Definitive Guide to Expand Sales Coverage
 
Cybersecurity Challenges with Generative AI - for Good and Bad
Cybersecurity Challenges with Generative AI - for Good and BadCybersecurity Challenges with Generative AI - for Good and Bad
Cybersecurity Challenges with Generative AI - for Good and Bad
 
Vectors are the new JSON in PostgreSQL (SCaLE 21x)
Vectors are the new JSON in PostgreSQL (SCaLE 21x)Vectors are the new JSON in PostgreSQL (SCaLE 21x)
Vectors are the new JSON in PostgreSQL (SCaLE 21x)
 
Generative AI for Cybersecurity - EC-Council
Generative AI for Cybersecurity - EC-CouncilGenerative AI for Cybersecurity - EC-Council
Generative AI for Cybersecurity - EC-Council
 
eAuditor Audits & Inspections - conduct field inspections
eAuditor Audits & Inspections - conduct field inspectionseAuditor Audits & Inspections - conduct field inspections
eAuditor Audits & Inspections - conduct field inspections
 
Fields in Java and Kotlin and what to expect.pptx
Fields in Java and Kotlin and what to expect.pptxFields in Java and Kotlin and what to expect.pptx
Fields in Java and Kotlin and what to expect.pptx
 
Webinar_050417_LeClair12345666777889.ppt
Webinar_050417_LeClair12345666777889.pptWebinar_050417_LeClair12345666777889.ppt
Webinar_050417_LeClair12345666777889.ppt
 
Enterprise Document Management System - Qualityze Inc
Enterprise Document Management System - Qualityze IncEnterprise Document Management System - Qualityze Inc
Enterprise Document Management System - Qualityze Inc
 
Deep Learning for Images with PyTorch - Datacamp
Deep Learning for Images with PyTorch - DatacampDeep Learning for Images with PyTorch - Datacamp
Deep Learning for Images with PyTorch - Datacamp
 
Understanding Native Mobile App Development
Understanding Native Mobile App DevelopmentUnderstanding Native Mobile App Development
Understanding Native Mobile App Development
 
ERP For Electrical and Electronics manufecturing.pptx
ERP For Electrical and Electronics manufecturing.pptxERP For Electrical and Electronics manufecturing.pptx
ERP For Electrical and Electronics manufecturing.pptx
 
20240319 Car Simulator Plan.pptx . Plan for a JavaScript Car Driving Simulator.
20240319 Car Simulator Plan.pptx . Plan for a JavaScript Car Driving Simulator.20240319 Car Simulator Plan.pptx . Plan for a JavaScript Car Driving Simulator.
20240319 Car Simulator Plan.pptx . Plan for a JavaScript Car Driving Simulator.
 
Optimizing Business Potential: A Guide to Outsourcing Engineering Services in...
Optimizing Business Potential: A Guide to Outsourcing Engineering Services in...Optimizing Business Potential: A Guide to Outsourcing Engineering Services in...
Optimizing Business Potential: A Guide to Outsourcing Engineering Services in...
 
online pdf editor software solutions.pdf
online pdf editor software solutions.pdfonline pdf editor software solutions.pdf
online pdf editor software solutions.pdf
 
Your Vision, Our Expertise: TECUNIQUE's Tailored Software Teams
Your Vision, Our Expertise: TECUNIQUE's Tailored Software TeamsYour Vision, Our Expertise: TECUNIQUE's Tailored Software Teams
Your Vision, Our Expertise: TECUNIQUE's Tailored Software Teams
 
Top Software Development Trends in 2024
Top Software Development Trends in  2024Top Software Development Trends in  2024
Top Software Development Trends in 2024
 

State of on call report 2014

  • 1. The inaugural report describing the on-call triumphs and challenges of the average IT professional.
  • 2. Motivation Methodology This report, the first of its kind, is designed to capture the challenges of those who do the hard job of being on-call. Over the last 20 years, software has slowly eaten the world. Businesses that historically operated on 8-hour business days are now 24x7, navigating a global economy that never sleeps. The internet enablement of everything has thrust what was originally an important support role into the lifeline of business. Then, a decade ago, Agile software development forced more pressure into the IT system by deploying software at an ever-increasing rate to the point it’s at today - where innovative companies can deploy new software multiple times a day. DevOps is at the forefront of addressing the question of how IT deals with these problems but like Agile before it, there’s a learning curve. All participants were 18+ and located in North America. We received 500 responses that were acquired through a variety of partners and channels. The statistical relevance of this survey is based on a 95% confidence and a +/- 5% margin of error.
  • 3. Who took this survey? Industry Breakdown COMPANIES Companies taking part were equally balanced between SMB and Enterprise. = Company Size 50% 28% 22% 4% 4% 5% 5% DOERS: RESPONDENTS WERE THE PRIMARY PEOPLE DEALING WITH THE ISSUES. Role: System Administrator, Operations, IT, Developer, VP/Director, Programmer 19% 36% 11% 16% 501+ 51-500 1-50 Half of all those who responded are from internet-forward businesses (internet services & software), meaning they deliver their value via the internet. Internet Service Software Professional Services Financial Services Media/Entertainment Healthcare Manufacturing Other
  • 4. On-call is not getting any better and many believe there are huge inefficiencies in the current process. Over 70% of respondents are doing something with DevOps and consider themselves Agile. It takes most companies an average of 10-30 minutes to resolve an incident with an average of 5 people involved in the resolution. Alert fatigue is a real thing (over 60% agree) and respondents believe that almost a quarter of all alerts are false alarms. Most teams believe that post-mortems (aka retrospectives) are an important piece of the process but are not doing them all of the time. TL;DR - Summary of Data
  • 5. Being on-call sucks. BIGGEST PROBLEMS NOTED: What respondents said: “I get stressed, and that causes tension which then affects my marriage.” “I always have to warn my family when it’s my week on-call. Making sure I have my laptop with me at all times when I’m on-call is a hassle. Having to plan family activities around when I’m on-call.” “I have had to leave movies, cut dates short, affected Thanksgiving dinner (now a family joke on what will break on Thanksgiving Day) - can be fairly disruptive. Also affects planning functions during on-call times.” “It affects my health due to complications of tension, and anxiety over missing family events.” OVER 60% BELIEVE THAT THE PROBLEMS AROUND BEING ON-CALL ARE ONLY “SORT OF” GETTING BETTER OR ARE ACTUALLY GETTING WORSE. Not enough people sharing the load (burnout) People not responding to calls of help (and lack of accountability) Not revisiting incidents to reduce noise and adjust thresholds (lack of reporting & incident follow-up) Lack of documentation that is easily available VICTOROPS INTEL: The average on-call rotation of our customers is 7.6 days.
  • 6. But it’s not all bad. We were amazed that in the face of adversity while being on-call, there were many people who reported that they enjoyed the responsibility of the role. The best part of the job? What respondents said: “You make an impact and truly help the customer.” “It only lasts a week.” “Being able to truly help someone out quickly when they are stressed and think it’s the end of the world/ can’t be fixed.” “I get to work on so many different situations so its never boring.” “Working with great engineers.” Want to make on-call suck less? You should invest in tools & processes that remove the pain.
  • 7. Organizationally... HARDWARE IS STILL KING. While respondents stated that most of their infrastructure is still physical, more and more are moving to the cloud every year. The infrastructure breakdown: RESPONDENTS USE A NUMBER OF MONITORING SYSTEMS, EVEN WITHIN THE SAME COMPANY, BUT THE MAJORITY OF ANSWERS WERE: cloud 36% metal VS 63% Nagios/Icinga, New Relic and Other (Pingdom, Splunk, Cacti, Cloudwatch, Solarwinds, Microsoft Network Monitor, Zabbix, Zenoss, Sensu, CA Nimsoft, AppDynamics, Loggly, Datadog, Sentry, BMC Remedy, Dotcom-Monitor, Circonus, Crittercism, LogicMonitor) VICTOROPS INTEL: We see that most customers are using as many as 5 monitoring services.
  • 8. DevOps means automation and solving the problem faster. AGILE PROCESSES ARE MATURE AND DEVOPS IS NOT FAR BEHIND. 60% of respondents consider themselves Agile 52% of teams have a year or more of DevOps experience. Most have heard of it but the adoption of processes is still a work in progress. 58% are using infrastructure automation tools Of those, 75% agree that these automation tools have made being on-call easier If you haven’t looked at automation tools like Puppet or Chef yet, you probably should. Many of the respondents use one or the other.
  • 9. The Job of On-Call | The Basics ON-CALL RESPONSIBILITIES ARE SHARED AMONGST A WIDE VARIETY OF ROLES. One of the difficult aspects of being on-call is that the role is inherently multidisciplinary. From issue to issue, the problem can be completely different. TODAY’S ON-CALL TEAMS ARE MADE UP OF MEMBERS FROM IT, SUPPORT, OPERATIONS, DEVOPS, AND DEVELOPMENT. Average duration of an on-call rotation: Less than a week One week Two weeks More than two weeks 20% 52% 22% 6% VICTOROPS INTEL: Across the VictorOps customer base, we see most companies resolve through collaborative problem solving, averaging 4-5 people in the ultimate resolution.
  • 10. Managing On-call What respondents said when we asked them about their homegrown solutions: “(Do) anything other than grow it yourself.” “Pros are that it is built specifically for our company and how we work to support our clients. Cons is that it’s now 7 years old, hasn’t been given the dev time to evolve with the company’s needs so is no longer really fit for purpose.” While the vast majority of respondents had homebuilt systems, they were not happy with them. The relatively new availability of tools like VictorOps, PagerDuty and OpsGenie are moving those numbers and we expect to see more people turning to SaaS solutions. TWO BIGGEST PROBLEMS WITH CURRENT SOLUTION: ALMOST 50% HAVE NO IDEA WHAT THEY ARE PAYING FOR THEIR CURRENT SOLUTION. ABOUT 70% USE HOMEGROWN TOOLS OR PROCESSES TO SOLVE ON-CALL PROBLEMS. The scheduling and communications of who is on-call and the ability to change that easily Sharing information and getting the right person involved to solve the problem faster This is largely due to the fact that most are using homebuilt tools. Internally-developed solutions are typically very expensive to build and support, even if they are quite simple. Additionally, most companies have not conducted an analysis of what downtime actually costs them.
  • 11. Alerts, oh my! 63% REPORT ALERT FATIGUE AS AN ISSUE. This is what we expected to see. We believe that using social toolsets and conducting retrospectives is helping, resulting in alarms being effectively tuned and noise being removed. VICTOROPS INTEL: The average VictorOps customer sends us over 123 monitoring system notifications a day. As a result, the average customer is notified of a critical issue 20.9 times a day. STEPS TAKEN TO REDUCE THE NOISE: Adjust alert thresholds Regularly evaluate and delete superfluous alerts Route incidents to specific people or teams Best practice: Consider having retrospectives once a week, when doing the on-call handoff, in order to address alert thresholds and how to improve them. VICTOROPS MATH: Out of our respondents, 64% believe that up to a quarter of all alerts are false alarms. Each false alarm could conservatively cost an organization approximately $72. Avg of 3 devs working to resolve each incident Avg time to resolve an incident is 30 minutes VictorOps data shows a company average of 7,716 alerts per year IF 1/4 of all those are assumed to be false, the total cost to an organization could conservatively be $138,888.
  • 12. EMAIL IS STILL THE NUMBER ONE WAY TO KNOW ABOUT PROBLEMS. SADLY. The when and how of alerting: OUTAGES WILL HAPPEN WHEN THEY HAPPEN. When asked about when the majority of incidents take place, the responses were divided pretty equally between daytime and nighttime. Incidents are just as likely to occur anytime. There is a plethora of reasons why this is ineffective (low signal-to-noise ratio, chance of email getting lost, no sense of urgency) but 82% report that email is how they are alerted to issues. Following that, the most popular means of alert delivery are (respondents chose all that applied): 82% Email 57% SMS 46% Telephone Call 37 % Push Notification 31% Dashboards 17% Paging Service 14% NOC 3% Other Learn how to NOT have alert details lost in your email inbox by using push notifications via VictorOps mobile apps.
  • 13. During the firefight... PEOPLE FIX PROBLEMS. Over 80% said that 1-5 people are normally involved in resolving a problem. TEAMS OFTEN UNDERESTIMATE The top 3 things that people are using to solve problems all involve communication and collaboration between people. The list of tools used during a firefight HOW LONG THE FIX WILL TAKE. involve (respondents chose all that applied): 44% said that it takes 10-30 minutes, on average, to resolve the problem while 33% said that resolution takes a bit longer, from 30 minutes to an hour. 72% Chat Platform 65% 1-to-1 Phone Call 50% Conference Call 33% Wiki Articles 30% Graph Tools 24% Video Conferencing 23% Google Hangout 23% Runbooks 33% 44%
  • 14. Other remediation CHATOPS IS A GROWING TREND. Only 28% are currently practicing ChatOps. In order to facilitate that process, most are using HipChat and before that, Campfire. Some are reporting the move to Slack, which shows that chat seems to be a fairly fluid capability inside companies. INTERNAL WIKIS ARE THE MOST USED MEANS OF ACCESSING INFORMATION TO SOLVE PROBLEMS. Most teams keep good wikis but often access to the right data is still a problem. The top reported issue to problem remediation was effective surfacing of correct internal wikis. Learn more about ChatOps & see how to attach runbooks to your alerts using annotations.
  • 15. After the firefight... Half of all companies conduct post-mortems, or retrospectives, but sadly, the majority of those (75%) only do it after significant outages. Of those that do post-mortems or retrospectives, 65% strive for them to be blameless. The purpose of the post-mortem is to learn, not point fingers or call anyone out. Most of the teams doing post-mortems (retrospectives) are using them primarily to make their teams smarter, but also to share findings with the executive team. VICTOROPS INTEL: Our system data suggests that companies that use our post-mortem/ retrospective tools have less false alarms and less downtime.
  • 16. Solving the problem faster. MOST TEAMS ARE USING COMMUNICATION AS A METRIC OF SUCCESS. AUDIO STILL DOMINATES. Ways to improve your remediation process (respondents chose all that applied): 56% are using audio/video conferencing as part of remediation but the majority of those only use it when the outage necessitates it. When asked about the most valuable thing done to solve problems faster, the majority of those surveyed answered … RUNBOOKS AND COLLABORATION. 56% Documentation 55% Reporting 50% Post-Mortems 44% Alert Accuracy 15% MTTR 8% MTBF
  • 17. In Summary DevOps is a growing trend from enterprise IT to SaaS solution providers. On-call support is no longer the exception but an ever-growing reality of what business has become. IT no longer operates under the doctrine of the “check engine” light, which used to alert to a generic problem that could be handled out-of-band. Today, with the emergence of DevOps, what was a support function inside of companies is now a critical part of business. The people, that make up these on-call teams, work real-time 24/7 to solve IT’s biggest challenges and are quickly becoming the backbone of what business is today. Full-stack applications involving multiple disciplines have changed the equation of what being on-call means. Teams now have 5 monitoring systems or more that are capable of providing very specific insights and data. Alongside this newfound power of visibility and measurement across systems, also comes the pain of alert fatigue and added complexity in problem solving. A new breed of tools, technologies and philosophies are being developed to help teams deal with this stress. CLEAR TRENDS INCLUDE: Widening the responsibilities of being on-call. Respondents reported that a wider array of disciplines are now on-call to support system problems. Alert fatigue is being managed with larger on-call teams. Investigation of new platform tools to manage and inform team members to problems. Most companies in the survey were using homebuilt systems to manage on-call people but the majority planned to move off those tools citing the lack of improvements in those tools compared to new off-the-shelf tools and services. Collaboration is essential to solving problems quickly. The top three ways of dealing with system problems were collaborative in nature, showing that problems can no longer be solved by just a few people, as systems have grown too complex. Scalable ways to bring other people into the resolution process are key to quick resolution. Ways to enable integrated documentation is a trend that is gaining speed. Many respondents stated it was easier to just solve the problem (again) than find the resolution in their documentation. Documentation needs to be delivered alongside the problem with annotation technology. The 2014 State of On-call report was the first of its kind opportunity to capture this sea-change in technology and how it is supported. We intend to continue to track this dynamic part of the technology problem and update this report annually to see how the story changes year to year.
  • 18. Many thanks to the following partners who helped us with this report: This survey was brought to you by your friends at VictorOps. We make being on-call suck less. SEE HOW HERE