Agile Islands 2020 - Dashboards and Culture

@spoole167@spoole167
Dashboards and Culture: How
Openness changes your
Behavior
Steve Poole, IBM

Our industry is constructed of silos

@spoole167@spoole167Two of the biggest ones ..

Between Developers and IT teams

And for a long time – its been a problem we’ve lived with

This talk is a story about how tackling the
communications gap made things better
• How it changed the behavior of the people involved..
• How you too can take a leap and leave your silo

About me
Steve Poole
IBM Lead Engineer / Developer advocate
JVM Developer
Open Source Advocate
DevOps Practitioner (whatever that means!)
Driving Change

Once upon a time..

Modernising an IT group
I took over a large dev focused European IT team
that provided development services to developers
A team that would have been classified as “Slow IT”.
My job - modernise the team.
Support “Fast IT” for cloud development teams
I learnt a lot.
Then I talked to customers etc. I learnt even more.

I learnt - executives are still
asking:

Compliance
Resilience, security
IT Dev
I learnt IT and Dev don’t communicate: they just shout at each other in their own
language

I understood – we just don’t share a common experience

IT Dev
I learnt – making IT teams read the Phoenix Project can be counter productive
“I liked the book up and ‘til the part
where they started to modernise”
“So you want me to put
myself out of a job?”

DevOps is about a different
contract between Dev &
Ops

IT
A contract realised in new forms of self-service
assets like PaaS or IaaS , Kubernetes, Docker etc
A contract that covers new availability
requirements & speed to market

And like all good contracts it needs
to be written down in a mutually
agreeable language..

Like a Dashboard

What is a dashboard?

@spoole167@spoole167https://commons.wikimedia.org/wiki/File%3ANuclear_Ship_Savannah_-_Reactor_Control_Room_-_Center_and_Left_Panels.jpg
Reactor Control ?

@spoole167@spoole167https://www.flickr.com/photos/johnseb/ Bletchley Park,
Code Breaking ?
Slido #K100

https://www.flickr.com/photos/nrmadriversseat/
Car Dashboard?

A ‘real’ dashboard
provides
Just enough status : speed
Just enough insight: fuel / range
Just enough warnings: seat belt
Clear emergency indicators

Back to my story

I really hate it when
someone tells me I have
a system that is down
before I notice.

I really hate it when someone tells me I have
a system that is down before I notice.
Makes me feel like this

I challenged my monitoring team to do all
that it took so that my group was always…

“First to Know”

We could have built a complicated system

Simple technologies
https://prometheus.io/
But even these have
a high learning curve

Start really simple http://dashing.io/

Within a year everyone has a dashboard
• Every dev team
• Every support team
• Even managers
• And Executives..
In fact we build a network of
monitors with raspberry pi’s
To run the dashboard systems.
And they are monitored too..
Now over 4 million
dashboard events per day

And yes we built a sophisticated
monitoring solution

MQSeries 7.5 (MQTT)
Event
Store
Tivoli
Monitoring
Nagios
Big Brother
Monitoring
Backups
SAN
API's
Custom Scripts
RTC
GitHub
SCM’s
Dashboards
(IBM Cloud)
Continuous Delivery
 Build, test and deploy within 2
minutes
 Averaging 8-10 deployments a
day
Data
Sources
Insight
But that’s not the point

Why did simple dashboards go viral?
• Every dev team
• Every support team
• Even managers
• And Executives..
Now over 4 million
dashboard events per day

Because of the method for defining them

Case Studies
Executive Dashboards
Availability Dashboards
Developer Dashboards
Unusual Dashboards

Executive Dashboards

@spoole167@spoole167 https://www.flickr.com/photos/ctsi-global/

A battle ground
Executives want ’levers’ to pull
Endless reinvention as they look for new insights..
Middle management want to control the message
Reduce the time spent ‘wasted’ on responding
Since (the elephant in the room) we all know the data
is inaccurate, old and any predictions mostly useless

Exec: Reduce IT spend
Make devs more productive
New Executive Dashboard
Display progress in moving local workload into the cloud
Development assertion This is much harder
than it looks. We don’t have the data AND
it will cost us more in the end

New Executive Dashboard
IT Challenges included:
1. Not being able to associate local workloads with the correct teams
2. Not being able to find all the servers
3. Not being able to accurately determine performance characteristics
• And dev workload owners expected to end up with a worse
environment so dragged their feet in helping

When you have to distill a common goal into
a dashboard
Just enough status
Just enough insight
Just enough
Clear emergency indicators

You find common ground together
• 1 person from IT worked with development
teams to get their data reclassified
• 1 person is not a threat
• 1 person makes a human connection
• 1 person asked ‘why’ , ‘what do you want’
0
2
4
6
8
10
12
Team 1 Team 2 Team 3 Team 4
Un assessed On Prem Units Able to move moved

Unexpected benefits
• Better predictions of cost of moving workloads to cloud +
classification = each team can realistically discuss the cost/benefit of
moving their workloads to the cloud with Executives
• Managers can justify budget increases more effectively –they can
show how their existing capacity is being used.
• IT support now have an improved understanding of what their clients
are using machines for.

All teams now want and need
the data to be accurate and
visible
•Which lead to improved
communications
between teams and
reduced the shouting

Availability Dashboards
Or how sharing brings understanding

Service Availability – a tale of impedance missmatch
• How do you calculate service availability?
• What’s your unit of time?
• What’s your availability period - 9x5, 24x5,
24x7?
• For SaaS products it is 24x7 ( minus very rare
scheduled downtimes)
• But what does available mean?

Development IT team
• Provides a critical service to developers
• Asked to provide 24x7 support
• Dev IT: “we already do that”
• Dev teams beg to differ – service has regular downtimes..
• Dev IT : “Our availability calculations don’t have to
include regular scheduled downtimes”
• Dev (and me) WTF?
• what about all these other unexpected downtimes?
• Dev IT : “They are caused by events beyond our control
and so are not included”
• Dev (and me) WTF?

Insight - Iceberg syndrome
• Many IT groups revolve around the SME
• Items move from one to another to get
completed so very few people have a vision of
an IT service as perceived by the end user.
• Published data is more for the service
providers use than the end user
• End users think they are dealing with an IT
department – but in reality they are dealing
with a small handful of people who have
responsibility for a part of the service.

Solution
• Have the IT team understand they are
responsible for reporting the availability of
their service whatever the cause of
downtime.
• Have the IT team spend time with their users
to see what can be done to mitigate the
impact of scheduled downtimes
• Defined what available meant (hint not just
pingable)

Culture changes?
• Creating a better definition of ‘available’
meant the IT team learnt what was important
to their end user
• Dashboards with ‘real’ status means that end
users don’t have to try to get answers via
ticket or phone - less stress. More trust.
• end users see the truth and can trust the IT
team more.

Developer Dashboards
Or how you can never share enough

Unusual Dashboards
Or how you can never have enough of a good thing

Build Server upgrades
• A chore.
• Several hundred machines needed urgent security patching
• mostly by hand! ( a long story)
• How do you keep focus and motivate the teams who are doing this?
https://commons.wikimedia.org/wiki/File:US_Navy_090716-N-8590G-011_Members_of_the_damage_control_team_of_the_guided-
missile_frigate_USS_John_L._Hall_(FFG_32)_compete_in_the_pipe-patching_event_of_the_Damage_Control_Olympics.jpg
Navy Pipe Patching event

You make it a competition
Navy Pipe Patching event
Team A
Team B
Team C
Team D
Team E

Diagnostic ’one-off’ dashboard
Jenkins build server problems..

Continuous delivery code/complexity metrics
Red indicates over complex code.
Radius indicates number of lines of code

Did I get my ”First to
know”?

Sort of
• Most teams now look at their dashboards
instead of monitoring emails, tickets etc.
• So do our clients
• So it’s more that we know together

But most importantly..
Our customers know we know.
They understand how we work and they can see us working
Mutual trust through openness..
Teams feel more empowered, more in control of their work
Know they are doing the right thing

So what have we learned?
That people
do change
1
Change comes
from mutual
understanding
2
By sharing
openly you’ll
learn more
3
All from moving
a little out of
our comfort
zones.
4

Thank you

Agile Islands 2020 - Dashboards and Culture

Recommended

Recommended

More Related Content

What's hot

What's hot (9)

Similar to Agile Islands 2020 - Dashboards and Culture

Similar to Agile Islands 2020 - Dashboards and Culture (20)

More from Steve Poole

More from Steve Poole (19)

Recently uploaded

Recently uploaded (20)

Agile Islands 2020 - Dashboards and Culture

Editor's Notes