Metrics to Power DevOps

1 Copyright ©2014 CollabNet, Inc. All Rights Reserved.
Metrics to Power DevOps
Toward making decisions based on data – not opinion
Sujoy Sen
Sr. Director, Field Engineering
CollabNet, Inc.
Kevin Hancock
Vice President
Practice Head – IT Transformation Consulting

Kevin Hancock, Senior Director Worldwide Field Operations
In his ten plus year at CollabNet, Kevin has led successful implementations of
CollabNet products and services at some of the world’s largest enterprises.
Kevin works with organizations to transform their globally distributed application
development teams into modern, agile organizations, helping to save millions of
dollars while accelerating software delivery. Kevin holds a Bachelor of Science
degree from Carnegie Mellon University, and a Masters degree from the
University of Pittsburgh.
Sujoy Sen, Practice Head – IT Transformation Consulting
Sujoy is part of the leadership team in the Tech Advisory group of TechMahindra.
He is heading a practice which helps customers improve the success rate of their
critical IT projects. Sujoy is a Certified Six Sigma Black Belt & Manager of
Organizational Excellence from American Society for Quality, a PMP, a CISA and
an Agile Trainer & Coach.
Sujoy has more than 19 years of experience across various domains and
functions. He has been part of Tech Mahindra’s IT & Quality consulting group for
the last 10 years.
Today’s Presenters

• Where are we now?
• What’s the problem?
• What do we need?
• How do we get there?
Agenda

4 Copyright ©2014 CollabNet, Inc. All Rights Reserved.4 Copyright ©2014 CollabNet, Inc. All Rights Reserved.
Where are we now?

IT Priorities are changing
McKinsey Global Survey, 2014
• IT is expected to improve business process effectiveness
• IT spending on innovation is expected to increase by 50% and
spending on transaction activities is expected to go down by
15%
• Further current spending on next gen infrastructure is at it
highest in 5 years
• To address the concerns and make effectives use of additional
infrastructure spending, IT should be more agile and responsive
resulting in more innovation
• To enable innovation, a culture of experimentation where IT is
allowed to test new innovations quickly with customer, capture
feedback and continuously improve
• This requires the ability of continuous delivery resulting in better,
faster and cheaper innovation
The focus has shifted from reducing cost to improving
the effectiveness of business processes

Agile Upstream
32%*
Source: February 5, 2014, “How Can You Scale Your Agile Adoption?” Forrester report
*Do standup, burndown/burnup, and
short iterations; product owner; and
constant feedback.
Agile
upstream • Agile Upstream are those
practices usually associated
with planning, managing and
executing a Software project.
• Scrum, Kanban, XP are all
used upstream in the
process.
• But what about
Downstream?

13%†
Source: February 5, 2014, “How Can You Scale Your Agile Adoption?” Forrester report
†Do upstream plus continuous
integration, continuous delivery, and
test-driven development.
Agile
downstream
• Agile Downstream refers to Test
automation (Test-driven
Development). Continuous
Integration, Continuous
Delivery.
• Test automation is a key part of
any Continuous Integration
process
• Successful Continuous
Integration requires Build
Management
• Successful Continuous Delivery
requires Release Management

• Agile Software
Development
• Continuous
Integration
• Continuous Delivery
• Visible and accessible
Metrics
• Access for ALL
Stakeholders (RBAC
permitting)
We need an Agile System

Few Companies Have Achieved Enterprise Agility
Plan (Upstream) Deploy (Downstream)
WorkgroupEnterprise
Perform Upstream Agile:
standup, burndown/burnup,
and short iterations; product
owner; and constant
feedback
13%
<20% A few %
32%
Perform Upstream + Downstream
Agile: Perform upstream plus
continuous integration, continuous
delivery, and test-driven development
Have Scaled Agile
across their organization
Have achieved
Enterprise Agility

What’s the Problem?

• Highly optimized silos or work centers
make sense for Waterfall
– Trust each work center to build the most
“perfect product” to pass to the next work
center
– We know this rarely works
• Agile Enterprises optimize for the
System
– Identify and fix bottlenecks
– Only improve a work center if it improves the
system
Silos of Data and Automation

• The first way:
– Systems Thinking
• The second way:
– Amplify feedback loops
• The third way:
– Culture of continual experimentation and
learning
A model for DevOps Empowerment

The First Way: Systems Thinking
• Focus on value streams
• Think globally
• Understand the system as a whole
• Stop problems early and often
Operations
(customer)
Development
(business)
• Centralised source code and
environment configuration.
• Consistent and automated
release processes.
• Decreased cycle times,
faster release cadence.

• Protect the integrity of the entire system of work, versus
completion of tasks
• Expose visual data so everyone can see how their decisions
affect the entire system
The second way: Amplify Feedback Loops
Operations
(customer)
Development
(business)
• Find and fix defects,
performance and availability
issues faster.
• Enable specialised functions
such as Ops and InfoSec to
collaborate more effectively.
• Increase communication across
the entire business.
• Get more work done

• Continual experimentation
• Tasking risks and learning from failure
• Repetition and practice - pre-requisites to mastery
• Create rituals that reward risk taking
• Introduce faults to increase resilience
The Third way: Culture of continual experimentation and learning
Operations
(customer)
Development
(business)

What Do We Need?

Key Metrics that Power DevOps
1. Change Volume
2. Lead Time (from dev to deploy)
3. Deployment Frequency
4. Percentage of Failed Deployments
5. Build Count By Day
6. File Churn
7. Build duration
8. Cycle time
Traditional agile metrics
have focused here
Enterprise DevOps Metrics
require consistency across
projects, enterprise-wide
visibility, and the ability to drill
to project data and people
Open ALM
Platform

• As an executive, I need to see across projects to determine the
relative success of different teams.
Portfolio Dashboards
Any metric is more valuable when it is put into context, and
compared with other teams.

• How many user stories and new lines of code are being
deployed?
Change Volume
The key is to understand the risk of a given deployment – more user
stories and more lines of code adds risk.

• How long does it take for a new requirement/story/task tp go
from the time when it hits development to when it successfully
gets deployed into production.
Average Age (from development to deployment)
Average age is an important indicator of efficiency in the process – when tracked using
value stream mapping, it can help the team to visualize areas in the process which need
improvement. Cycle time should reduce as the team gets a better hold of the lifecycle.

• How often is the team deploying new code?
Team Deployments
For an Agile team delivering code in 2 or 4-week sprints, this metric should
become predictable, and correlate closely, if not exactly to those sprints.

• On average, how long are builds taking to complete?
Build Duration
This metrics should remain stable over time. Spikes in build duration should be
red-flagged and investigated.

• How is the team trending in terms of cycle time, over the
lifecycle of a project?
Cycle Time
Technical debt will add to cycle time, but so will general complexity. Most
projects start out strong, but over time degrade. Choose the level that is
acceptable and seek stability at that level.

Velocity and Estimation
• How are teams doing? Will they finish what is promised?
Velocity will let us know how much work a team can get done, if they are
consistent in their estimation and improving; while the burnup will let us
know if we are finishing work quicker than it is being added to the queue

How Do We Get There?
The Cloud Framework to Power DevOps

Connect Teams, Processes, and Tools

TechM ADOPT Platform with TeamForge
Your data center
Public clouds
Dashboards
Control center for delivery pipelines, with social activity streams and open pluggable API’s
GitEye
Code Search
Code Review
Git + Gerrit
Jenkins
Hudson
Tracker Discussions Wiki Documents Reports File Releases
JIRA
Selenium
PPM
One Platform, Multiple Possibilities
QTP
Platform 4

Tool Snapshot
Project
Dashboard
Agile Backlogs
Configuration
Management
Unit Test / Code
quality tool
integration
Jenkins
integration
Collaboration
Tools
Code Review
Tool
Customized
Reports
Release
Planning
Defect
Tracker
Defect
Tracker/Impediment
Tracker
Infrastructure
Automation
Traceability for
application lifecycle
activities
Task
Tracking /
Managing
Sprint
Platform 4

Scrum
Team 2
Test (T) QA
(Q)
Scrum
Team 1
Dev (D)
T A Prod
Prod (P)
D
Sprint
Build?
Build?
Unit tests
Unit tests
Unit tests
Build?
Continuous Integration
Continuous Testing
Automate
d Tests
Automate
d Tests
Unit tests
Continuous Deployment
Continuous Release & Monitoring
ContinuousPlanning
Continuous Delivery
Build (B)
Daily Daily
Daily/Release
point
• Devops should create a
flow that results in faster
through put time
• Our approach ensures that
we improve the overall
throughput by bringing in
process, tools, automation
and training interventions
where the flow gets stuck
• TechM proposes a holistic
approach to enable Devops
in an organization by
slowing eliminating legacy
practices which don’t
promote collaboration
Focus on Cadence and Orchestration

Simplified process for continuous delivery
Task
to
Package
Idea
to
task
Package
to Deployment
Business
vision
Products
Identification
Product
Roadmap
Product Epics (product
backlog)
Release Plans
Sprint Plans
(Sprint backlog)
Tasks
Design Code
Automated
Unit test
Automated Code
quality check
Trigger
Integration
Automated
Build
Package
Check-in to
repository
Deploy to
QA Env*
Manual/ Automated
SIT Testing
Deploy to
UAT Env*
Manual / Automated UAT
Testing
Deploy to Production
Env*
S
E
S EStart End
Automated monitoring in
production
I2T
T2P
P2D
Processes
1
Processes
1

Phased Approach
Isolated Teams
SME Shared
Teams
Continuous
Planning
Continuous
Release and
Monitoring
Continuous
Integration
Continuous
Testing
Continuous
Delivery
Product Teams
Multi Layered
Teams
Multidisciplinary
Project Teams
Demand pipeline
established; Release plans
are developed and
maintained for individual
applications; Projects are
able to establish
traceability across
requirements, design,
development, testing and
defect management is
maintained through out
the lifecycle
Projects are able
demonstrate the
behavior of continuous
feedback; as opposed
to a periodic schedule,
builds are triggered by
every commit to the
single code repository
Project is able to align
itself with enterprise
release cadence;
environments are
provisioned ‘on
demand’; releases
deployed across
multiple environments
including data
migration
Continuous regression
reduces the number of
defects; most
integrated bugs are
fixed regularly to avoid
last minute changes;
test cases reused;
‘non-functional’ code
quality measured and
technical debt
improved
Dev and ops team work
together seamlessly
delivering both projects
and managing
operations. Metrics are
captured to monitor
process performance
and continuous
improvement.
Level 1
Level 2
Level 3
Level 4
Level 5
VALUE
COLLABORATION
Capability Model
2
• Our assessment framework is built
on this staged approach. Staged
approach does necessarily mean
that it is a sequence
• TechM would evaluate organization
as-is status against these stages
• As-is assessment would cover
people, process, tools, policies, KPIs
and flow
• As an outcome of the assessment
we would provide the gaps and the
transformation roadmap to cover
these gaps
• The Devops consultant/ Program
manager would work towards
implementing this transformation
roadmap

Transformation Phases

iSPEED tm Methodology
Study Plan Enable and Expand Diffuse
• Review historical data,
documents, tools and processes
• Understand the organization
structure
• Interview key stakeholders
using diagnostic tool
• Prepare business case
• Identify multiple solutions
blueprints
• Prepare transformation
roadmap
• Identify Enterprise Change
Enablement team
• Develop workflows as per
selected blueprint
• Review the workflows with all
stakeholders
• Tune up workflows
• Develop detailed process,
templates, checklists, guidelines
• Modify tool workflows
• Facilitate deployment planning
workshops, prioritize process
activities , prepare deployment
plans
• Enable teams by training and
support
• Identify projects and deploy
• Aggregate data
• Enable Continuous
Improvement
• Quantify benefits and Showcase
projects
• Collate and analyze
performance data
• Benchmark and improve
• Repot
• Establish improvement log
• Continue deployment support
Interview
• Interview key
customer
stakeholders
• Understand customer
context
• Abstract key business
drivers
• Enlist the top
priorities
2 to 3 WKS 4 to 6 WKS 8 to 12 WKS OngoingMax 4hrs
Feasibility Decision
• Business case
• Solution blueprint
• Transformation roadmap
• Process Workflows
• Processes, templates, checklist
• Trainings documents
• Project data
• Audit & Management reports
• Management reports
• Improvement log
• Fixed Price
• Fixed Price
• Time and Hire
• Outcome based pricing
• Fixed Price
• Time and Hire
• Outcome based pricing
• Time and Hire
Deliverables
Engagement model

DevOps Transformation Assessment (DOT)
ADCM provides a structured and tested
approach for any organizations Devops
strategy. Outcome of the assessment
provides a roadmap that can be used for
Devops strategy and rollout

DOT Analytics
0
0.5
1
1.5
2
2.5
3
3.5
4
Business IT
Alignment
Portfolio planning
Release planning
Capability planning
Capacity planning
Infrastructure
planning
Agile delivery
planning
Task planning
Value addition
Collaboration
Institutionalization
• Each stage has multiple focus areas.
The depiction here is for continuous
planning
• To arrive at the score, TechM
consultants would speak to various
stakeholders and take them through a
series of questions
• Each focus area is evaluates for
• the value addition that it is
currently bringing,
• The collaboration it facilitates
• Level of implementation that
has been achieved
• Focus areas where today nothing exists
is given a score of zero
• Further for each area a detailed
improvement plan is prepared
Continuous planning

DOT Roadmap Sample
Continuous planning W1 W2 W3 W4 W5 W6 W7
Business IT
alignment
People
Process
Tooling
Flow & integration
Performance & outcomes
Deliverables
Portfolio
Planning
People
Process
Tooling
Flow & integration
Deliverables
Release
Planning
People
Process
Tooling
Flow & integration
Deliverables
Capability
Planning
People
Process
Tooling
Flow & integration
Deliverables
Capacity
Planning
People
Process
Tooling
Flow & integration
Deliverables
Infrastructure planning
People
Process
Tooling
Flow & integration
Deliverables
Agile delivery planning
People
Process
Tooling
Flow & integration
Deliverables
Task
planning
People
Process
Tooling
Flow & integration
Deliverables

Thank You
Contact Details:
Kevin Hancock - CollabNet
kevin@collab.net
Sujoy Sen – Tech Mahindra
Sujoy.Sen@TechMahindra.com

© 2014 CollabNet, Inc., All rights reserved. CollabNet is a
trademark or registered trademark of CollabNet Inc., in the US
and other countries. All other trademarks, brand names, or
product names belong to their respective holders.
CollabNet, Inc.
8000 Marina Blvd., Suite 600
Brisbane, CA 94005
www.collab.net
+1-650-228-2500
+1-888-778-9793
blogs.collab.net
twitter.com/collabnet
www.facebook.com/collabnet
www.linkedin.com/company/collabnet-inc

Metrics to Power DevOps

Empfohlen

Empfohlen

Weitere ähnliche Inhalte

Was ist angesagt?

Was ist angesagt? (20)

Andere mochten auch

Andere mochten auch (20)

Ähnlich wie Metrics to Power DevOps

Ähnlich wie Metrics to Power DevOps (20)

Mehr von CollabNet

Mehr von CollabNet (20)

Kürzlich hochgeladen

Kürzlich hochgeladen (20)

Metrics to Power DevOps

Hinweis der Redaktion