The document discusses emerging trends in DevOps in the healthcare industry. It begins with an overview of the current state of DevOps adoption in healthcare, which has grown steadily from 2017-2021. However, many healthcare organizations are still only in the middle stages of DevOps maturity. The document then examines how standardization, metrics, and a DevOps center of excellence can help organizations advance in their DevOps journey. It outlines a case study of how to scale DevOps practices through the use of internal platform teams. The document concludes by discussing how artificial intelligence and machine learning can be leveraged to augment DevOps practices across development, integration, testing, monitoring, and operations.
BSides Seattle 2024 - Stopping Ethan Hunt From Taking Your Data.pptx
Dev ops
1. Emerging Trends in DevOps
The prescription for accelerated speed-to-market,
effective patient care and greater collaboration in
Healthcare IT
January 2022
Prepared & Presented by:
Megha Sinha
2. 2
1Current State of DevOps in the Healthcare Industry
DevOps Maturity Assessment and Adoption Curve
2
The future of DevOps – Revolutionizing with ML & AI
4
AGENDA
3 Impact of Standardization and Metrics in DevOps
Maturity
3. 3
Current State of DevOps in the
Healthcare Sector
- Where do we stand vis-à-vis other evolved sectors
4. 4
Study of the state of DevOps adoption revealed healthy growth during the period
2017 – 2021
BUSINESS PERFORMANCE
43% of organizations, have seen their business
perform above or in line with expectations.
SPEED-TO-MARKET
51% of Healthcare IT teams deploy one or more
database changes compared to 47% in other
sectors
ADOPTION
73% of those in Healthcare have now adopted
DevOps
CLOUD ADOPTION
Healthcare IT has witnessed a rise in cloud
adoption from 42% in 2019 to 55% in 2021
20 18 15 13 11
33
30
27
18
15
35
39
43
50
48
12 13 15
18
26
0
20
40
60
80
100
120
2017 2018 2019 2020 2021
Growth of DevOps in Healthcare IT
Not Adopted Plan to Adopt Adopted on Some Projects Adopted on All Projects
Source:https://www.red-gate.com/solutions/database-devops/entrypage/state-database-devops-healthcare-2021,
5. 5
The healthy growth of DevOps led to positive implications on the state of healthcare
despite a pandemic-struck world
AI/ML driven
critical insights
Regulatory compliance
and security
Data-led improved
patient care
Detect and analyze patterns across
data in real-time to identify health
conditions, potential treatment
methods, and their outcomes-and
take life-saving decisions at speed
DevOps’ core principle, “infrastructure
as code,” vulnerability analysis,
implementation with principle of “least
privilege” allows healthcare providers
to integrate security fundamentals
inside the application module itself.
Self-check-in systems, electronic
prescription filling, and automatic
appointments built with CI/CD
pipelines address patients’ needs
for convenient and quality
healthcare.
Disruptive Innovation in
Healthcare
A wristband that can read your
mind, Cancer-diagnosing artificial
intelligence, Rehab in virtual
reality, Drone-delivered medical
supplies have all been possible
due to the integration of cutting-
edge technology, speed and scale
of operations. DevOps as a
practice plays a critical role in this
integration
Improved Infrastructure
Efficiency
Event-driven serverless architecture
allows developers to leverage only the
resources they need for the app or
microservice being developed. The
utility consumption of resources
employed when leveraging a
healthcare cloud provider (AWS or
Azure) on their HIPAA-compliant
architecture offers ways to manage
resources efficiently.
Reduced Operational
Expenses (OPEX)
A multinational US-based conglomerate
with more than 80,000 employees and
$30B in annual revenue designed a
DevOps-as-a-Service offering. The
proposed solution cut their OPEX by
45% while ensuring high availability of
the products, full regulatory compliance
of operations, integration of new
products and features, as well as ML
and Big Data analytics services.
6. 6
Which leads us to the most burning question –
‘How to define a successful DevOps Strategy?’
DevOps is not just
Automation
DevOps is not
the Cloud
DevOps requires
team identities and
clear interaction
paradigms
Presence of
Platform team is
key to achieving
DevOps at Scale
Risk taking, fast
flow optimization
and efficient
feedback loops
are required
CALMS Framework
7. 7
So, What has Worked Well and What Needs to Change in the Healthcare IT DevOps
Strategy?
C - Culture
L - Lean IT
A - Automation
S - Sharing
M - Measurement
58%
47%
29%
29%
74%
55%
55%
51%
22%
22%
73%
56%
0% 10% 20% 30% 40% 50% 60% 70% 80%
Adoption of Cloud Infrastructure
Frequency of deployments
Automation of Builds, Testing, and Deployment
Version Control
Culture of Adoption
Teamwork and Collaboration
Healthcare
Other Sectors
• DevOps adoption rates have increased
• 80 % of respondent companies stuck in the middle level of DevOps Maturity journeys has remained stagnant.
• There is a wide-gap between the have’s and have-not (team) in the same organization.
Source: Puppet State of DevOps Report 2021
9. 9
Let us take a case-based approach to delve further into the conundrums of requisite
moves to emerge successful in streamlining IT development and delivery
Background Challenges
What should they do?
A leading US healthcare
payer with a large IT
structure adopted agile and
DevOps practices to gain
true business value through
faster feature releases,
better service quality and
cost optimization. Teams
achieved initial success
with their Pilots on CI/CD
pipeline automation
strategy, monitoring,
alerting and logging.
1. Teams that have had success on pilot projects are being
inundated with requests for new projects, applications, and
infrastructure.
2. The successful team are fully utilized on day-to-day projects
and business requests with little capacity for strategic
planning, architecture improvement, process maturity and
technical debt efforts.
3. Attempts to guide the adoption of DevOps solutions across
all teams have not been fruitful.
4. The pilot projects are also looking for better recipes for a
Future-Ready DevOps Solution
10. 10
So, what should the organization do?
• With only so many hours in a day, existing staff is not likely to
be able to take on more work; the only option is to expand that
capability, but how?
• With good understanding of the DevOps Culture, how should
the organization design a Roadmap for Pilot-to-Scale?
• What Arsenals do the leadership teams need to achieve
success?
11. 11
Industry leading DevOps bodies prescribe Maturity Assessment Models to map
organizational Current-state and Future-State for success-at-scale
1. Leverage DASA* DevOps Maturity Assessment Framework to evaluate capabilities against current practices, DevOps goals, and business model.
2. Document current state analysis and identified gaps within IT infrastructure and operations, application development and service desk.
3. Implement solutions to plug the identified gaps.
DASA Maturity Model Source: https://drive.google.com/file/d/0B2mJR_0L0FroWnc5WU9VdTljSkk/view?resourcekey=0-LE-_-HmJjMpMGOYBuh1eSg
SOLUTION
0
1
2
3
4
5
Courage
Teambuilding
DevOps Leadership
Continuous
Improvement
Infrastructure
Engineering
Security, Risk,
Compliance
Continuous Delivery
Programming
Test Specification
Business Analysis
Business Value
Optimization
Architecture and
Design
Maturity Scores (Maximum Score - 5)
Results
Journey
Mapping
*
*
*
*
*
*
*
*
* *
*
*
Knowledge Area
* Skill Area
*
Level 1
Novice
Level 2
Competent
Level 3
Proficient
Level 4
Expert
Level 5
Master
DevOps maturity Score – 3.1
Current State
Phase 1
Phase 2
The detailed model here explains what will the final-state
look like.
12. 12
Further Analysis revealed that success limited to few teams was not a failure of DevOps
Strategy but the emergence of new or advanced hurdles
• Organizations stuck in the
Mid-level of DevOps
Maturity and Adoption
• DevOps as an approach is
followed in various teams
but lacks centralized
standardization
• Customer experience is
inconsistent
• Every technical team
implements security and
risk regulations but there is
no centralized monitoring
platform available to track
such implementations
• Not all teams speak the
same language in DevOps
Cause and Effect Analysis of the DevOps stagnation in healthcare enterprises
Culture
Automation
Lean IT
Measurement
Sharing
• Absence of platform team
model
• Lack of defined Interaction
paradigms
• Excessive cognitive
load * on delivery
teams
*Cognitive load examples - good practices,
automation, and support that negatively impact
performance metrics
• DevOps approaches over focussed on the
hyper care and feeding of CI/CD Pipelines
• Most of the Repetitive tasks are automated,
but no real optimization for the team
• Absence of self-service capabilities and
shortage of skills
• Teams have defined their own
standards for CI/CD pipeline and
automation metrics
• Absence of standardized usage of
tools, frameworks and processes
• Absence of metrics to measure
effectiveness of ‘teams of teams’
working model
• Infrastructure topologies are not designed
for promoting fast flow of change
• Absence of the culture to map customer
value stream to CI/CD pipelines designed
• Failure to create cultures of knowledge
sharing and best practices
• Unclear responsibilities and aversion to
risk
• DevOps is promoted passively.
13. 13
What do Metrics and Standardization
bring to the table?
- Why are some organizations able to evolve, while the vast majority are stuck in the middle?.
14. 14
Shifting focus on the Solution, Let us explore ‘How’ should we move ahead in the
Journey?
Stop talking about
culture, start doing
stuff
01
Scale DevOps practices
with internal platform
teams
02
Focus on Integrating
security fully into the
software delivery process
03
• Build a ‘Team of Teams’
• Devise clear team definition and
interaction paradigms
• Encourage employee involvement in
the change management process
• Enable self-service offerings for developers • Provide self-service capabilities for
security and compliance validation
• Improve abilities to quickly remediate
critical vulnerabilities
• Reduce complexity and bring
standardization in the technology stack
• Define organization-wide coherent
measurement criteria
15. 15
CALMS - Culture & Change Management
Launched DevOps Center of
Excellence as an enterprise entity
to lead and govern DevOps
1 CoE topology was defined
2
Clear Interaction paradigms were
defined and enforced
3
Centralized Platform was designed and launched as a
deliverable from Step 3 for DevOps, Measurement
dashboard with uniform metrics, tools and architecture
patterns were introduced by respective teams
4 Regular reviews, organization-wide Kx and team report-outs were
formally launched, tracked and improved to ensure success of the
Platform.
5
DevOps CoE
Stream
Aligned
Team
Enabling
Team
Complicated
Sub-system
Team
Platform
Team
2 Members
each from
different
business
stream/function
Representation
from IT experts
for each
business
stream/function
A cross-
functional team
of DevOps
SMEs -
Team to configure,
develop, and
support a
centralized DevOps
Platform/application
Collaboration
Facilitation
X-as-a-Service
Working together for a defined period to discover new things
(APIs, practices, technologies, etc.).
One team helps and mentors another team.
One team provides, one team consumes something
“as a Service.”
16. 16
CALMS - Automation and Lean IT
The platform has a collection of
DevOps toolchain that is managed
as a product to support delivery
team customers
1
The toolchain is co-created and
continually reviewed with stream aligned
team and complicated sub-system to
ensure value and relevance
2
The product manager of the platform
teams actively markets the toolchain
to drive adoption across the
organization
3
Suggestive
Continually monitor the Platform toolchain
adoption by measuring the following metrics
Net Promoter Score (i.e., the willingness of
delivery teams to recommend the product to
others).
Percentage of delivery teams using the product.
Percentage decrease in feature cycle time.
• 28-day rolling and/or seven-day rolling uptime
• Incident frequency
• Mean investigation time per incident (e.g., how
long it takes to find out
what went wrong)
• Percentage of services or capabilities with
vulnerability-patching SLAs
• Cost per work unit
• Developer throughput rate
• Deployment rate
• Rollback rate
• Conformance metrics (how close a service is
to using “paved roads,” i.e.,
standards provided by the platform team)
17. 17
CALMS – Measurement
The DevOps CoE defined organization-wide coherent measurement criteria to measure the effectiveness of toolchain in the
platform
DEV/CI QA DEPLOY RELEASE OPERATE
Development
Lead Time
Rework required by
defects, build
breakage, downtime
Idle time
Work-in-progress
technical debt
Cycle time
Idle Time
Defects
discovered/escaped
, impact of defects
MTTD
Deployment Lead
Time
Deployment
frequency,
duration
Change Success
rate
MTTR
Release
frequency, %
automated
Time/cost per
release
Predictability
planned vs Actual
MTTR
Cost/ frequency
of outage
On-call after
business hours
Performance/
utilization
--Cycle Time--
--Visibility--
--Scale--
18. 18
CALMS – Sharing
The DevOps CoE Set up Communities of Practice, centralized Knowledge exchange repositories and drove democratization initiatives
Role Responsibilities
Community
Leader(s)
Community
Facilitator(s)
Community
Advocate(s)
• Manage the ongoing health and direction of the CoP.
• Track and encourage member participation.
• Communicate CoP results and success stories to
the DevOps CoE Lead
• Plan and facilitate CoP events.
• Collect member feedback and support the
community leader in setting the CoP agenda and
direction.
• Create formal and informal connections between
CoP members.
• Socialize and promote the CoP
• Monthly Learning Sessions
• Weekly mailers to share automation patterns - deployment
automation, test automation, anything another team might
find useful.
• Actively market the toolchain by making the tools easy to
find and implement,
• Mailers highlighting how the toolchain enables delivery
teams to achieve their objectives more quickly and easily.
• Quarterly Release of Technology Radar representing new
DevOps tools added both in the trial phase, established
phase and tools being phased out.
Activities carried out in Phase 1
19. 19
A collective summary of the benefits and progress made along the DevOps Maturity Curve
Benefits Realized
0.00%
10.00%
20.00%
30.00%
40.00%
50.00%
60.00%
70.00%
80.00%
Before After
0.00%
10.00%
20.00%
30.00%
40.00%
50.00%
60.00%
70.00%
80.00%
Before After
% of self-service interfaces % of improvement in DevOps
processes
20. 20
The future of DevOps –
Revolutionizing with ML & AI
- Why are some organizations able to evolve, while the vast majority are stuck in the middle?.
21. 21
AIOps technologies will enable intelligent automation, reduce toil in IT
operations and increase adoption of AI-augmented DevOps practices.
By 2025, leaders must shift their focus from building automated systems to instead deliver autonomous systems
Descriptive and Diagnostic Predictive and Prescriptive
2020 2025
Deploy
Support Monitor
AI-Enabled
Systems
• Scale support with prescriptive
analytics
• Wrap incident alerts with
contextual insights
• Triage and deduplicate alerts
• Train AI models to learn from
human interventions to automate
future root cause analysis
• Assess risk score of builds
• Instrument apps to gather
telemetry to train AI models
• Optimize cloud workloads for cost
and performance
• Automate “measure-interpret-act”
functions
• Go from exception-based monitoring to
continuous monitoring
• Correlate insights from multiple data
sources
22. 22
How should organizations prepare themselves to embrace the future?
By 2025, leaders must shift their focus from building automated systems to instead deliver autonomous systems
Step 1: Identify the Current Challenges with the evolution of Digital systems and apps
Step 2: Assess if these challenges are caused by overwhelming amounts of data generation and can be solved by AI
Step 3: Take Steps to Identify how to make your DevOps AI-driven
Adopting
Advanced APIs
Identifying
Related Models
Parallel Pipeline
Pre-trained
Model
Public Data Due Identity
Broaden
Horizons
What these mean?
23. 23
Where should we utilize AI and ML to augment DevOps
A stage-wise use case and impact of leveraging AI and ML in DevOps
Development
Integration
Testing
Monitoring
Feedback
Deployment
Operations
Addressing Emergencies - ML can deal with sudden alerts by training systems continuously on identifying repeating patterns
and inadequate warnings, thus filtering the process of sudden alerts.
Business Assessment - DevOps pays high regard to understand of code release for achieving business goals, ML tools deal
that with its pattern-based functionality by analyzing user metrics and alert the concerned business teams and coders in case of
any issue.
Early Detection - ML can create key patterns that can analyze and also predict user behavior, such as analyzing configuration to
meet the expected performance levels, response rate and detect an issue at an early stage
Triage Analytics - ML tools can help you identify issues in general processing and also manage release logs to create coordination
with new deployments.
Dealing with Production Cycles - ML helps to understand and analyze resource utilization, among other patterns to detect
possibilities of abnormal patterns such as memory leaks
Quality Check - ML performs efficient review Quality Assessment results and builds test pattern library based on discovery
Application Progress - Application of ML on DevOps tools helps to addresses the irregularities around code volumes, long build
time, delays in code check-ins, slow-release rate, improper resourcing and process slowdown, among others
24. 24
End-to-End Functional View of Intelligent DevOps
Collect data
Source: https://www.wipro.com/infrastructure/the-power-of-ai-in-devops/
25. 25
Looking forward
➢ The disruption in the consumption and subscription of services across all domains has led to
intelligent ways of enabling various platforms to get connected with the end users.
➢ The momentum in DevOps will be led by how historical and transactional data is intelligently used
for faster product releases
➢ The change, on one hand, and stability, on the other will help the industry to adapt to other
technological disruptions in near future
➢ Managing DevOps platforms with a product mindset and platform team approach brings benefits
30. 30
•Adopting Advanced APIs: Moving development teams to gain hands-on experience in working with canned APIs like Azure, AWS and GCP
that allow the deployment of robust AI/ML capabilities into their software without having to create self-developed models. Further, they can focus
on integrating add-ons such as voice-to-text and other advanced patterns.
•Identifying Related Models: The next step after the above would be identifying similar AI/ML APIs. Doing this, development becomes easier
with successful ML/AI model deployments and individual teams can work on further enhancements and apply the same to additional use cases.
•Parallel Pipeline: Given the fact that AI and ML are at the experimentation stage, it will be important to also consider running parallel pipelines
so that things won’t go bad in case of any failure or sudden halts. The better way to deal with would be adding ML/AI capabilities in a step-wise
manner, gradually in line with the projects’ progress avoiding significant delays.
•Pre-trained Model: A well-documented, pre-trained model can drastically cut down the threshold for adoption of ML and AI capabilities. A pre-
trained model can be helpful in recognizing user behavior or inputs in a specific search. If it can at least match the basic aspects of the user
search pattern, further add-ons to it can yield better results that can fully match with the user behavior pattern. So, having a pre-trained model is
key to AI/ML adoption at an initial phase.
•Public Data: Finding the initial training data is a key challenge in adopting AI/ML. None would actually feed this data. So, where will you drive
information from? That’s where you require public data sets. It may not exactly meet your full requirements but can at least fill the gaps to
enhance project viability.
•Due Identity: One starts witnessing the true potential of ML/AI only after the software runs and shows completion at a high rate, quality and
performance compared to the traditional approach. So, it will be important for any organization to identify and forward the success stories that
they see out of AI/ML adoption to further teams keeping them updated.
•Broaden Horizons: Developers should ideally be in continuous quest of knowing new and staying updated. This applies more to AI/ML use
cases. For this, organizations should encourage teams by easing their ways to access MI/AL sandboxes and general-purpose APIs without
additional formalities that come as part of the corporate procurement process.
AI Readiness of DevOps Systems