My INSURER PTE LTD - Insurtech Innovation Award 2024
Using Machine Learning to Optimize DevOps Practices
1. Using Machine Learning to Optimize
DevOps Practices
Building Learning into Monitoring and Feedback
Peter Varhol
2. About me
• International speaker and writer
• Degrees in Math, CS, Psychology
• Technology communicator
• Former university professor, tech journalist
• Cat owner and distance runner
• peter@petervarhol.com
3. Agenda
• What is machine learning?
• How is machine learning applied to DevOps?
• Challenges in training these systems
• What constitutes an issue?
• Summary and conclusions
4. What is Machine Learning?
• Layered algorithms that change parameters based on feedback
from know data
• Can be linear or nonlinear
• Algorithms can be fixed in production or adaptive
• Fixed – algorithms do not adjust once deployed
• Adaptive – algorithms continually adjust to new data
• Usually part of a larger system
5. Adaptive Systems
• Airline pricing
• Ticket prices change three times a day based on demand
• It can cost less to go farther
• It can cost less later
• Ecommerce systems
• Recommendations try to discern what else you might want
• Can I incentivize you to fill up the plane?
6. Why Use Adaptive?
• The “right” result will vary over time
• Trying to optimize a particular result
• Revenue
• The problem domain is not static
Confidential, Dynatrace LLC
7. How Are Fixed Systems Used?
• Transportation
• Self-driving cars
• Aircraft/Drones
• Ecommerce
• Recommendation engines
• Medical
• Diagnosis systems
8. Why Use Fixed Machine Learning Systems
• The problem domain is static
• The expectations remain constant
• The right answer is known under most conditions
• The original algorithms remain valid over a long period of time
9. DevOps Practices Generate Data
• During development
• Agile metrics, JIRA issues, test case metrics
• During continuous integration
• System test metrics
• During continuous deployment
• Quality metrics for deployments
• After deployment and into production
• Application availability and performance
• Usage log files
10. Focus on Monitoring
• Ongoing data on availability and performance
• RUM
• Synthetic tests
• Application monitoring
• Monitoring tackles the back end of DevOps
• Identifying unhealthy trends
• Diagnoses failures and poor performance
• Recommends action
• Fixed or adaptive depends on your goals
11. Where Do Predictive Analytics Come In?
• Big data makes possible predictions of future events
• Are we going to fail?
• How will we perform with traffic surges?
• As well as past events
• What went wrong and how do we fix it
• We can rely on past data
• Adaptive systems may not perform as well
• Clear goals needed
12. What Technologies Are Involved?
• Neural networks
• Genetic algorithms
• Rules engines
13. Neural Networks
• Set of layered algorithms whose variables can be
adjusted via a learning process
• The learning process involves training with
known inputs and outputs
• The algorithms adjust coefficients to converge on
the correct answer (or not)
• You freeze the algorithms and coefficients, and
deploy
• Or you optimize on a particular set of characteristics
15. Genetic Algorithms
• Use the principle of natural selection
• Create a range of possible solutions
• Try out each of them
• Choose and combine two of the better
alternatives
• Rinse and repeat as necessary
16. Bringing in DevOps
• DevOps has data that can be used to train neural networks
• Health of the application
• Trends in application traffic and responsiveness
• Application failure
17. Machine Learning Helps DevOps
• Decisions are complex
• Why is the CPU maxed?
• What is causing disk thrashing?
• Why did the network slow?
• Why did the application fail?
• Data is massive
• Potentially thousands of data points a day
18. How Good Are Decisions?
• Expert versus machine
• Given the same data
• In many domains they tie
• With additional data, the human can be better
• But machine learning will get better
• But only as good as the data
19. We Want to Do Two Things
• Identify trends that may indicate future problems
• Increasing response times
• More page errors
• Diagnose faults once they have happened
• Why did the application fail?
• How can we fix it as quickly as possible?
20. Fixed Algorithms Work for Some Problems
• Immediate performance and failure identification
• Diagnosis of failures and performance issues
• These are readily identifiable from known data
21. Adaptive Systems Supplement These Tools
• Predictions of future events
• Performance
• Availability
• The target is moving
• So we need current data to adjust the algorithms
22. The Machine Helps the DevOps Expert
• The machine learning app provides:
• Early warning on possible performance issues and failures
• Immediate notification of failure or impending failure
• Trend analysis of data to predict unhealthy outcomes
• The machine learning is an assistant
• It can’t fix anything
• It can’t necessarily identify the root cause
23. What is the Goal?
• We have many ways of monitoring
• Many of them are represented at this conference
• Each measures something a little different
• Latency, response time, availability, network, DNS . . .
• Too much data can be no better than no data at all
• Machine learning can correlate across
measurements
• Focus to eliminate false positives
24. Intelligent Systems Are Sometimes Wrong
• The problem domain is ambiguous
• There is no single “right” answer
• “Close enough” is good
• We don’t know quite why the software
responds as it does
• We can’t easily trace code paths
25. Testing Machine Learning Systems
• Have objective acceptance criteria
• Test with new data
• Don’t count on all results being accurate
• Understand the architecture of the network as a part of
the testing process
• Communicate the level of confidence you have in the
results to management and users
26. A Cautionary Tale
• All events are not created equal
• AI systems treat events equally
• A failure of a system during busy season is the same as any other
• DevOps pros know otherwise
• And can exert additional effort in response
• And actually fix the problem
• We can’t automate what we don’t understand
• You need the human in the loop
Confidential, Dynatrace LLC
27. Conclusions
• DevOps is a natural environment for machine learning
systems
• Any activity that generates data and requires a decision is fair game
• Monitoring is low-hanging fruit
• Fixed systems for failure and diagnosis, adaptive for trend
analysis
Confidential, Dynatrace LLC
These types of software are becoming increasingly common, in areas such as ecommerce, public transportation, automotive, finance, and computer networks. They have the potential to make decisions given sufficiently well-defined inputs and goals. In some instances, they are characterized as artificial intelligence, in that they seemingly make decisions that were once the purview of a human user or operator.
Most machine learning systems are based on neural networks. A neural network is a set of layered algorithms whose variables can be adjusted via a learning process. The learning process involves using known data inputs to create outputs that are then compared with known results. When the algorithms reflect the known results with the desired degree of accuracy, the algebraic coefficients are frozen and production code is generated. Today, this comprises much of what we understand as artificial intelligence.
But there is a type of software where having a defined output is no longer the case. Actually, two types. One is machine learning systems. The second is predictive analytics, or adaptive systems.
Have objective acceptance criteria. Know the amount of error you and your users are willing to accept.
Test with new data. Once you’ve trained the network and frozen the architecture and coefficients, use fresh inputs and outputs to verify its accuracy.
Don’t count on all results being accurate. That’s just the nature of the beast. And you may have to recommend throwing out the entire network architecture and starting over.
Understand the architecture of the network as a part of the testing process. Few if any will be able to actually follow a set of inputs through the network of algorithms, but understanding how the network is constructed will help testers determine if another architecture might produce better results.
Communicate the level of confidence you have in the results to management and users. Machine learning systems offer you the unique opportunity to describe confidence in statistical terms, so use them.
One important thing to note is that the training data itself could well contain inaccuracies. In this case, because of measurement error, the recorded wind speed and direction could be off or ambiguous. In other cases, the cooling of the filament likely has some error in its measurement.