Ensuring Security and Feature Freshness with VMware Tanzu Observability

©2020 VMware, Inc.
Ensuring Security and
Feature Freshness with
Tanzu Observability
SpringOne 2021

©2020 VMware, Inc. 2
Hi There!
Amber Salome
Chicago, IL
Senior Solutions Architect
VMware Tanzu Platform Services
My focus areas:
Tanzu product consulting and enablement
SRE practices and implementation

➔ Introductions and presentation context
➔ Introduce SRE
➔ Discuss patching and upgrading in an SRE context
➔ Build a custom solution for monitoring legacy and
vulnerability budgets
➔ Visualize vulnerability and legacy budget data
➔ Pull our ﬁndings together into a multi-budget policy
➔ Demonstrate business value
Agenda
During this session, we will...

Participated in a long term client engagement where we managed all aspects of their platform
We became the platform engineers!
One of our goals for this engagement:
Establish an SRE practice around monitoring the necessary patches and upgrades on their
platform
Presentation Context Tanzu Platform Services

Presentation Context Tanzu Platform Services
Participated in a long term client engagement where we managed all aspects of their platform
We became the platform engineers!
One of our goals for this engagement:
Establish an SRE practice around monitoring the necessary patches and upgrades on their
platform
Which is where this presentation comes in!
What tools did we build and how did we mature an SRE practice from just monitoring
availability to holistically applying SRE to security and feature freshness?

6
Confidential │ ©2020 VMware, Inc.
SRE Introduction

Service Level Indicator (SLI)
Service Level Objective (SLO)
Error Budget
SRE Basics
What are we measuring?
ie. Something that is critical to our system’s performance
and impacts user experience
What value are we expecting of our SLI?
ie. A measurement of how our SLI metric should perform.
How much error can we tolerate before the SLO is
breached?

Service Level Indicator
Service Level Objective - 99.5%
Error Budget
Example: Application Availability
We expect end users to be able to access applications
running on the platform.
We expect that 995 out of 1000 requests to an application
on the platform will receive a successful response.
We can tolerate 5 failed application access attempts out of
1000

9
How do we apply SRE to
patching and upgrading
processes?

Your service doesn’t provide value if it’s unavailable or too slow to use, SRE helps
Reliability is a Product Feature
Your users continually expect:
New features that address their needs
These features to work [nearly] all the time
A system that is secure and protected
against common security vulnerabilities

What matters when maintaining a cutting edge platform?
Jim Thomson and David Laing in Extending the Error Budget Model to Security and Feature Freshness
SRECON 2019
Feature Freshness
Security
Availability

So how do you quantify it?
Availability Feature Freshness
Security
Vulnerability
Budget
Error Budget Legacy Budget

Vulnerability Budget
“How long can we aﬀord to be vulnerable to
CVEs in our dependencies?”
PATCHES
Days since patch was released
Service Level Objective
Maximum age threshold in days of an
unpatched system
Legacy Budget
“New enough that it gets support; and not so
old that no-one wants to use it”
UPGRADES
Days since feature release
Service Level Objective
Maximum age threshold in days of an system
that hasn’t been upgraded
From Jim Thomson and David Laing in Extending the Error Budget Model to Security and Feature Freshness

So how do you quantify it?
Vulnerability
Budget
Error Budget
ie. app availability
or cf push success
Legacy Budget
Security

Balancing Stability with Change
Vulnerability
Budget
Error Budget
ie. app availability
or cf push success
Legacy Budget
Stability potentially decreased
when change occurs
Reliability increased by introducing change
Inherent tension
Patches Updates + new features
Security

16
Building a custom reliability
SRE practice

Built a Go tool that targets Operations Manager and Tanzu Network
Products deployed on Tanzu Application Service (TAS)
Calculating Vulnerability and Legacy Budget
Operations
Manager admin
credentials
Tanzu Network
API token
Product Names
Installed Version
Installation Date
of all deployed
products
Provide credentials Query Operations Manager
Latest Patch Version
Latest Patch Release Date
For each queried product
Query Tanzu Network
Vulnerability Budget Spent
Calculate
Latest Upgrade Number
Latest Upgrade Release Date
Legacy Budget Spent

Calculating Vulnerability and Legacy Budget
Vulnerability Budget Spent
time passed from the next
available patch release date
Calculate Metrics
Example
Deployed product is at
v2.4.3 but the most recent
version is v2.10.5.
Legacy Budget Spent
time passed from the next
available upgrade release
date
Vulnerability budget calculates
the time passed since 2.4.4 came
out (minor version)
Legacy budget calculates the
time passed since 2.5.0 came
out (major version)

Automated with Concourse to collect data every week day
When run, outputs data in a table and sends metrics to Tanzu Observability with the Go SDK sender:
Exporting the data
budget.vulnerability.spent
budget.legacy.spent

20
Upgrade and Patching Data
in Tanzu Observability

Vulnerability Budget Current State
Red > 30d
Ops-manager and CF need to be patched
Yellow > 15d
Mysql and wavefront should be considered for
patching soon
Green < 14d
Everything else has been recently patched
Insights: Knowing when to patch

Legacy Budget Current State
Red > 90d
(SCS appears to need an upgrade, **special case)
Yellow > 60d
CF should be considered for an upgrade soon
Green < 60d
Everything else has been recently upgraded
Insights: Knowing when to upgrade

Vulnerability Budget Over Time
Each color segment represents
the vulnerability budget spent
(days since next available patch)
for ONE product

Insights:
A regular patching schedule
Full platform
patching Full platform
patching
Vulnerability Budget Over Time

Legacy Budget Over Time
Each color segment represents
the legacy budget spent (days
since next available patch) for
ONE product

Legacy Budget Over Time
Orange tile
upgraded
Green tile
upgraded
Yellow tile
upgraded
Purple tile (cf)
upgraded, but not
to latest upgrade
Insights:
A regular upgrade schedule

27
Pulling it all together into a
budget policy

So, what matters when maintaining a cutting edge platform?
Jim Thomson and David Laing in Extending the Error Budget Model to Security and Feature Freshness
Feature Freshness
Security
Availability

Prioritizing these three goals to form a policy
Putting it all together
vulnerability budget running low
legacy budget running low
error budget in abundance
error budget running low
Patch the system to recover vulnerability budget
Even if running low on error budget
Upgrade the system to recover legacy budget
Even if it may consume some error budget
Postpone the upgrade until suﬃcient error budget is
available

available

available. Focus on reliability work.

33
Demonstrating Business
Value

Demonstrating Business Value
After three months we did a reliability review to demonstrate the value of monitoring error budget, legacy
budget, and vulnerability budget.
What we aim for
99.5% SLO
What we have achieved over the last 90 days
99.9918% Availability
Application Availability CF Push Availability
What we aim for
99% SLO
What we have achieved over the last 90 days
99.97% Availability

Demonstrating Business Value
We demonstrated that we could do the following WHILE delivering above our availability SLO:
- Maintain all products upgraded within a 90 day window (legacy budget)
- Maintain all products patched within a 30 day window (vulnerability budget)
- Invest time into maturing our SRE process and developing tools/automation to maintain their systems
Overachieving can be an antipattern
- We had aimed for 99.9% app availability within a single-foundation
- Users expect the higher level of performance
- It is unsustainable to run at this level on a system that is only meant to perform at a lower SLO
- Consider trading reliability for cost savings - work reprioritization
- Invest error budget in chaos engineering, more frequent upgrade schedule, etc

A stable and reliable platform is not only deﬁned by availability, but security and staying up to
date with new features
A holistic SRE practice around platform ensures availability, security, and feature freshness
Change can introduce instability, but tracking error, vulnerability, and legacy budgets + balancing
spending allows for prioritization of patching and upgrading when necessary
(and instills conﬁdence in prioritizing work)
Ending Notes

Acknowledgements
Big thank yous to:
Zac Bergquist (Teammate, Editor)
Josh Ghiloni (Teammate, Editor)
Clayton Hynﬁeld (Editor)
Samah Majadla (Editor)
Jim Thomson and David Laing in
Extending the Error Budget Model to Security and Feature Freshness (Foundational Concepts)

Disclaimer
This presentation may contain product features or functionality that are currently under
development.
This overview of new technology represents no commitment from VMware to deliver these
features in any generally available product.
Features are subject to change, and must not be included in contracts, purchase orders, or
sales agreements of any kind.
Technical feasibility and market demand will aﬀect ﬁnal delivery.
Pricing and packaging for any new features/functionality/technology discussed or
presented, have not been determined.
The information in this presentation is for informational purposes only and may not be incorporated into any contract. There is no commitment or
obligation to deliver any items presented herein.

Ensuring Security and Feature Freshness with VMware Tanzu Observability

Empfohlen

Empfohlen

Weitere ähnliche Inhalte

Was ist angesagt?

Was ist angesagt? (20)

Ähnlich wie Ensuring Security and Feature Freshness with VMware Tanzu Observability

Ähnlich wie Ensuring Security and Feature Freshness with VMware Tanzu Observability (20)

Mehr von VMware Tanzu

Mehr von VMware Tanzu (20)

Kürzlich hochgeladen

Kürzlich hochgeladen (20)

Ensuring Security and Feature Freshness with VMware Tanzu Observability