SlideShare ist ein Scribd-Unternehmen logo
1 von 51
Downloaden Sie, um offline zu lesen
Container Patching
Making It Less Gross Than The Seattle Gum Wall
Greg Castle
@mrgcastle, @gregcastle@infosec.exchange
GKE Security, Google Cloud
Weston Panther
GKE Security, Google Cloud
Simple View Of Patching
0 days Scanner detects
15 days Maintainer patches
30 days Production patched
Severity FedRAMP
Targets
CRITICAL/
HIGH
30 days
Medium 90 days
Low 180 days
A Trip Down Empathy Lane
2 weeks 🛡 …Bi-weekly cluster scan…
2 weeks 🛡 Hey web team, webfrontend is missing 2 CRITICAL patches
3 weeks 🛡 Friendly ping?
3 weeks 🕸 Not our code, maybe django base container?
3 weeks 🛡 Django container team, can you patch?
4 weeks 🐍 These vulns are in perl, we don’t even use perl, do we need to patch?
A Trip Down Empathy Lane
4 weeks 🛡 Yes, or better yet, remove perl.
OUT OF FedRAMP/PCI SLO
5 weeks 🐍 Patched 🎉 acme-django:v2.1.1
5 weeks 🛡 Hey web team, rebuild with acme-django:v2.1.1
6 weeks 🕸 Done!
7 weeks 🛡 Still running the old version?
A Trip Down Empathy Lane
8 weeks 🕸 Forgot to update the K8s manifest. Done!
9 weeks 🛡 …Still no? Also there’s three new HIGH vulns, but let’s get this done first.
10 weeks 🕸 Had to soak in QA first, updated for prod rollout.
11 weeks 🛡 Fixed! Who else runs django apps..?
Why It’s Gross
Humans at every step
Which layer needs patching?
No inventory
Patching unused code
Vulns faster than patches
= Slow, incomplete, unscalable
patching
Is the majority of the industry doing
better than this today?
88% of respondents:
“Challenging to ensure containerized
applications are free from vulnerabilities”
https://www.slim.ai/blog/container-report-2022/
GKE Container Patching
Case Study
Enforcement Points
Prevent: minimal containers
Detect: scanning capability/coverage
Fix: ownership, dependencies, release
Monitor: dashboards, alerting, escalations
What Containers?
Vendor/MSP containers
Containers you rebuild
K8s manifests you update
What Do We Know Anyway?
Patching for 1000s of containers across GKE,
Anthos and adjacent products
But…our environment constraints help a lot:
● Mandatory use of compiled language
● Mandatory container repo
● Mandatory base images
● Control over code/config pre-submit
● Control over release
Container/K8s Delivery Pipeline
Source
Container
Build
Package Run
Deploy
Staging Repo Prod Repo
Inventory
Dev
Good Start: Runtime Detection
Source
Container
Build
Package Run
Deploy
Staging Repo Prod Repo
Inventory
Dev
Runtime scanner:
detection and
inventory
Better: Prevention Complementing Detection
Source
Container
Build
Package Run
Deploy
Staging Repo Prod Repo
Inventory
Dev
Prevention requires integration with pipeline
Prevent
Prevent: Problems
So many containers
So many dependencies
Meeting SLO is hard without reducing volume
#Prevent
Prevent: Strategy
● Standardize base containers
● Minimal containers: Less code, less vulns, less patching
● Remove unused code: separate build and runtime images
● Two approaches:
○ Start small: Scratch, Distroless, Wolfi/Chainguard
Images
○ Slim down: SlimToolkit
● Challenge: apply consistently everywhere
#Prevent
Our Solution
● Standardize on Distroless
○ Just enough to run golang binaries
● All containers in a single registry
○ Inventory
○ Availability
#Prevent
Our Solution
Source
Dev Pre-Submit Check
image = gcr.io/gke-release/*
deployment.yaml
Container /etc/os-release
HOME_URL=https://github.com/
GoogleContainerTools/distroless
#Prevent
Is distroless
Alternatives
Container
Build
Package Run
Deploy
Staging Repo Prod Repo
Is distroless
image = gcr.io/gke-release/*
#Prevent
Alternatives: Admission
Container
Build
Run
Deploy
Prod Repo
isDistroless
attestation
Admission
image = gcr.io/gke-release/*
Verify isDistroless attestation
#Prevent
On GKE: Use Binary
Authorization
Demo:
Admission
#Prevent
Prevent: Summary
● Identify and use enforcement points
● Standardize on patchable base containers
● Standardize on container registries for inventory
#Prevent
Detect
Detect: Problems
● Which containers to scan?
● Which scanner?
○ Different coverage
○ Different vuln sources
○ Duplicate handling
○ Filtering noise
● Which layer has the vuln?
#Detect
All Container Images
Which Container? Our Solution
Source
Dev
List of Containers in
Production
Scanner
Pre-Submit Check
Image Fully Patched
#Detect
Which Container? Alternatives
Inventory
List of Containers In
Production
Scanner
In registry, but what is
running in production?
Daemonset
Prod Repo
Run
#Detect
Language Pack
Scanning
SBOM Consumption
VEX Support
Supplemental
CVE Sources
Scan programs in your container:
● Rust Cargo.lock
● Python egg files
● Go binaries / go.mod
● etc.
Scanners are starting to
support SBOMs
Filter out remediated
vulnerabilities based on VEX
More vulns from more places
● OS vendor feeds
● Github Advisories
Database
● Language-specific DBs
(vuln.go.dev)
#Detect
Which Scanner?
Base Image
Detection
Reachability
Analysis
Additional Scans
Try to determine the base
images:
● From metadata in image
manifest
● From the Dockerfile
Try to figure out if the code is
actually in use:
● Typically uses source
● Can use symbol table in
a binary
Scan all the things:
● CIS benchmarks
● Hardcoded keys
● Misconfigurations (root
user, host volume
mounts, etc.)
#Detect
Which Scanner?
False Positives vs. Coverage
#Detect
Which one is correct?
● Vulnerable module
(golang.org/x/crypto/ssh)
● Built with old golang
version (1.18.1)
● On old debian base
(buster-20210208)
Which Scanner? Our Solution
Public containers: probably “more than one”
Identify gaps and false positives
See what our customers see
#Detect
Detect: Noise
● CVEs that will never be patched (debian CVE-2004-0971,
CVE-2005-2541, CVE-2010-4756)
● Ancient low priority vulns without patches (debian
CVE-2011-4116, CVE-2016-2781)
● OS vendor has a lower rating than NVD (debian
CVE-2022-37434)
● CVE is for a different architecture (golang
CVE-2021-38297)
● CVEs that are clearly overrated (CVE-2020-29363: 9.8
down to 7.5) #Detect
User Control
Scanner Control
● Codepath is unused
● Recent CVEs with no patch
Noise: Golang Specific
● Problem: all vulns in whole version or module detected
● Solution: govulncheck can report only reachable
vulnerabilities
● go.dev/blog/vuln
#Detect
Demo:
govulncheck
#Detect
Detect: Summary
● Take advantage of new advances in coverage
● Look to your scanner vendor to help with noise
● Use silence/ignore where it fits threat model
#Detect
Fix
Problems: Multi-layer Complex Process
CVE 2023-123 in
kube-proxy
Scanner
This Container? Find Owner File Bug/
Send PR
Wait For
Rebuild
Dependent
Containers?
Done
Find Parent
yes
no
no
yes
#Fix
Our Solution: Base Images
#Fix
gke-distroless
debian-base
debian-iptables
v1
v1
v1
v2
v2
v3
v3
1. Scan the latest base images
2. If fixable vulns, rebuild
3. Repeat for eternity
v4
Our Solution: Ownership
#Fix
Find Owner
Source
Dev Pre-Submit Check
Image Ownership File
deployment.yaml
Our Solution: Simplified Process
CVE 2023-123 in
kube-proxy
Scanner Find owner File Bug/
Send PR
Wait for
rebuild
Done
#Fix
Summary: Fix
● Track container parent-child relationships
● Automate patching base images
● Comprehensive inventory and ownership
● Use existing ticket systems to track
#Fix
Monitor
Monitor: Problems
#Monitor
Container isn’t patched - who
is watching? Who do we
escalate to?
Which containers have the
CVE? Which applications use
this container?
Are we meeting our SLOs?
What are the gaps and pain
points?
Is CVE-123 patched? Has it
rolled out everywhere?
What gets measured…
Our Solution
#Monitor
New Finding?
Scanner
Detects CVE
Find Existing Bug
yes
Nearing SLO? Escalate
no
Bug Filed
Add Comment Past SLO?
yes yes
CVEs
Containers
Applications
GKE Version
Monitor: Composition
#Monitor
CVE-2021-44228?
Monitor: Visibility
#Monitor
New Image
Created
PR Merged
Manifest Updated
Qualification Starts
Rollout Begins
Patched
Clock starts
Dashboards provide status at-a-glance Track progress with metrics over time
Measure each step to find pain points
Active CVE Count By Image
Image Name Image Tag # Fixable CVEs
fake-image v1.0.1 55
fake-image v1.0.3 45
demo-image v3.5 20
nginx 1.22.1 10
Monitor: Alternatives
● Inventory: Scanners
● Composition:
○ Lyft: Cartography graph database
○ SBOMs / GUAC
○ Ignore layers, just patch: copacetic, crane rebase
● SLO:
○ Bug management software
○ Track commits and rollouts
#Monitor
Summary: Monitor
● Track SLOs over time
● Track patch/release stages to identify bottlenecks
● Use existing systems for escalation/dashboarding
#Monitor
Summary
● Standardize on registries and minimal containers
● Enforce as far left as possible
● Scanners for inventory + visibility
● Record ownership of containers
● Auto-patch if possible
● Tickets to track/escalate
Prefer automation (doing) over telling
Links
● Demo code
● Slim.ai container report
● Lyft patching blogpost
● Separate build and runtime images
● Small images: Scratch, Distroless, Wolfi/Chainguard Images
● SlimToolkit
● AllowedRepos Gatekeeper policy
● Sigstore: signing, policy controller
● GKE Binary Authorization: attestations, image policy
● Opensource scanners: trivy, clair
● Google Container Analysis
● GUAC
● The Seattle Gum Wall
Feedback
@mrgcastle
@gregcastle@infosec.exchange
Appendix: Feature Request Wishlist Idea
#Detect
Prisma Cloud has received 150 reports that dispute this severity
Aqua Security has received 231 reports that dispute this severity
Google Container Analysis has received 109 reports that dispute this severity
If enough users report a critical vuln as inaccurate, the scanner manually evaluates,
updates the severity for all their users, and works with NIST to correct NVD
#Detect

Más contenido relacionado

Was ist angesagt?

Strategy considerations for building a security operations center
Strategy considerations for building a security operations centerStrategy considerations for building a security operations center
Strategy considerations for building a security operations centerCMR WORLD TECH
 
Privileged Access Management (PAM)
Privileged Access Management (PAM)Privileged Access Management (PAM)
Privileged Access Management (PAM)danb02
 
Cyber Security 101
Cyber Security 101Cyber Security 101
Cyber Security 101Cloudflare
 
Overview of the Cyber Kill Chain [TM]
Overview of the Cyber Kill Chain [TM]Overview of the Cyber Kill Chain [TM]
Overview of the Cyber Kill Chain [TM]David Sweigert
 
Siem ppt
Siem pptSiem ppt
Siem pptkmehul
 
Security operations center-SOC Presentation-مرکز عملیات امنیت
Security operations center-SOC Presentation-مرکز عملیات امنیتSecurity operations center-SOC Presentation-مرکز عملیات امنیت
Security operations center-SOC Presentation-مرکز عملیات امنیتReZa AdineH
 
[Round table] zeroing in on zero trust architecture
[Round table] zeroing in on zero trust architecture[Round table] zeroing in on zero trust architecture
[Round table] zeroing in on zero trust architectureDenise Bailey
 
Image Scanning Best Practices for Containers and Kubernetes
Image Scanning Best Practices for Containers and KubernetesImage Scanning Best Practices for Containers and Kubernetes
Image Scanning Best Practices for Containers and KubernetesDevOps.com
 
introduction to Azure Sentinel
introduction to Azure Sentinelintroduction to Azure Sentinel
introduction to Azure SentinelRobert Crane
 
Identity and Access Management 101
Identity and Access Management 101Identity and Access Management 101
Identity and Access Management 101Jerod Brennen
 
SOC Architecture Workshop - Part 1
SOC Architecture Workshop - Part 1SOC Architecture Workshop - Part 1
SOC Architecture Workshop - Part 1Priyanka Aash
 
Upgrade Your SOC with Cortex XSOAR & Elastic SIEM
Upgrade Your SOC with Cortex XSOAR & Elastic SIEMUpgrade Your SOC with Cortex XSOAR & Elastic SIEM
Upgrade Your SOC with Cortex XSOAR & Elastic SIEMElasticsearch
 
Microsoft Defender and Azure Sentinel
Microsoft Defender and Azure SentinelMicrosoft Defender and Azure Sentinel
Microsoft Defender and Azure SentinelDavid J Rosenthal
 
Next-Gen security operation center
Next-Gen security operation centerNext-Gen security operation center
Next-Gen security operation centerMuhammad Sahputra
 

Was ist angesagt? (20)

Strategy considerations for building a security operations center
Strategy considerations for building a security operations centerStrategy considerations for building a security operations center
Strategy considerations for building a security operations center
 
Microsoft Zero Trust
Microsoft Zero TrustMicrosoft Zero Trust
Microsoft Zero Trust
 
Privileged Access Management (PAM)
Privileged Access Management (PAM)Privileged Access Management (PAM)
Privileged Access Management (PAM)
 
Cyber Security 101
Cyber Security 101Cyber Security 101
Cyber Security 101
 
The State of DevSecOps
The State of DevSecOpsThe State of DevSecOps
The State of DevSecOps
 
cn-series-se-presentation.pptx
cn-series-se-presentation.pptxcn-series-se-presentation.pptx
cn-series-se-presentation.pptx
 
Introduction to DevSecOps
Introduction to DevSecOpsIntroduction to DevSecOps
Introduction to DevSecOps
 
Overview of the Cyber Kill Chain [TM]
Overview of the Cyber Kill Chain [TM]Overview of the Cyber Kill Chain [TM]
Overview of the Cyber Kill Chain [TM]
 
Siem ppt
Siem pptSiem ppt
Siem ppt
 
Security and Compliance
Security and ComplianceSecurity and Compliance
Security and Compliance
 
Security operations center-SOC Presentation-مرکز عملیات امنیت
Security operations center-SOC Presentation-مرکز عملیات امنیتSecurity operations center-SOC Presentation-مرکز عملیات امنیت
Security operations center-SOC Presentation-مرکز عملیات امنیت
 
Secure Design: Threat Modeling
Secure Design: Threat ModelingSecure Design: Threat Modeling
Secure Design: Threat Modeling
 
[Round table] zeroing in on zero trust architecture
[Round table] zeroing in on zero trust architecture[Round table] zeroing in on zero trust architecture
[Round table] zeroing in on zero trust architecture
 
Image Scanning Best Practices for Containers and Kubernetes
Image Scanning Best Practices for Containers and KubernetesImage Scanning Best Practices for Containers and Kubernetes
Image Scanning Best Practices for Containers and Kubernetes
 
introduction to Azure Sentinel
introduction to Azure Sentinelintroduction to Azure Sentinel
introduction to Azure Sentinel
 
Identity and Access Management 101
Identity and Access Management 101Identity and Access Management 101
Identity and Access Management 101
 
SOC Architecture Workshop - Part 1
SOC Architecture Workshop - Part 1SOC Architecture Workshop - Part 1
SOC Architecture Workshop - Part 1
 
Upgrade Your SOC with Cortex XSOAR & Elastic SIEM
Upgrade Your SOC with Cortex XSOAR & Elastic SIEMUpgrade Your SOC with Cortex XSOAR & Elastic SIEM
Upgrade Your SOC with Cortex XSOAR & Elastic SIEM
 
Microsoft Defender and Azure Sentinel
Microsoft Defender and Azure SentinelMicrosoft Defender and Azure Sentinel
Microsoft Defender and Azure Sentinel
 
Next-Gen security operation center
Next-Gen security operation centerNext-Gen security operation center
Next-Gen security operation center
 

Ähnlich wie Container Patching: Cloud Native Security Con 2023

Security Testing for Containerized Applications
Security Testing for Containerized ApplicationsSecurity Testing for Containerized Applications
Security Testing for Containerized ApplicationsSoluto
 
DevSecCon Tel Aviv 2018 - Security Testing for Containerised Apps by Omer Levi
DevSecCon Tel Aviv 2018 - Security Testing for  Containerised Apps by Omer LeviDevSecCon Tel Aviv 2018 - Security Testing for  Containerised Apps by Omer Levi
DevSecCon Tel Aviv 2018 - Security Testing for Containerised Apps by Omer LeviDevSecCon
 
Achieving Full Stack DevOps at Colonial Life
Achieving Full Stack DevOps at Colonial Life Achieving Full Stack DevOps at Colonial Life
Achieving Full Stack DevOps at Colonial Life DevOps.com
 
Introduction of Tools for providing rich user experience in debugger
Introduction of Tools for providing rich user experience in debuggerIntroduction of Tools for providing rich user experience in debugger
Introduction of Tools for providing rich user experience in debuggerNaoto Ono
 
Creating a reasonable project boilerplate
Creating a reasonable project boilerplateCreating a reasonable project boilerplate
Creating a reasonable project boilerplateStanislav Petrov
 
Extensible dev secops pipelines with Jenkins, Docker, Terraform, and a kitche...
Extensible dev secops pipelines with Jenkins, Docker, Terraform, and a kitche...Extensible dev secops pipelines with Jenkins, Docker, Terraform, and a kitche...
Extensible dev secops pipelines with Jenkins, Docker, Terraform, and a kitche...Richard Bullington-McGuire
 
Rooted con 2020 - from the heaven to hell in the CI - CD
Rooted con 2020 - from the heaven to hell in the CI - CDRooted con 2020 - from the heaven to hell in the CI - CD
Rooted con 2020 - from the heaven to hell in the CI - CDDaniel Garcia (a.k.a cr0hn)
 
What can possibly go wrong if i dont e2 e test my packages?
What can possibly go wrong if i dont e2 e test my packages?What can possibly go wrong if i dont e2 e test my packages?
What can possibly go wrong if i dont e2 e test my packages?Juan Picado
 
Skip Staging! Test Docker, Helm, and Kubernetes Apps like a Pro
Skip Staging! Test Docker, Helm, and Kubernetes Apps like a ProSkip Staging! Test Docker, Helm, and Kubernetes Apps like a Pro
Skip Staging! Test Docker, Helm, and Kubernetes Apps like a ProCodefresh
 
Big feature - small sprint
Big feature - small sprint Big feature - small sprint
Big feature - small sprint Igor Goldshmidt
 
Containerize your Blackbox tests
Containerize your Blackbox testsContainerize your Blackbox tests
Containerize your Blackbox testsKevin Beeman
 
Managing Software Dependencies and the Supply Chain_ MIT EM.S20.pdf
Managing Software Dependencies and the Supply Chain_ MIT EM.S20.pdfManaging Software Dependencies and the Supply Chain_ MIT EM.S20.pdf
Managing Software Dependencies and the Supply Chain_ MIT EM.S20.pdfAndrew Lamb
 
Groovy there's a docker in my application pipeline
Groovy there's a docker in my application pipelineGroovy there's a docker in my application pipeline
Groovy there's a docker in my application pipelineKris Buytaert
 
OSMC 2017 | Groovy There is a Docker in my Dashing Pipeline by Kris Buytaert
OSMC 2017 | Groovy There is a Docker in my Dashing Pipeline by Kris Buytaert OSMC 2017 | Groovy There is a Docker in my Dashing Pipeline by Kris Buytaert
OSMC 2017 | Groovy There is a Docker in my Dashing Pipeline by Kris Buytaert NETWAYS
 
Adventures in Asymmetric Warfare
Adventures in Asymmetric WarfareAdventures in Asymmetric Warfare
Adventures in Asymmetric WarfareWill Schroeder
 

Ähnlich wie Container Patching: Cloud Native Security Con 2023 (20)

Sonatype DevSecOps Leadership forum 2020
Sonatype DevSecOps Leadership forum 2020Sonatype DevSecOps Leadership forum 2020
Sonatype DevSecOps Leadership forum 2020
 
Security Testing for Containerized Applications
Security Testing for Containerized ApplicationsSecurity Testing for Containerized Applications
Security Testing for Containerized Applications
 
DevSecCon Tel Aviv 2018 - Security Testing for Containerised Apps by Omer Levi
DevSecCon Tel Aviv 2018 - Security Testing for  Containerised Apps by Omer LeviDevSecCon Tel Aviv 2018 - Security Testing for  Containerised Apps by Omer Levi
DevSecCon Tel Aviv 2018 - Security Testing for Containerised Apps by Omer Levi
 
Achieving Full Stack DevOps at Colonial Life
Achieving Full Stack DevOps at Colonial Life Achieving Full Stack DevOps at Colonial Life
Achieving Full Stack DevOps at Colonial Life
 
Introduction of Tools for providing rich user experience in debugger
Introduction of Tools for providing rich user experience in debuggerIntroduction of Tools for providing rich user experience in debugger
Introduction of Tools for providing rich user experience in debugger
 
Creating a reasonable project boilerplate
Creating a reasonable project boilerplateCreating a reasonable project boilerplate
Creating a reasonable project boilerplate
 
Extensible dev secops pipelines with Jenkins, Docker, Terraform, and a kitche...
Extensible dev secops pipelines with Jenkins, Docker, Terraform, and a kitche...Extensible dev secops pipelines with Jenkins, Docker, Terraform, and a kitche...
Extensible dev secops pipelines with Jenkins, Docker, Terraform, and a kitche...
 
Rooted con 2020 - from the heaven to hell in the CI - CD
Rooted con 2020 - from the heaven to hell in the CI - CDRooted con 2020 - from the heaven to hell in the CI - CD
Rooted con 2020 - from the heaven to hell in the CI - CD
 
What can possibly go wrong if i dont e2 e test my packages?
What can possibly go wrong if i dont e2 e test my packages?What can possibly go wrong if i dont e2 e test my packages?
What can possibly go wrong if i dont e2 e test my packages?
 
Pluginize ALL the things
Pluginize ALL the thingsPluginize ALL the things
Pluginize ALL the things
 
Go at Skroutz
Go at SkroutzGo at Skroutz
Go at Skroutz
 
Skip Staging! Test Docker, Helm, and Kubernetes Apps like a Pro
Skip Staging! Test Docker, Helm, and Kubernetes Apps like a ProSkip Staging! Test Docker, Helm, and Kubernetes Apps like a Pro
Skip Staging! Test Docker, Helm, and Kubernetes Apps like a Pro
 
Big feature - small sprint
Big feature - small sprint Big feature - small sprint
Big feature - small sprint
 
Containerize your Blackbox tests
Containerize your Blackbox testsContainerize your Blackbox tests
Containerize your Blackbox tests
 
Managing Software Dependencies and the Supply Chain_ MIT EM.S20.pdf
Managing Software Dependencies and the Supply Chain_ MIT EM.S20.pdfManaging Software Dependencies and the Supply Chain_ MIT EM.S20.pdf
Managing Software Dependencies and the Supply Chain_ MIT EM.S20.pdf
 
Putting the pro in programmer
Putting the pro in programmerPutting the pro in programmer
Putting the pro in programmer
 
12 tricks to avoid hackers breaks your CI / CD
12 tricks to avoid hackers breaks your  CI / CD12 tricks to avoid hackers breaks your  CI / CD
12 tricks to avoid hackers breaks your CI / CD
 
Groovy there's a docker in my application pipeline
Groovy there's a docker in my application pipelineGroovy there's a docker in my application pipeline
Groovy there's a docker in my application pipeline
 
OSMC 2017 | Groovy There is a Docker in my Dashing Pipeline by Kris Buytaert
OSMC 2017 | Groovy There is a Docker in my Dashing Pipeline by Kris Buytaert OSMC 2017 | Groovy There is a Docker in my Dashing Pipeline by Kris Buytaert
OSMC 2017 | Groovy There is a Docker in my Dashing Pipeline by Kris Buytaert
 
Adventures in Asymmetric Warfare
Adventures in Asymmetric WarfareAdventures in Asymmetric Warfare
Adventures in Asymmetric Warfare
 

Último

Where developers are challenged, what developers want and where DevEx is going
Where developers are challenged, what developers want and where DevEx is goingWhere developers are challenged, what developers want and where DevEx is going
Where developers are challenged, what developers want and where DevEx is goingFrancesco Corti
 
Extra-120324-Visite-Entreprise-icare.pdf
Extra-120324-Visite-Entreprise-icare.pdfExtra-120324-Visite-Entreprise-icare.pdf
Extra-120324-Visite-Entreprise-icare.pdfInfopole1
 
Explore the UiPath Community and ways you can benefit on your journey to auto...
Explore the UiPath Community and ways you can benefit on your journey to auto...Explore the UiPath Community and ways you can benefit on your journey to auto...
Explore the UiPath Community and ways you can benefit on your journey to auto...DianaGray10
 
EMEA What is ThousandEyes? Webinar
EMEA What is ThousandEyes? WebinarEMEA What is ThousandEyes? Webinar
EMEA What is ThousandEyes? WebinarThousandEyes
 
How to release an Open Source Dataweave Library
How to release an Open Source Dataweave LibraryHow to release an Open Source Dataweave Library
How to release an Open Source Dataweave Libraryshyamraj55
 
Novo Nordisk's journey in developing an open-source application on Neo4j
Novo Nordisk's journey in developing an open-source application on Neo4jNovo Nordisk's journey in developing an open-source application on Neo4j
Novo Nordisk's journey in developing an open-source application on Neo4jNeo4j
 
Design and Modeling for MySQL SCALE 21X Pasadena, CA Mar 2024
Design and Modeling for MySQL SCALE 21X Pasadena, CA Mar 2024Design and Modeling for MySQL SCALE 21X Pasadena, CA Mar 2024
Design and Modeling for MySQL SCALE 21X Pasadena, CA Mar 2024Alkin Tezuysal
 
The New Cloud World Order Is FinOps (Slideshow)
The New Cloud World Order Is FinOps (Slideshow)The New Cloud World Order Is FinOps (Slideshow)
The New Cloud World Order Is FinOps (Slideshow)codyslingerland1
 
Introduction to RAG (Retrieval Augmented Generation) and its application
Introduction to RAG (Retrieval Augmented Generation) and its applicationIntroduction to RAG (Retrieval Augmented Generation) and its application
Introduction to RAG (Retrieval Augmented Generation) and its applicationKnoldus Inc.
 
The Zero-ETL Approach: Enhancing Data Agility and Insight
The Zero-ETL Approach: Enhancing Data Agility and InsightThe Zero-ETL Approach: Enhancing Data Agility and Insight
The Zero-ETL Approach: Enhancing Data Agility and InsightSafe Software
 
3 Pitfalls Everyone Should Avoid with Cloud Data
3 Pitfalls Everyone Should Avoid with Cloud Data3 Pitfalls Everyone Should Avoid with Cloud Data
3 Pitfalls Everyone Should Avoid with Cloud DataEric D. Schabell
 
2024.03.12 Cost drivers of cultivated meat production.pdf
2024.03.12 Cost drivers of cultivated meat production.pdf2024.03.12 Cost drivers of cultivated meat production.pdf
2024.03.12 Cost drivers of cultivated meat production.pdfThe Good Food Institute
 
CyberSecurity - Computers In Libraries 2024
CyberSecurity - Computers In Libraries 2024CyberSecurity - Computers In Libraries 2024
CyberSecurity - Computers In Libraries 2024Brian Pichman
 
How to become a GDSC Lead GDSC MI AOE.pptx
How to become a GDSC Lead GDSC MI AOE.pptxHow to become a GDSC Lead GDSC MI AOE.pptx
How to become a GDSC Lead GDSC MI AOE.pptxKaustubhBhavsar6
 
20140402 - Smart house demo kit
20140402 - Smart house demo kit20140402 - Smart house demo kit
20140402 - Smart house demo kitJamie (Taka) Wang
 
Emil Eifrem at GraphSummit Copenhagen 2024 - The Art of the Possible.pptx
Emil Eifrem at GraphSummit Copenhagen 2024 - The Art of the Possible.pptxEmil Eifrem at GraphSummit Copenhagen 2024 - The Art of the Possible.pptx
Emil Eifrem at GraphSummit Copenhagen 2024 - The Art of the Possible.pptxNeo4j
 
Webinar: The Art of Prioritizing Your Product Roadmap by AWS Sr PM - Tech
Webinar: The Art of Prioritizing Your Product Roadmap by AWS Sr PM - TechWebinar: The Art of Prioritizing Your Product Roadmap by AWS Sr PM - Tech
Webinar: The Art of Prioritizing Your Product Roadmap by AWS Sr PM - TechProduct School
 
.NET 8 ChatBot with Azure OpenAI Services.pptx
.NET 8 ChatBot with Azure OpenAI Services.pptx.NET 8 ChatBot with Azure OpenAI Services.pptx
.NET 8 ChatBot with Azure OpenAI Services.pptxHansamali Gamage
 
Automation Ops Series: Session 2 - Governance for UiPath projects
Automation Ops Series: Session 2 - Governance for UiPath projectsAutomation Ops Series: Session 2 - Governance for UiPath projects
Automation Ops Series: Session 2 - Governance for UiPath projectsDianaGray10
 
Top 10 Squarespace Development Companies
Top 10 Squarespace Development CompaniesTop 10 Squarespace Development Companies
Top 10 Squarespace Development CompaniesTopCSSGallery
 

Último (20)

Where developers are challenged, what developers want and where DevEx is going
Where developers are challenged, what developers want and where DevEx is goingWhere developers are challenged, what developers want and where DevEx is going
Where developers are challenged, what developers want and where DevEx is going
 
Extra-120324-Visite-Entreprise-icare.pdf
Extra-120324-Visite-Entreprise-icare.pdfExtra-120324-Visite-Entreprise-icare.pdf
Extra-120324-Visite-Entreprise-icare.pdf
 
Explore the UiPath Community and ways you can benefit on your journey to auto...
Explore the UiPath Community and ways you can benefit on your journey to auto...Explore the UiPath Community and ways you can benefit on your journey to auto...
Explore the UiPath Community and ways you can benefit on your journey to auto...
 
EMEA What is ThousandEyes? Webinar
EMEA What is ThousandEyes? WebinarEMEA What is ThousandEyes? Webinar
EMEA What is ThousandEyes? Webinar
 
How to release an Open Source Dataweave Library
How to release an Open Source Dataweave LibraryHow to release an Open Source Dataweave Library
How to release an Open Source Dataweave Library
 
Novo Nordisk's journey in developing an open-source application on Neo4j
Novo Nordisk's journey in developing an open-source application on Neo4jNovo Nordisk's journey in developing an open-source application on Neo4j
Novo Nordisk's journey in developing an open-source application on Neo4j
 
Design and Modeling for MySQL SCALE 21X Pasadena, CA Mar 2024
Design and Modeling for MySQL SCALE 21X Pasadena, CA Mar 2024Design and Modeling for MySQL SCALE 21X Pasadena, CA Mar 2024
Design and Modeling for MySQL SCALE 21X Pasadena, CA Mar 2024
 
The New Cloud World Order Is FinOps (Slideshow)
The New Cloud World Order Is FinOps (Slideshow)The New Cloud World Order Is FinOps (Slideshow)
The New Cloud World Order Is FinOps (Slideshow)
 
Introduction to RAG (Retrieval Augmented Generation) and its application
Introduction to RAG (Retrieval Augmented Generation) and its applicationIntroduction to RAG (Retrieval Augmented Generation) and its application
Introduction to RAG (Retrieval Augmented Generation) and its application
 
The Zero-ETL Approach: Enhancing Data Agility and Insight
The Zero-ETL Approach: Enhancing Data Agility and InsightThe Zero-ETL Approach: Enhancing Data Agility and Insight
The Zero-ETL Approach: Enhancing Data Agility and Insight
 
3 Pitfalls Everyone Should Avoid with Cloud Data
3 Pitfalls Everyone Should Avoid with Cloud Data3 Pitfalls Everyone Should Avoid with Cloud Data
3 Pitfalls Everyone Should Avoid with Cloud Data
 
2024.03.12 Cost drivers of cultivated meat production.pdf
2024.03.12 Cost drivers of cultivated meat production.pdf2024.03.12 Cost drivers of cultivated meat production.pdf
2024.03.12 Cost drivers of cultivated meat production.pdf
 
CyberSecurity - Computers In Libraries 2024
CyberSecurity - Computers In Libraries 2024CyberSecurity - Computers In Libraries 2024
CyberSecurity - Computers In Libraries 2024
 
How to become a GDSC Lead GDSC MI AOE.pptx
How to become a GDSC Lead GDSC MI AOE.pptxHow to become a GDSC Lead GDSC MI AOE.pptx
How to become a GDSC Lead GDSC MI AOE.pptx
 
20140402 - Smart house demo kit
20140402 - Smart house demo kit20140402 - Smart house demo kit
20140402 - Smart house demo kit
 
Emil Eifrem at GraphSummit Copenhagen 2024 - The Art of the Possible.pptx
Emil Eifrem at GraphSummit Copenhagen 2024 - The Art of the Possible.pptxEmil Eifrem at GraphSummit Copenhagen 2024 - The Art of the Possible.pptx
Emil Eifrem at GraphSummit Copenhagen 2024 - The Art of the Possible.pptx
 
Webinar: The Art of Prioritizing Your Product Roadmap by AWS Sr PM - Tech
Webinar: The Art of Prioritizing Your Product Roadmap by AWS Sr PM - TechWebinar: The Art of Prioritizing Your Product Roadmap by AWS Sr PM - Tech
Webinar: The Art of Prioritizing Your Product Roadmap by AWS Sr PM - Tech
 
.NET 8 ChatBot with Azure OpenAI Services.pptx
.NET 8 ChatBot with Azure OpenAI Services.pptx.NET 8 ChatBot with Azure OpenAI Services.pptx
.NET 8 ChatBot with Azure OpenAI Services.pptx
 
Automation Ops Series: Session 2 - Governance for UiPath projects
Automation Ops Series: Session 2 - Governance for UiPath projectsAutomation Ops Series: Session 2 - Governance for UiPath projects
Automation Ops Series: Session 2 - Governance for UiPath projects
 
Top 10 Squarespace Development Companies
Top 10 Squarespace Development CompaniesTop 10 Squarespace Development Companies
Top 10 Squarespace Development Companies
 

Container Patching: Cloud Native Security Con 2023

  • 1. Container Patching Making It Less Gross Than The Seattle Gum Wall Greg Castle @mrgcastle, @gregcastle@infosec.exchange GKE Security, Google Cloud Weston Panther GKE Security, Google Cloud
  • 2. Simple View Of Patching 0 days Scanner detects 15 days Maintainer patches 30 days Production patched Severity FedRAMP Targets CRITICAL/ HIGH 30 days Medium 90 days Low 180 days
  • 3. A Trip Down Empathy Lane 2 weeks 🛡 …Bi-weekly cluster scan… 2 weeks 🛡 Hey web team, webfrontend is missing 2 CRITICAL patches 3 weeks 🛡 Friendly ping? 3 weeks 🕸 Not our code, maybe django base container? 3 weeks 🛡 Django container team, can you patch? 4 weeks 🐍 These vulns are in perl, we don’t even use perl, do we need to patch?
  • 4. A Trip Down Empathy Lane 4 weeks 🛡 Yes, or better yet, remove perl. OUT OF FedRAMP/PCI SLO 5 weeks 🐍 Patched 🎉 acme-django:v2.1.1 5 weeks 🛡 Hey web team, rebuild with acme-django:v2.1.1 6 weeks 🕸 Done! 7 weeks 🛡 Still running the old version?
  • 5. A Trip Down Empathy Lane 8 weeks 🕸 Forgot to update the K8s manifest. Done! 9 weeks 🛡 …Still no? Also there’s three new HIGH vulns, but let’s get this done first. 10 weeks 🕸 Had to soak in QA first, updated for prod rollout. 11 weeks 🛡 Fixed! Who else runs django apps..?
  • 6. Why It’s Gross Humans at every step Which layer needs patching? No inventory Patching unused code Vulns faster than patches = Slow, incomplete, unscalable patching
  • 7. Is the majority of the industry doing better than this today?
  • 8. 88% of respondents: “Challenging to ensure containerized applications are free from vulnerabilities” https://www.slim.ai/blog/container-report-2022/
  • 9. GKE Container Patching Case Study Enforcement Points Prevent: minimal containers Detect: scanning capability/coverage Fix: ownership, dependencies, release Monitor: dashboards, alerting, escalations
  • 10. What Containers? Vendor/MSP containers Containers you rebuild K8s manifests you update
  • 11. What Do We Know Anyway? Patching for 1000s of containers across GKE, Anthos and adjacent products But…our environment constraints help a lot: ● Mandatory use of compiled language ● Mandatory container repo ● Mandatory base images ● Control over code/config pre-submit ● Control over release
  • 12. Container/K8s Delivery Pipeline Source Container Build Package Run Deploy Staging Repo Prod Repo Inventory Dev
  • 13. Good Start: Runtime Detection Source Container Build Package Run Deploy Staging Repo Prod Repo Inventory Dev Runtime scanner: detection and inventory
  • 14. Better: Prevention Complementing Detection Source Container Build Package Run Deploy Staging Repo Prod Repo Inventory Dev Prevention requires integration with pipeline
  • 16. Prevent: Problems So many containers So many dependencies Meeting SLO is hard without reducing volume #Prevent
  • 17. Prevent: Strategy ● Standardize base containers ● Minimal containers: Less code, less vulns, less patching ● Remove unused code: separate build and runtime images ● Two approaches: ○ Start small: Scratch, Distroless, Wolfi/Chainguard Images ○ Slim down: SlimToolkit ● Challenge: apply consistently everywhere #Prevent
  • 18. Our Solution ● Standardize on Distroless ○ Just enough to run golang binaries ● All containers in a single registry ○ Inventory ○ Availability #Prevent
  • 19. Our Solution Source Dev Pre-Submit Check image = gcr.io/gke-release/* deployment.yaml Container /etc/os-release HOME_URL=https://github.com/ GoogleContainerTools/distroless #Prevent
  • 20. Is distroless Alternatives Container Build Package Run Deploy Staging Repo Prod Repo Is distroless image = gcr.io/gke-release/* #Prevent
  • 21. Alternatives: Admission Container Build Run Deploy Prod Repo isDistroless attestation Admission image = gcr.io/gke-release/* Verify isDistroless attestation #Prevent On GKE: Use Binary Authorization
  • 23. Prevent: Summary ● Identify and use enforcement points ● Standardize on patchable base containers ● Standardize on container registries for inventory #Prevent
  • 25. Detect: Problems ● Which containers to scan? ● Which scanner? ○ Different coverage ○ Different vuln sources ○ Duplicate handling ○ Filtering noise ● Which layer has the vuln? #Detect
  • 26. All Container Images Which Container? Our Solution Source Dev List of Containers in Production Scanner Pre-Submit Check Image Fully Patched #Detect
  • 27. Which Container? Alternatives Inventory List of Containers In Production Scanner In registry, but what is running in production? Daemonset Prod Repo Run #Detect
  • 28. Language Pack Scanning SBOM Consumption VEX Support Supplemental CVE Sources Scan programs in your container: ● Rust Cargo.lock ● Python egg files ● Go binaries / go.mod ● etc. Scanners are starting to support SBOMs Filter out remediated vulnerabilities based on VEX More vulns from more places ● OS vendor feeds ● Github Advisories Database ● Language-specific DBs (vuln.go.dev) #Detect Which Scanner?
  • 29. Base Image Detection Reachability Analysis Additional Scans Try to determine the base images: ● From metadata in image manifest ● From the Dockerfile Try to figure out if the code is actually in use: ● Typically uses source ● Can use symbol table in a binary Scan all the things: ● CIS benchmarks ● Hardcoded keys ● Misconfigurations (root user, host volume mounts, etc.) #Detect Which Scanner?
  • 30. False Positives vs. Coverage #Detect Which one is correct? ● Vulnerable module (golang.org/x/crypto/ssh) ● Built with old golang version (1.18.1) ● On old debian base (buster-20210208)
  • 31. Which Scanner? Our Solution Public containers: probably “more than one” Identify gaps and false positives See what our customers see #Detect
  • 32. Detect: Noise ● CVEs that will never be patched (debian CVE-2004-0971, CVE-2005-2541, CVE-2010-4756) ● Ancient low priority vulns without patches (debian CVE-2011-4116, CVE-2016-2781) ● OS vendor has a lower rating than NVD (debian CVE-2022-37434) ● CVE is for a different architecture (golang CVE-2021-38297) ● CVEs that are clearly overrated (CVE-2020-29363: 9.8 down to 7.5) #Detect User Control Scanner Control ● Codepath is unused ● Recent CVEs with no patch
  • 33. Noise: Golang Specific ● Problem: all vulns in whole version or module detected ● Solution: govulncheck can report only reachable vulnerabilities ● go.dev/blog/vuln #Detect
  • 35. Detect: Summary ● Take advantage of new advances in coverage ● Look to your scanner vendor to help with noise ● Use silence/ignore where it fits threat model #Detect
  • 36. Fix
  • 37. Problems: Multi-layer Complex Process CVE 2023-123 in kube-proxy Scanner This Container? Find Owner File Bug/ Send PR Wait For Rebuild Dependent Containers? Done Find Parent yes no no yes #Fix
  • 38. Our Solution: Base Images #Fix gke-distroless debian-base debian-iptables v1 v1 v1 v2 v2 v3 v3 1. Scan the latest base images 2. If fixable vulns, rebuild 3. Repeat for eternity v4
  • 39. Our Solution: Ownership #Fix Find Owner Source Dev Pre-Submit Check Image Ownership File deployment.yaml
  • 40. Our Solution: Simplified Process CVE 2023-123 in kube-proxy Scanner Find owner File Bug/ Send PR Wait for rebuild Done #Fix
  • 41. Summary: Fix ● Track container parent-child relationships ● Automate patching base images ● Comprehensive inventory and ownership ● Use existing ticket systems to track #Fix
  • 43. Monitor: Problems #Monitor Container isn’t patched - who is watching? Who do we escalate to? Which containers have the CVE? Which applications use this container? Are we meeting our SLOs? What are the gaps and pain points? Is CVE-123 patched? Has it rolled out everywhere? What gets measured…
  • 44. Our Solution #Monitor New Finding? Scanner Detects CVE Find Existing Bug yes Nearing SLO? Escalate no Bug Filed Add Comment Past SLO? yes yes
  • 46. Monitor: Visibility #Monitor New Image Created PR Merged Manifest Updated Qualification Starts Rollout Begins Patched Clock starts Dashboards provide status at-a-glance Track progress with metrics over time Measure each step to find pain points Active CVE Count By Image Image Name Image Tag # Fixable CVEs fake-image v1.0.1 55 fake-image v1.0.3 45 demo-image v3.5 20 nginx 1.22.1 10
  • 47. Monitor: Alternatives ● Inventory: Scanners ● Composition: ○ Lyft: Cartography graph database ○ SBOMs / GUAC ○ Ignore layers, just patch: copacetic, crane rebase ● SLO: ○ Bug management software ○ Track commits and rollouts #Monitor
  • 48. Summary: Monitor ● Track SLOs over time ● Track patch/release stages to identify bottlenecks ● Use existing systems for escalation/dashboarding #Monitor
  • 49. Summary ● Standardize on registries and minimal containers ● Enforce as far left as possible ● Scanners for inventory + visibility ● Record ownership of containers ● Auto-patch if possible ● Tickets to track/escalate Prefer automation (doing) over telling
  • 50. Links ● Demo code ● Slim.ai container report ● Lyft patching blogpost ● Separate build and runtime images ● Small images: Scratch, Distroless, Wolfi/Chainguard Images ● SlimToolkit ● AllowedRepos Gatekeeper policy ● Sigstore: signing, policy controller ● GKE Binary Authorization: attestations, image policy ● Opensource scanners: trivy, clair ● Google Container Analysis ● GUAC ● The Seattle Gum Wall Feedback @mrgcastle @gregcastle@infosec.exchange
  • 51. Appendix: Feature Request Wishlist Idea #Detect Prisma Cloud has received 150 reports that dispute this severity Aqua Security has received 231 reports that dispute this severity Google Container Analysis has received 109 reports that dispute this severity If enough users report a critical vuln as inaccurate, the scanner manually evaluates, updates the severity for all their users, and works with NIST to correct NVD #Detect