This document summarizes Cameron Townshend's presentation on securing software supply chains. It discusses how software is increasingly comprised of open source components, with developers downloading thousands of packages annually. However, not all components are of equal quality, and many downloads contain known vulnerabilities. The document argues that organizations need greater visibility into their software bills of materials, and to automate processes to continuously identify risks from open source and remediate vulnerabilities across the entire software development lifecycle.
4. W. Edwards Deming, 1945
What is software supply chain management?
A new (yet proven) way of thinking.
1. Source parts from fewer and better suppliers.
2. Use only the highest quality parts.
3. Never pass known defects downstream.
4. Continuously track location of every part.
9. 59,000 data breaches
have been reported to GDPR regulators since May 2018
source: DLA Piper, February 2019
10. 10
Business applications are under attack…
Of enterprises suffered at
least one breach in last 12
months.
51%
Of enterprise attacks are
perpetrated by external
actors.
43%
Of external attacks target
web apps and known
vulnerabilities.
68%
Forrester: Best Practices for Deploying And Managing Enterprise Password Managers – Jan 2018
11. Everyone has a software supply chain.
(even if you don’t call it that)
26. Social normalization of deviance
“People within the organization become so much accustomed to a
deviant behavior that they don't consider it as deviant, despite the
fact that they far exceed their own rules for elementary safety.”
Diane Vaughan
27. Breaches increased 71%
24%
suspect or have verified a
breach related to open source
components in the 2019 survey
14%
suspect or have verified a
breach related to open source
components in the 2014 survey
source: DevSecOps Community Survey 2014 and 2019
28. The speed of exploits has compressed 93%
Sources: Gartner, IBM, Sonatype
29. source: 2019 DevSecOps Community Survey
Quickly identify who is faster than their adversaries
30. March 7
Apache Struts releases
updated version to
thwart vulnerability
CVE-2017-5638
Today
65% of the Fortune 100
download vulnerable
versions
3 Days in March
March 8
NSA reveals Pentagon
servers scanned by
nation-states for
vulnerable Struts
instances
Struts exploit published
to Exploit-DB.
March 10
Equifax
Canada Revenue Agency
Canada Statistics
GMO Payment Gateway
The Rest of the Story
March 13
Okinawa Power
Japan Post
March 9
Cisco observes "a high
number of exploitation events."
March ’18
India’s AADHAAR
April 13
India Post
December ’17
Monero Crypto Mining
Equifax was not alone
31. Complete software bill of materials (SBOM)
2019 No DevOps Practice 2019 Mature DevOps Practices
19%
50%
Source: 2019 DevSecOps Community Survey
34. 1.3 million vulnerabilities in OSS components undocumented
No corresponding CVE advisory in the public NVD database
35. July
2017
8
3
10
4
The new battlefront
Software Supply Chain Attacks
Study found credentials online affecting publishing
access to 14% of npm repository. +79,000
packages.
Malicious npm Packages “typosquatted” (40
packages for 2 weeks. Collecting env including
npm publishing credentials).
1
10 Malicious Python packages
Basic info collected and sent to
Chinese IP address
2
Golang go-bindata github id deleted and
reclaimed.
5
ssh-decorator Python Module stealing private ssh
keys.
7
npm event-stream attack on CoPay.11
Sep
2017
Homebrew repository compromised.
9
Jan
2018
Feb
2018
Mar
2018
6
Aug
2018
Conventional-changelog compromised
and turned into a Monero miner.
Blog: “I’m harvesting credit card numbers
and passwords from your site. Here’s
how.”
Backdoor discovered in npm get-cookies
module published since March.
Unauthorized publishing of mailparser.
Gentoo Linux Repository Compromised.
Malicious Eslint discovered to be stealing npm
credentials.
Aug
2017
Oct
2017
Nov
2017
Dec
2017
Apr
2018
May
2018
Jun
2018
Jul
2018
Sep
2018
Oct
2018
Nov
2018
Dec
2018
36. At what point in the development process does your
organization perform automated application analysis?
2019 No DevSecOps Practice 2019 Mature DevSecOps Practices
43. 1. An up to date inventory of open-source components utilized in the
software
2. A process for identifying known vulnerabilities within open source
components
3. 360 degree monitoring of open source components throughout the
SDLC
4. A policy and process to immediately remediate vulnerabilities as
they become known
January 2019
source: https://blog.pcisecuritystandards.org/just-published-new-pci-software-security-standards
46. 47
1985
RMS/GNU/FSF
1990
Scrum / Linux
1995
Java / JavaScript
2000
LAMP Stack
2005
Agile Everything
2015
Rise of DevOps
2018
450B OSS components
requested annually
Apache Maven (9M)
Central Repo (80B)
Nexus Repo OSS (100K)
Nexus Intelligence(8B)
4th Annual State of the Software Supply Chain Report
History in motion.
Writing code is out. Borrowing code is in.
47. Repository Pro
Manage libraries, build artifacts, and release
candidates across the SDLC.
Say hello to the Nexus Platform.
Automatically enforce open source policy and control risk across every phase of the SDLC.
Auditor
Examine OSS components within
production apps.
Firewall
Automatically stop risk from
entering your SDLC.
Lifecycle
Continuously identify risk, enforce policy,
and remediate vulns across every phase of
your SDLC.
The trend for companies to change can be seen here. Since 2010 to 2018 the top 10 brands has changed dramatically. We now see that the 4 most popular brands in the world are all tech companies and their dramatic growth has been driven by Software and Open Source Software. Apple, Google, Amazon and Microsoft.
It hardly seemed like the start of a revolution, but oh boy, it was
in 1945, when W. Edwards Deming started advising Japanese manufacturers to
detect and fix defects at the beginning of the manufacturing process.
Within five years, companies Mitsubishi and Toyota Motor Co. had become disciples.
By the 1960, Deming’s TQM practices were an intrinsic part of the Japanese culture and were playing rise to their global dominance.
In 1981, Ford adopted these principles and within 6 years became the most profitable US auto manufacturer
Now tied into high-performance production processes, six-sigma manufacturing today aims a defect rate goal of 3.4 parts per million.
“Cease dependence on mass inspection.”
Emphasize performance of the entire system and never pass a defect downstream
Inspection does not improve quality. Nor guarantee quality. Inspection is too late.
Harold F. Dodge: “You cannot inspect quality into a product”
Automatic inspection and recording require constant vigil.
It was then no mistake in 2010, when Jez Humble and Dave Farley advised people to “Build Quality In” in their seminal book “Continuous Delivery”
Build quality in
If you are going to be fast you have to build quality in
as people heard about and strove to achieve Allspaw’s 10 deploys a day.
Feedback from releases
Single object is built, tested and deployed, you do not build for each environment
You learn from releases – share story of MunichRe 2 releases a year and both were disasters, my failure at the CAB
It hardly seemed like the start of a revolution three years later
when Gene Kim shared the Three Ways of DevOps inside The Phoenix Project,
with the first way being
The principles of Flow, which accelerate the delivery of work from Development to Operations to our customers “Emphasize performance of the entire system end-to-end and never pass a defect downstream.”
The principles of Feedback, which enable us to create ever safer system of work;
The principles of Continual Learning and experimentation, which foster a high-trust culture and a scientific approach to organizational improvement as part of our daily work.
Since then, the quest for speed in software manufacturing has been a holy grail.
In our 2019 DevSecOps community survey of 5,500 people, 47% reported their ability to deploy multiple times a week.
We’ve forgotten something along the way.
The European Union has reported 59,000 report data breaches since GDPR started in may 2018
People attacking applications
Crack in the system
Everyone uses the software supply chain.
You rely on suppliers that are writing code for you and do you know what they have written?
There's a really interesting site out there called moduleaccounts.com. It has a simple value, it keeps track of the number of different components, or packages that are available across the different development languages, from pipi, to nuget, to bower, to maven, components, etc. And it shows the increase in the number of these components that are available to the developer ecosystem, or the developer population, over time. We used some data from that site to see that over a thousand new open-source projects were created each day. People delivering a new kind of software, a new kind of component.
Then, from the general population of all open-source projects worldwide, we were able to estimate that ten thousand new versions of components are introduced every day. There's this huge supply of components entering the ecosystem, and available to our software supply chains. When we look at the central repository that Sonatype manages, of maven style or java open-source components, we looked across 380 thousand open-source projects, and found that on average those projects were releasing fourteen new versions of their components every year. That's great from a supply chain aspect, that the suppliers are very active, actively releasing new software, actively releasing new innovations, and actively improving the software that they're making available to developers worldwide.
This is from the State of the Software Supply Chain report
Supply - Sonatype runs Maven Central. We saw 146B downloads from Maven Central in 2018. That is almost 100% increase from 2017 figure of 87 B
Laurie Voss f
6M developers downloaded 7B Javascript
Supply - NPM saw 7B downloads per week. That is 350B downloads per year.
Demand - In a modern application we see that 85% of the code is now sourced from 3rd parties.
Move faster by standing on the shoulders of others
Demand - The average firm downloads 170,000 components annually. 3500 unique different external suppliers?
Demand - Npm advises us that the average Javascript developer downloads 60,660 separate java packages per year
There’s a reason why new components come out
Java - We found that 5.5% of all downloads were known vulnerable versions in 2016
It’s actually gotten worse. 1 in 10 of downloads had a known vulnerability when they were downloaded
Are you doing 3ways of devops.
You’re not paying attention to this.
6sigma – 3.4 parts per million
NPM – advised that 51% of their downloads in October 2018 are vulnerable. Get the exact source
Laurie Voss
6sigma says that in manufacturing you will have 3.4 defects per million
170,000 * 11.1% = 18870 vulnerable libraries
66600 * 51% = 30,936
Give example of the Challenger space shuttle tragedy Oring
Between 2014 and 2019 we saw 71% increase in breaches from open source libraries
Heartbleed 2014 – CVSS score of 5
2019 – struts was a CVSS score of 9
Attack window has shrunk from 45 days to 3days
Leading organisations can release a feature or a patch multiple times a week.
Struts 2 attacks
CVE published – new version published in March
Struts exploit published to Explout-db
Struts has a well known signature
65% of global 100 are still downloading vulnerable versions of Struts
Visibility is the starting point. Do you have a complete Bill of Materials
Are we using struts
And where is it
Struts 2 download behavior has not changed. The vulnerable version is still being downloaded
Ever since 2009 when John Aspaw shared Etsy’s practice of 10 deploys a day, the rest of the development industry has been trying to catch up.
11 – 200,000 malicious event-stream downloaded Nov 2018
Do you have a Open Source Governance Policy and do you follow it?
Devops – automation is hard to ignore. Once you surface this to the developer.
In the 2018 State of the software Supply Chain
We analysed 60000 application
We found 11.7% flowing in unmanaged
In managed supply chain we saw 6.1% so we improved 50%
You are not allowed to put known defective parts in to a manufacturing process and ship it to customers. But it is not illegal to ship to a customer.
OWASP top 10 developed a number of years ago as a starting point
-FDA requires a BOM for any approval
-PCI – requires BOM
-Singapore -MAS
The new standard requires organizations to govern their use of open source software, and it states that any application utilized as part of the payment process, must be secure by design.
https://blog.sonatype.com/hygiene-for-open-source-softwareis-now-a-pci-requirement
“Cease dependence on mass inspection.”
Inspection does not improve quality. Nor guarantee quality. Inspection is too late.
Harold F. Dodge: “You cannot inspect quality into a product”
Automatic inspection and recording require constant vigil.