In 2009, we were awakened to Allspaw and Hammond’s “10 deploys a day”. In 2010, Jez Humble and Dave Farley advised us to “build quality in”. But in 2019, breaches hit 24% of software development teams. Are we staking our future on a pace we haven’t yet learned to secure? In a year long collaboration between Gene Kim and Dr. Stephen Magill, we objectively examined and empirically documented software release patterns and cybersecurity hygiene practices across 36,000 commercial development teams and open source projects. Our research uncovered different software development and cybersecurity hygiene behaviors that we categorized as Exemplars, Laggards, Features First, and Cautious. In this session, I will reveal the insights we uncovered. Attendees will learn which techniques, team structures and release patterns exemplary development teams have been championed at large enterprises like ABN AMRO, Walmart, and SEGA, as well as within open source project teams from the likes of Elasticsearch, Mulesoft, and SonarSource. I’ll also share observations of exemplary DevSecOps practices that deliver 50% more commits, release new code 2.4X faster, and remediate security vulnerabilities 2.9X faster.

1. Derek E. Weeks
VP and DevOps Advocate, Sonatype
Co-founder, All Day DevOps
@weekstweets
Exemplars, Laggards, and Hoarders
A Data-Driven Look at Open Source Software Supply Chains

2. …once it ceases to
sacrifice quality for speed
C R E D I T : N E I L B E Y E R S D O R F

3. 2018: Nicole Forsgren

4. 2013: Gene Kim

5. Everyone has a software supply chain.
(including open source projects)

6. OSS download volumes are a proxy for build automation.
@weekstweets

7. 85%
of your code is
sourced from external
suppliers
@weekstweets

8. 313,000
java component
downloads annually
2,778
Component suppliers
8,200
Component release
27,704
8.8% with known
vulnerabilities
@weekstweets

9. …faster is better
in the enterprise

10. …faster is better
for open source?

11. How do you pick
the best suppliers?

12. Two Different Worlds
Enterprise Open Source
Multiple deploys per day Versioned releases
Consistent development team Fluid group of developers
Predictable, well-resourced Variable resource availability
@weekstweets

13. With Similar Metrics
Enterprise Open Source
Deployment Frequency Release Frequency
Organizational Performance Popularity
Mean Time to Restore Time to Remediate Vulnerabilities
@weekstweets

14. With Similar Metrics
Enterprise Open Source
Deployment Frequency Release Frequency
Organizational Performance Popularity
Mean Time to Restore Time to Remediate Vulnerabilities
@weekstweets

15. @weekstweets

16. Attributes Measure
Popularity Avg. daily Central Repository downloads
Size of Team Avg. unique monthly contributors
Development Speed Avg. commits per month
Release Speed Avg. period between releases
Presence of CI Presence of popular cloud CI systems
Foundation Support Associated with an open source foundation
Security More complicated
Update Speed More complicated
@weekstweets

17. @weekstweets
Hypothesis 1
Projects that release frequently have better outcomes.

18. @weekstweets
Hypothesis 1
Projects that release frequently have better outcomes.
(VALIDATED)

19. Projects that release frequently:
are 5x more popular.
attract 79% more developers.
have 12% greater foundation support rates.
@weekstweets

20. With Similar Metrics
Enterprise Open Source
Deployment Frequency Release Frequency
Organizational Performance Popularity
Mean Time to Restore Time to Remediate Vulnerabilities
@weekstweets

21. 1945: W. Edwards Deming

22. @weekstweets
The Key Metrics:
Time to Remediate
Time to Update
Stale Dependencies

23. Security: Time to Remediate (TTR)
@weekstweets

24. Security: Time to Remediate (TTR)
@weekstweets
B Vulnerable Time

25. Security: Time to Remediate (TTR)
@weekstweets
C Vulnerable Time

26. Security: Time to Remediate (TTR)
@weekstweets
C Remediation Time

27. Security: Time to Update (TTU)
@weekstweets
C Update Time (for B)

28. Security: Time to Update (TTU)
@weekstweets
C Update Time (for A)

29. Security: Stale Dependencies
@weekstweets
Stale Dependency

30. Time to Remediate Security Vulnerabilities

31. Time to Remediate Security Vulnerabilities
Do these update
quickly in general?

32. Time to Remediate vs. Time to Update Dependencies (TTU)
@weekstweets
Most projects stay secure by
staying up to date.

33. Hypothesis 2
Projects that update dependencies more frequently
are generally more secure.
@weekstweets

34. Hypothesis 2
Projects that update dependencies more frequently
are generally more secure.
(VALIDATED)
@weekstweets

35. Hypothesis 3
Projects with fewer dependencies will stay more up to date.
@weekstweets

36. Hypothesis 3
Projects with fewer dependencies will stay more up to date.
(REJECTED)
Components with more dependencies actually have better MTTU.
@weekstweets

37. More dependencies
correlate with larger
development teams.
@weekstweets
Larger development
teams have 50% faster
MTTU and release 2.6x
more frequently.

38. More dependencies
correlate with larger
development teams.
@weekstweets
Larger development
teams have 50% faster
MTTU and release 2.6x
more frequently.

39. @weekstweets
Hypothesis 4
More popular projects will be better about staying up to date.

40. 1999: Eric S. Raymond

41. @weekstweets
Hypothesis 4
More popular projects will be better about staying up to date.
(REJECTED)
There are plenty of popular components with poor MTTU.
Popularity does not correlate with MTTU.

42. 5 Behavioral Clusters for OSS “Suppliers”
@weekstweets
Small Exemplar
(606)
Large Exemplar
(595)
Laggards
(521)
Features First
(280)
Cautious
(429)
Small development
teams (1.6 devs),
exemplary MTTU.
Large development teams (8.9
devs), exemplary MTTU, very
likely to be foundation
supported,
11x more popular.
Poor MTTU, high stale
dependency count, more
likely to be commercially
supported.
Frequent releases, but
poor TTU.
Still reasonably
popular.
Good TTU,
but seldom
completely up
to date.
Rest of the population: 8,142

43. Exemplars release
fast and tend to be
more popular.
Pick suppliers from
here.

44. Not all popular
projects are
exemplary and
release fast
Don’t pick suppliers
from here.

45. We schedule updating
dependencies as part of our
daily work
We strive to use the
latest version (or latest-
N) of all our dependencies
We use some process to
add a new dependency
(e.g., evaluate, approve,
standardize, etc.)
We have a process to
proactively remove
problematic or unused
dependencies
We have automated tools
to track, manage, and/or
ensure policy compliance of
our dependencies
46%
YES
50%
YES
30%
YES
37%
YES
Enterprise Developers Manage Dependencies
@weekstweets
n = 658
38%
YES

46. When you climb the mountain every day, it’s easier.
@weekstweets

47. @weekstweets

48. @weekstweets

49. 2010: Jez Humble

50. Automation continues
to prove difficult to
ignore.
Do you have an open source policy and
do you follow it?

51. For organizations who
tamed their supply
chains, the rewards
were impressive.
@weekstweets

52. @weekstweets
imagine quality at insane speed

53. @weekstweets
imagine automated rating
of OSS projects

54. @weekstweets
imagine automated rating
of OSS projects

55. @weekstweets
imagine automated rating
of OSS projects

56. @weekstweets
imagine secure applications

57. @weekstweets
imagine machines
developing their own code

58. weeks@sonatype.com