From ONUG Fall 2022: "Shift Left'' and automation have turned from ideals to meaningless buzzwords. Instead of riding the hype train, let's get real and cover practical and real-world examples taken from actual product security successes. Not every business is the same, neither will their DevSecOps program. In this talk, I'll cover the fundamentals of common to successful DevSecOps programs as well as a grab bag of useful techniques to consider. These are lessons learned doing AppSec at a wide variety of companies including Rackspace, Pearson, a fortune 500 financial, Duo Security and Cognizant Healthcare. Bruce Lee said "Research your own experience. Absorb what is useful, reject what is useless, add what is essentially your own". The goal of this talk is to provide you with enough examples to build your own pragmatic and practical DevSecOps program or maybe absorb a new technique or two into your existing program.

1. Old West Edition
Practical
DevSecOps
Fundamentals of Successful Programs

2. HOWDY!
● Reformed programmer & AppSec Engineer
● Noname Security - Distinguished Engineer
● 14 years in the OWASP Community
○ OWASP DefectDojo (core maintainer)
○ OWASP AppSec Pipeline (co-leader)
○ OWASP WTE (leader)
● 22 + years using FLOSS & Linux
● Currently a Go language fanboy
● Ee Dan in Tang Soo Do Mi Guk Kwan
(2nd degree black belt)
● Founder 10Security

3. Starting from
Zero
Steps 2 Take
Example of how to
expand your horizon
1 Final Thought
Conclusions and key
takeaways
How this whole thing got
started
3 Ways of
DevOps
Crawl, walk run with
DevSecOps
TABLE OF CONTENTS
2
4
3
1

4. Starting
from Zero
1
And maybe heading towards hero

5. Steam Locomotives

6. The Iron Horse Straddles
America
● Radically changed travel in the US
● Travel time across the US
○ Pre-train: 6 months + $1,000
○ Post-train: 1 week + $150
● Town that had a stop prospered
○ Those that didn’t, faded away

7. Trains == Change
● Trains changed the landscape for better or worse
● The US ‘got smaller’ - travel was in reach to more people
● Expanded markets, more customers
● ‘Cost’ of going west went way down

8. Trains <==> DevSecOps
● Trains changed the landscape for better or worse
DevSecOps changed IT for better or worse
● The US ‘got smaller’ - travel was in reach to more people
Batch / change size got smaller (CICD)
● Expanded markets, more customers
Increased agility, more customers
● ‘Cost’ of going west went way down
‘Cost’ of experiments goes way down

9. When will we see this?
DevSecOps AppSec
Your bit of
IT here

10. 3 Ways of
DevOps
2
Before there was DevSecOps, there was DevOps

11. Workflow
?
?
The 3 ways of DevOs
1
3
2

12. Look at your purpose and those
processes which aid it
● Make sure the process is correct from the beginning to end
then look at ways to speed up that process
● Value Stream - the name of the process which provides value to the
business
● Working from left to right e.g. like a timeline
business / development => customer / operations
● Flow [rate] - speed work goes through the process.
#1 Workflow

13. Each Step Repeatable
#1 Workflow
● Remove all haphazard and ad hoc work from the process
● Repeat until stable
○ I like doing the first couple of times manually with a ‘run book’
● Scripting languages are your friends
● Config Mgmt - Puppet, Chef, Salt, Ansible, Terraform, Helm, …
● Create deployable artifacts from a branch/release e.g. .rpm, .deb, .msi, …
● Make sure what you do can be done once or 10,000 times

14. Never Pass along Defects
#1 Workflow
● While working left to right, don’t pass on failures
● Test early and often
● Increase rigor of testing as you work left to right
● When a failure occurs, end that flow and start a new one (after corrections)
● The further right you are, the more expensive failure is
○ Concentrate your early work on the left side (intake)
● e.g. in AppSec, defects are false positive findings

15. Local Optimizations with a
Global View
#1 Workflow
● Ensure no single-step optimizations degrade the overall performance of the
workflow
● Find the bottleneck in your workflow and start there
○ Upstream changes will just back things up
○ Downstream changes won’t manifest since input is constrained
● Each new optimization creates a new bottleneck
○ Iterate on these

16. Increase the flow of work
#1 Workflow
● Make sure you have a well-defined, repeatable process first
● Look for manual steps that can be automated
● Look for duplicate work that can be removed or eliminated
● Measuring/tracking time taken at each step is crucial
● Ask yourself:
Where does the flow ebb?

17. Workflow
Improve Feedback
?
The 3 ways of DevOs
1
3
2

18. Open yourself to upstream and
downstream information
#2 Improve Feedback
● Feedback loops occur when information is gathered from
○ upstream (business / development)
○ downstream (customer / operations)
● Make visible problems, concerns, potential improvements
○ Share this broadly inside the company
● Learn as you move left to right so improvements aren’t lost
● Requests are opportunities to better fulfill business’s needs
○ There’s rarely enough feedback, capture and look for more
○ Feedback collected can be used to optimally improve the system

19. Understand and Respond to your
Customers
#2 Improve Feedback
● Customers are also inside your business
● Customer is more than the end ‘consumer’ at the end of the process
○ Each step is the customer of the previous step
○ Understand what the next steps need from you to succeed
● Remember, feedback isn’t guaranteed
○ Encourage it by responding
● Responses are required of external and internal customers
● Make feedback & responding quick, easy and readily available

20. Shorten Feedback Loops
#2 Improve Feedback
● Remove any intermediaries and impediments to feedback
● Communicate as directly as possible, skipping steps/people if possible
○ e.g . The person who finds a problem communicates with the person
who fixes the problem
● The more hands that hold the feedback, the more change to get garbled
● If possible, intermediaries should be software, not people
● Whispered secret across a classroom
○ How much change occurs?

21. Amplify All Feedback
#2 Improve Feedback
● Shout it from the mountain tops
● No heroes quietly fixing things or applying workarounds
● Open, honest communication of feedback, especially of problems
○ File a bug report
○ Halt the process at that step (e.g. pull the Andon cord to stop the line)
● Make having problems OK and hiding problems a fireable offense
○ GM and all those green status sheets…

22. Embed Knowledge where Needed
#2 Improve Feedback
● Keep specialized knowledge out of people’s heads and into the system
○ Special configurations, business requirements, etc
○ Check it into source control - automatic versioning!
● git blame anyone? Find out where/when regressions occurred
● Moving left to right, keep info in the stage that requires it
● Docs to build a package stored in the repo for that package
● Deploy automation in the repo with configuration templates, etc

23. Workflow
Improve Feedback
Continual Experimentation
& Learning
The 3 ways of DevOs
1
3
2

24. Create a culture of innovation and
experimentation
#3 Continual Experiments
● The fundamentals are now solid, what can your new knowledge buy you?
● The business culture must allow for and embrace innovation &
experimentation
● Two essential things must be understood by the business
○ We can learn from failed experiments / risks we take
○ Mastery comes from repetition and practice
● And you won’t be a master the first N times you practice

25. “I fear not the
man who has
practiced ten
thousand kicks
once,
But I fear the man
who has practiced
one kick ten
thousand times.”

26. Rituals are created that reward
risk taking
#3 Continual Experiments
● Reward risk + learning
● Don’t just talk about rewarding risk, walk the walk
● Trying new things and failing is OK when you gain knowledge
● Consider this creating your own feedback in a very tight loop
● Get real about this:
○ Failures should be noted positively in annual reviews
if and only if a lesson was learned
● Edison invented the lightbulb by running out of things that didn’t work

27. Mgmt allocates time for projects
to improve the system
#3 Continual Experiments
● Plan to improve or you’re planning on stagnation
● Invest in improving the system created
○ By providing value to the business,
it should want to maximize that return
● Prune any technical debt - all debt is not bad
○ Some is good, none has opportunity costs, too much will crush you
● Amplifying feedback helps sell this to the business
● Can keep mistakes from being repeated

28. Faults are introduced to increase
resilience
#3 Continual Experiments
● Practice emergencies so emergencies feel routine
● Think fire drills aka Chaos Monkey
● You need to be a very mature org to do this
● Wonderful feedback look
○ How would your programming change if you knew the DB could
disappear at any second?
● How else to check redundancy?
○ e.g. Think ‘trying to restore from backups’ after they were encrypted by
malware

29. Try crazy or audacious things
#3 Continual Experiments
● Stretch out of your comfort zone
● Requires embracing failures since many of these won’t work
● Forces out-of-the-box thinking
● Provides new perspectives on existing systems
○ You may thing A will break first, but B falls over instead
● Can help find false bottlenecks, bad assumptions, the dreaded “unknown
unknowns”
● Yet another source of feedback so make sure and learn from it publicly

30. The Phoenix
Project
_If you haven’t_
_read that yet…
Right after this talk is over, go out and
get this book & “Beyond The Phoenix
Project” to read them. Period

31. Steps 2 Take
3
Iterate from zero to hero

32. Introducing AppSec Pipelines

33. Key features of AppSec Pipelines
AppSec Pipelines
● Designed for iterative improvement
● Provides a re-usable path for AppSec activities
● Provides a consistent process for both the team and your constituency
● One way flow with well defined states
● Relies heavily on automation
● Grow functionality organically over time
● Gracefully interconnects with the development process (e.g customers)

34. AppSec Pipelines

35. Gen 1 Pipelines
Look at your team’s purpose and
those processes which aid it

36. Spending time
optimizing
anything other
than the critical
resource is an
illusion
_W. Edwards Deming_

37. AppSec Personnelle
Critical Resource
● They are the critical resource
- optimize their work
● Automate things that don’t require a human brain
● Drive up consistency
● Increase tracking of work status
● Increase flow through the pipeline
● Increase visibility and metrics
● Reduce any dev team friction with application security

38. Gen 1 Pipeline

39. Then,
once your house
is in order…

40. Gen 2 Pipelines
Look outside at your team’s purpose and
those processes which aid it

41. Gen 2 Pipeline
Dev Pipeline AppSec Pipeline
Drop tool(s)
into
their pipeline

42. Weaponizing CICD
Gen 2 Pipelines
● Zero false positives
○ Pretend FPs give you anaphylactic shock
● Health Checks vs Scanning
○ Run health checks all the time
● Home of specific issue tests
○ Find a vulnerability, write a test
● Cadence for longer running tests
○ These NEVER break a build
○ Every X builds or every Y days

43. Gen 2 Pipeline

44. (A minor aside)

45. Single source of truth for findings
OWASP DefectDojo
● AppSec Programs, QA / QE, Product Security, Pen Testers
○ Custom report generation
○ Metrics and Dashboards
○ App and Infrastructure findings
○ 150+ security tools supported
● OWASP Flagship project, 8+ years being open sourced from Rackspace
● Community and contributor friendly
○ 305 contributors
● Github: 2.4k stars, 1.1k forks, monthly releases

46. The Heart of your DevSecOps
OWASP DefectDojo

47. Gen 3 Pipelines
Look to scale your team’s reach and
dramatically increase speed and visibility

48. What is a Gen 3 Pipeline?
Gen 3 Pipelines
● A way to conduct AppSec testing in an automated fashion
● Run by the AppSec team for the AppSec team to:
○ Provide visibility into software security
○ Provide security findings to the dev teams
● A means to scale the AppSec team coverage
○ Not in-depth testing but “you must be this high”
○ Allow some testing to be ‘pre-calculated’ for
manual assessments
○ Event-driven (mostly) or on a schedule
● Creates a baseline of security

49. What a Gen 3 Pipeline isn’t:
Gen 3 Pipelines
● Magic pixie dust
● A gate blocking production deploys
● Something to add to existing CICD systems
● Pipelines create artifacts
○ CICD artifacts are a deployed app
○ AppSec Pipeline artifacts are security findings

51. Gen 3 Pipelines

52. Gen 3 Pipelines

53. AppSec Pipeline evolution
1
2
3
Gen 1
Your team's
purpose and
what aids it
Gen 2
Look outside
your teams
purpose and
what aids it
Gen 3
Event Driven,
inter-connected
pipelines

54. Iteration will be required

55. Iteration will be required
Obligatory
coffee stain

56. 1 Final Thought
4
Conclusions and Key Takeaways

57. ● Reverse IP Lookup
● IP reputation
● IP blacklist check
● Domain reputation
● Known IP
(aka we own it)
● Check IP against our
cloud account
OK, AppSec, what else?
DFIR ● Who registered the
domain
● How long registered
● Shodan lookup
● Geolocate IP
● Whois
● Check against threat
intel feed(s)

58. Useage
5,100 Runs
25k+
Container Executions
Size
15 Repos
4 months
Event-driven Pipeline numbers

59. 2014 2015 2016
Number of
Assessments 44 224 414
Headcount N/A -3.5 -2
Percentage
Increase N/A 450% 107%
AppSec Program Numbers

60. 840.91%
Percentage Increase
From 2014 - 2016

61. What are you waiting for…
● Build a Pipeline & remove painful drudgery
from your life
● Co-opt good ideas from other disciplines
● Get your DevSecOps on!

62. TLDR for this talk

63. CREDITS: This presentation template was created by
Slidesgo, including icons by Flaticon, and infographics &
images by Freepik
THANKS!
matt.tesauro@owasp.org
mattt@nonamesecurity.com
Twitter: matt_tesauro
Do you have any questions?