DevOps - its all about doing the right thing, much like the teachings in the Bible. A quick overview of DevOps, how many of the tenants of DevOps are shared with Christianity and how Pearson is putting DevOps into AppSec with an AppSec Pipeline.

1. DevOps
from a Christian Perspective

2. 5 months with Pearson
Application Security Lead Engineer
Prior to Pearson
● Rackspace - Lead Engineer, Product Security
● AppSec consulting
o VP Services, Praetorian
o Consultant Trustwave’s Spiderlabs
● TEA - Senior Security Engineer
● DIR - Penetration Tester
● Texas A&M University
o Systems Analyst, Sys Admin, Developer, DBA
o Lecturer in MIS department
● Viatel -
Internet App Developer
Who am I?

3. Other professional experience
● OWASP Live CD / OWASP WTE
o Project lead 2008 to present
o Over 300K downloads
o http://appseclive.org
● OWASP Foundation Board of Directors
o International charity focused on improving software security
● Multiple speaking engagements internationally
at AppSec, DHS, ISC2, SANS… conferences
● Application Security Training internationally
● B.S. Economics, M.S. in MIS
o Strong believer in the value of cross-
discipline study
Who am I?

4. A Christian
● Life long Lutheran
o Lutheran private schools through 8th
Grade
o Pro Bono consulting work for Lutheran Foundation of TX
o My kids attend Cross Lutheran School in New Braunfels
● Prefer contemporary to traditional services
● I've never had that big “aha” moment
o Many times in my life I had to lean upon my faith
●
Kids illnesses
●
Hospice care
●
Parents illness
Who am I?

5. DevOps

6. The old way...
Very early and prescriptive requirements and design
Long development cycles
Waterfall Approach
Groups work in Silos - Dev, SysAdmin, QA, Security
Possible feedback from bug reports but little else
Throwing code over the wall
Traditional Software Dev & Ops

7. Waterfall Development

8. Why waterfall can be prone to failure
Very difficult to capture all the requirements...
- before design is complete
- some may not surface until implementation
All features “locked in” during early stages
Operational considerations occur very late
- Ops resorts to 'work arounds' and ad-hoc installs
- Dev assumptions may violate Ops policy
“Worked on my laptop...”
Waterfall Development

9. Waterfall Development – from this

10. Waterfall Development – to this

11. Why DevOps came to be
What's different about DevOps
Web/Cloud companies needed
- high availability
- fast introduction of new features
Easy for users to switch to a competing service + fist mover advantage
No media to ship with SaaS models
Cultural change – not just new cool tech aka CI/CD, Docker...
Focus on clear business objectives
Dev and SysAdmins share responsibility for uptime, deploys, downtime
Emphasize people and process, repeatability
Goal is better uptime and lower operational costs
The DevOps Answer

12. The Phoenix Project
3 Ways of DevOps
Strategies for Improving
Operations

13. Workflow
The 3 Ways of DevOps
1
2
3

14. Look at your purpose and those process which aid it
From the Bible
Make sure the process is correct from beginning to the end
Then look at ways to speed up that process
Value Stream – the name a the process which provides value to the business
Working from left to right – think of a time line:
business / development => customer / operations
Flow [rate] – the speed work goes through the process
I make known the end from the beginning, from ancient times, what is
still to come. I say, 'My purpose will stand, and I will do all that I
please.' - Isaiah 46:10
The end of a matter is better than its beginning; Patience of spirit is
better than haughtiness of spirit. - Ecclesiastes 7:8
#1 - Workflow

15. An example workflow
Software release process
● Code written
● Code committed to a code repository
● Unit test the code
● Package the code for deployment
● Integration testing
● Deploy code to production
#1 - Workflow

16. Making things repeatable
Remove all haphazard and ad hoc work from the process
Repeat until stable, I like doing the first couple times manually with a 'run book'
Scripting languages are your friends
Config Mgmt – Puppet, Chef, Salt, Ansible, Jenkins, CFEngine, …
Creating deployable artifacts from a branch/release aka .rpm / .deb / .msi
Make sure what you do can be done on 1 server or 10,000 servers
Repetition is the mother of skill
- Master Bret Riley, Sa Bom Nim (Master Instructor TSD-MGK)
#1 - Workflow
Each Step Repeatable

17. Work left to right but don't pass on failures
From the Bible
Test early and often
Increase the rigor of testing as you work left to right
When a failure occurs, end that flow and start a new one after corrections
The further right you are, the more expensive failure is
So whoever knows the right thing to do and fails to do it, for him it is
sin. - James 4:17
The King will reply, ‘Truly I tell you, whatever you did for one of the
least of these brothers and sisters of mine, you did for me.
- Matthew 25:40
#1 - Workflow
Never Pass on Defects

18. Your fix cannot be my new problem
From the Bible
Ensure no single-step optimizations degrade the overall performance
of the workflow
Spending time optimizing anything other than the critical resource is
an illusion.
Find the bottle neck in your workflow and start there
- Upstream changes will just back things up
- Downstream changes won't manifest since input is limited
Each new optimization creates a new bottleneck – iterate on this
So whatever you wish that others would do to you, do also to them,
for this is the Law and the Prophets. - Matthew 7:12
#1 - Workflow
Local optimizations with a global view

19. Now go faster
From the Bible
Make sure you have a well-defined, repeatable process first
Look for manual steps that can be automated
Look for duplicate work that can be removed/eliminated
Measuring/tracking time taken at each step is crucial
Where does the flow ebb?
Give, and it will be given to you. A good measure, pressed down,
shaken together and running over, will be poured into your lap. For
with the measure you use, it will be measured to you. - Luke 6:38
#1 - Workflow
Increase the flow of work

20. Workflow
Improve Feedback
The 3 Ways of DevOps
1
2
3

21. Open yourself to upstream and downstream information
From the Bible
Feedback loops occur when information is gathered from
- upstream (business / development)
- downstream (customer / operations)
Make visible problems, concerns, potential improvements – share this publicly
Learn as you move left to right so improvements aren't lost
Requests are opportunities to better fulfill the needs of the business
There is rarely enough feedback, capture and look for more
Feedback collected can be used to optimally improve the system
For there is nothing hidden that will not be disclosed, and nothing
concealed that will not be known or brought out into the open.
- Luke 8:17
#2 – Improve Feedback

22. Customers are also inside your business
From the Bible
Customer is more then the 'consumer' at the end of the process
- Each step is the customer of the previous step
- Understand what the next steps need from you to succeed
Remember, feedback isn't guaranteed - encourage it by responding
- Responses are required of external and internal customers
Make feedback & responding quick, easy and readily available
Where there is no guidance, a people falls, but in an abundance of
counselors there is safety. - Proverbs 11:14
My dear brothers and sisters, take note of this: Everyone should be
quick to listen, slow to speak and slow to become angry. - James 1:19
#2 – Improve Feedback
Understand and respond to your customers

23. Remove any intermediaries and impediments to
feedback
From the Bible
Communicate directly as possible, skipping steps/people if possible
- e.g. The person who finds a problems communicates with the person who
can fix the problem
The more hands that hold the feedback, the more chance to get garbled
If possible, intermediaries should be software not people
Whispered secret across a classroom, how much change occurs?
A person finds joy in giving an apt reply — and how good is a timely
word! - Proverbs 15:23
#2 – Improve Feedback
Shorten Feedback loops

24. Shout it from the mountain tops
From the Bible
No heroes quietly fixing things or applying workarounds.
Open, honest communication of feedback, especially of problems
- File a bug report
- Halting the process at that step (pull the cord to stop the line)
Public feedback == full knowledge to solve the problem in the optimal way
Make having problems OK and hiding problems a fireable offense
Cease to hear instruction, my son, and you will stray from the words
of knowledge. - Proverbs 19:27
Let the wilderness and its towns raise their voices; let the settlements
where Kedar lives rejoice. Let the people of Sela sing for joy; let them
shout from the mountaintops. - Isaiah 42:11
#2 – Improve Feedback
Amplify all feedback

25. Go all in
From the Bible
Keep specialized knowledge out of people's heads and into the system
- special configurations, business requirements, etc
- Check it into source control – automatically versioned.
- git blame anyone? You can find out where/when regressions occurred
Moving left to right, keep needed info in the stage that requires it
- Docs to build a package stored in the repo for that package
- Deploy automation in repo with configuration templates, etc
Gold there is, and rubies in abundance, but lips that speak knowledge
are a rare jewel. - Proverbs 20:15
#2 – Improve Feedback
Embed knowledge when needed

26. Workflow
Improve Feedback
Continual Experimentation and Learning
The 3 Ways of DevOps
1
2
3

27. Create a culture of innovation and experimentation
From the Bible
The fundamentals are now solid, what can your new knowledge buy you?
The business culture must allow for and embrace innovation / experimentation
Two essential things must be understood by the business and all involved
- We can learn from the failed experiments and risks we take
- Mastery comes with repetition and practice
and you won't be a master the first N times you practice
The mind of the discerning acquires knowledge, and the ear of the
wise seeks it. – Proverbs 18:15
But be doers of the word and not hearers only, deceiving yourselves.
– James 1:22
#3 – Continual Experimentation & Learning

28. Reward risk + learning
From the Bible
Don't just talk about rewarding risk, walk the walk
Trying new things and failing is OK when you gain knowledge
Consider this creating your own feedback in a very tight loop
Get real about this – failures should be noted positively in annual reviews
if and only if a lesson was learned
Edison invented the lightbulb by running out of things that didn't work
Be very careful, then, how you live—not as unwise but as wise,
making the most of every opportunity, because the days are evil.
Therefore do not be foolish, but understand what the Lord’s will is.
Ephesians 5:15–17
#3 – Continual Experimentation & Learning
Rituals are created that reward risk taking

29. Plan to improve or you're planning on stagnation
From the Bible
Invest in improving the system created
- By providing value to the business, it should want to maximize that return
Prune any technical debt – all debt is not bad
- some is good, none has opportunity costs, too much will crush you
Amplifying feedback helps sell this to the business
Can keep mistakes from being repeated
For I know the plans I have for you,” declares the LORD, “plans to
prosper you and not to harm you, plans to give you hope and a future.”
- Jeremiah 29:11
There is a time for everything, and a season for every activity under the
heavens – Ecclesiastes 3:1
#3 – Continual Experimentation & Learning
Mgmt allocates time for projects to improve the system

30. Practice emergencies so emergencies feel routine
From the Bible
Fire drills aka Chaos Monkey
You need to be a very mature org to do this
Wonderful feedback loop
- How would your programming change if you knew the DB could go away
at any time?
How else can you check redundancy? Think trying to restore from backups
For God gave us a spirit not of fear but of power and love and self-
control. - 2 Timothy 1:7
(Yeah, a bit of a stretch)
#3 – Continual Experimentation & Learning
Faults are introduced to increase resilience

31. Stretch out of your comfort zone
From the Bible
Requires embracing of failures since many of these won't work
Forces out-of-the-box thinking
Provides new perspectives on existing systems
- You may think A will break first, but B falls over instead
Can help find false bottlenecks, bad assumptions, the dreaded unknown unknowns
Yet another source of feedback so make sure and learn from it publicly
Take pains with these things; be absorbed in them, so that your
progress will be evident to all. - 1 Timothy 4:15
#3 – Continual Experimentation & Learning
Try crazy or audacious things

32. Everything shouldn't be bigger in Texas
I got nothing for this one...
Do small releases frequently
- Release become ordinary not extraordinary
- Feedback loops are quick, positive changes can happen quicker
- Bugs are easier to find & fix in a smaller code base / diff
Reduction in code latency (code latency is how long written code is idle)
- Customers won't see new features until deployed. Happy Customer == $$
- Start making returns on your coding investment – aka ROI
Feels counter-intuitive but bigger changes == more complexity, less practice,
customer wait for features/bug fixes which means more risk
A pebble every day or a boulder every quarter?
Bonus Material
Small Batches Are Better

33. What Does DevOps
Mean For My AppSec
Program?

34. The AppSec Pipeline

35. Key Features of AppSec Pipelines
● Designed for iterative improvement
● Provides a reusable path for AppSec activities to
follow
● Provides a consistent process for both the team and
our constituency
● One way flow with well-defined states
● Relies heavily on automation
● Has the ability to grow in functionality organically
over time
● Gracefully interconnects with the development
process

36. Spending time
optimizing anything
other than
the critical resource
is an illusion.

37. Key Goals of AppSec Pipelines
• Optimize the critical resource - AppSec personnel
● Automate all the things that don’t require a
human brain
● Drive up consistency
● Increase tracking of work status
● Increase flow through the system
● Increase visibility and metrics
● Reduce any dev team friction with
application security

38. Pipeline - Intake
• “First Impression”
• Major categories of Intake
• Existing App
• New App
• Previously tested App
• App to re-test findings
• Key Concepts
• Ask for data about Apps only
once
• Have data reviewed when an
App returns
• Adapt data collected based
on broad categories of Apps

39. Pipeline – the Middle
● Inbound request triage
● Ala Carte App Sec
● Dynamic Testing
● Static Testing
● Re-Testing mitigated
findings
● Mix and match based on risk
● Key Concepts
● Activities can be run in
parallel
● Automation on setup,
configuration, data export
● People focus on customization
rather than setup

40. Pipeline – the End
● Source of truth for all AppSec
activities
● ThreadFix is used to
● Dedup / Consolidate findings
● Normalize scanner data
● Generate Metrics
● Push issues to bug
trackers
● Report and metrics automation
● REST + tfclient
● Source of many touch points with
external teams

41. Why we like AppSec Pipelines
● Allow us to have visibility into WIP
● Better understand/track/optimize flow of
engagements
● Average static test takes ...
● Great increase in consistency
● Easier re-allocation of engagements between staff
● Each step has a well defined interface
● Knowing who has what allows for more informed
“cost of switching” conversations
● Flexible enough for a range of skills and app maturity

42. Books to read...

43. The Phoenix Project The Practice of Cloud System
Administration
Gene Kim, Kevin Behr and George Spafford
Books to read
Thomas A. Limoncelli, Strata R. Chalup,
Christina J. Hogan

44. The Bible The Shack
Books to read
William P. Young

45. Thank you !