These slides cover application security in an agile/DevOps environment. The following topics are covered: workflow, assurance, SAST, DAST, Threat modeling, Pentesting, code review, mistake proofing, ...

1. The application security aspect
of cyber is very, very tough
Louis Nyffenegger, Ash Fox

2. Agenda
- Introduction
- Assurance
- Devops + Agile
- Agile Security Assurance
- Code Review
- Dynamic Testing
- Hacks

4. About us + disclaimer
We are not talking on behalf of any current or former
employer
No material comes from any current or former employer
This is a roundup of our experiences and lessons learnt
We are not going to cover everything as this is a very big
subject

5. Why this talk?
Application security is challenging
We’ve both been pentesters/consultants (amongst other things) and moved into
internal teams
Consultant
Appsec Eng

6. Assurance

7. Assurance
As an organization, you want assurance that your applications can withstand likely
adversaries
- nation state vs organized crime vs disgruntled hacker
- Think about, what resources do they have? How much will they bring to bear on
you?
- Assurance level = degree of review scrutiny and secure dev and design
- More attacker resources = higher level of assurance required
We gain assurance through a variety of activities that the design and
implementation of a system can withstand these adversaries

8. Assurance Toolbox
Design Review + Threat modelling
Source code review
Penetration testing / Web app Assessment
Developer Training
Bug bounties
SAST and DAST
High Effort
Highly Effective
Less Effort
Less Effective
You cannot review all of the things all of the time

9. Devops, Agile,
Source code
& releases
wut!?

10. Waterfall vs Agile

11. Devops and Agile
We are not going to get overly bogged down defining these
- Agile = see Agile manifesto. Set of development principles for adaptive,
iterative, evolving software development. Scrum, Kanban etc.
- DevOps = about bringing developers closer to production support, breaking
down barriers. Increase speed of building, testing, deploying and breakfix.
Most modern organisations writing code are looking to move faster
- Deploy faster
- Respond to changing markets faster

12. Devops and Agile
Where does your assurance process fit into your development lifecycle?
Microsoft’s Security Development Lifecycle (SDL) is worth looking at
Every organisation does things ‘a little different’
Where do you fit your assurance activities? Where are you catching problems?
Inception -> Elaboration -> Construction (Development) -> Transition
(Deployment)

13. Patching
Patching should be an on-going exercise
You need to patch often so you know you can patch easily and quickly if
necessary
If it’s hard, do it often!
Agile Assurance

14. A faster release cycle means you
can get fixes in production faster!
Agile+Devops should decrease your
“patch time”
The pros

15. Where do those assurance activities fit?
Going to talk a little about our experiences applying those assurance tools in real
life
Insert
Security

16. Design Review + Threat modelling
Really valuable task
Establish a ‘shared understanding’ between developers and security about
realistic threats to their application
Eliminate entire bug classes before they occur
Can be very time consuming, hard to schedule (everyone commences work on
new features at once)
Need to selectively identify features to engage with teams on

17. Employing Design Review + Threat modelling
This activity is most effective before any code is written, ie inception
Can ensure developers know what level of security they need before they even
start
You can identify problems before they happen (how many times do you know what
problems you’re going to find in a feature before even looking at it?)

18. Code review
What we all wish we spend all of our time doing
Can be really effective at identifying vulnerabilities, it’s the “ground truth”
Can be time consuming and grinding
Need to be tactical in terms of how it’s applied, you can’t review all of the code.
You will often ‘depth first’ search down a rabbit hole
Best combined with ‘grey box’ assessment where you can build and do some
dynamic testing of code
Really difficult if you need to learn a new language or framework for every
codebase

19. Employing code review
Really effective when combined with repository monitoring
Really effective when you can find ‘the right time’ to perform it, you don’t want too
much ‘code churn’ whilst you’re reviewing it
Code review feature branches before they’re merged into master

20. Penetration Testing / Web App Assessment
What most of us are used to doing as consultants
Makes a good ‘catch all’
Can identify issues missed in code review, particularly around deployment,
configuration, integration issues. You won’t catch backup.tgz in the webroot during
a code review.
Check out https://flaws.cloud for what can go wrong with your AWS config!
Is often ‘too late’ in Agile development, deploying high security issues = fail
(Fortunately Devops + Agile gives you good opportunity to fix things fast!)

21. Employing Penetration Testing / Web App
Assessment
Do these regularly, try to identify pieces of functionality you’ve missed during
earlier development stages
You’re likely mandated by compliance regimes to do this, use them wisely
Using external testers can help identify your blind spots but it’s not necessarily
needed if you have for example a bug bounty

22. Developer Training
Understand the languages and frameworks your developers are working with
Don’t give contrived examples, ie avoid the SQL injection string concat example in
Java. Every developer scoffs ‘I use hibernate!’
Always good to surprise developers with an ‘a hah!’ moment. Teach them
something cool!
● Crypto attacks
● Tricky bypass
● ...

23. Employing Developer Training
A few ways to implement
- Online services such as secure code warrior and codebashing
- In person
- In-house CTFs
- “Cheat sheets”
Can make it a require to commit to certain repositories (ie, must have PCI secure
dev training to commit to payments repo)
Take note of where you’re finding mistakes and incorporate it into your training.
Are you getting the same question again? Finding the same vulnerability pattern?
Training will help you scale

24. Bug Bounties
Not a replacement for penetration tests, web app assessments or ‘boots on the
ground’. Do not let bug bounties replace your security program.
Great to open a channel of communications with security researchers, build
rapport.
Can be very very noisy. Be prepared to handle it.
Findings are excellent signal, are you focusing your internal efforts in the wrong
place?
Can you tune your tools to eliminate entire classes of vulnerability based on bug
bounty results?

25. Employing Bug Bounties
Recommend using a company such as Bugcrowd/Hackerone, they have
experience that can help you avoid shooting yourself in the foot
Be prepared to facepalm, a lot
Be prepared to pan for the gold
Do something useful with the results, where are most of the problems occurring?
Build a process to handle incoming reports and managing their lifecycle!

26. SAST
Can literally be white noise machines
Require a lot of tuning. High upfront costs but a well tuned SAST can be amazing.
Choose the right SAST, especially for your code base + frameworks etc.
Do not rely on a SAST to catch all of your problems
People are increasingly deploying them ‘close to developers’. This is good but
don’t ever send your developers false positives. You will burn credibility and be
ignored.

27. Employing SAST
What do you want to scan? Every feature branch? On merge to master? Master
every week?
How long does a scan take? Not unheard of taking 5 days.
Tune your results first, disable noisy rules entirely.
Treat your bug bounty findings as inspiration for new rules

28. DAST
Very shallow but can be highly effective (‘ OR ‘1’=’1)
Your adversaries are probably already running them against you
They struggle a lot with modern web applications
- DOM XSS
- OAuth + non form based authentication schemes
The ‘search space’ to identify all of your endpoints, understand how your
applications handle them (is it a filesystem path? A controller?) is huge
DAST findings are likely what low resource adversaries will find first and own with
you

29. Employing DAST
Most “Agile Security” or “Devops Security” talks right now are about using Jenkins to run ZAP scans
This is not a bad strategy, but ensure your scanner actually has the code coverage you think it does.
- Is it authenticating correctly?
- Is it crawling correctly?
You should start with your own very simple automated checks
Has someone messed with X-Frame-Options? Has someone changed our content security policy?
Do not ‘break the build’ on day 1 but this should be your long term strategy. If it’s not, don’t bother with
CI/CD

30. This is hard
This is hard, it takes time and different things work in different environments
Find what works for you and the development teams
Build relationships!

31. Agile
security
hacks

32. CI/CD Overview
Develop Build & Test Deploy

33. CI/CD Overview
Develop Build & Test Deploy

34. Source code
Anyone not familiar with git/version control?
Every company/team is different
○ What’s your branching strategy
○ Work from the strategy to inject security
Depending of your tooling (speed, accuracy), goals and workflow you can use a lot
of different strategies

35. Workflow
● Each dot is a commit
Source:
http://nvie.com/posts/a-successful-git-branching-model/

36. Workflow
Each dot is a commit
Only scan the master
branch?

37. Workflow
Each dot is a commit
Scan all commits?

38. Workflow
Each dot is a commit
Only scan the develop branch
regularly?

39. Workflow
Each dot is a commit
Scan and compare regularly?
○ Scan at t1:
○ Scan at t2

40. Monitoring source code
What else can you do?
○ Find new developers
■ Introduce the security team
■ Enroll for secure coding training
■ Review the first commits to look for obvious bugs
○ Get an “activity indicator” of a repository
■ What is currently actively developed in the company based on the number of commits
■ What is currently dying and may be in a maintenance mode
○ Grep for bugs
○ Detect new features
○ …
○ Verifying signed commit
○ Hardcoded credentials ([hopefully] high-entropy string?)

41. Test & build
Unit/Integrations/.. tests should include security
You can check for dependencies
You can work:
Using the same CI/CD pipeline (and break the build)
Within your own pipeline

42. Deployment
Provisioning of the stack
Deployment of the application
ssh without host-key verification?
Signed artifacts?

43. Deployment: Managing secrets
At the minimum, keep them out of source, store them encrypted (seperate from the keys ideally!). Good
approach is to leverage PKI combined with something like e-heira or e-yaml.
Protect them from an attacker on systems (this is really more about managing trust relationships across
your architecture), Once someone is on the box, they’re getting the auth creds
Different keys in each environment, or prod vs non-prod at a minimum
Make sure you can rotate them! Build this in from the start, ie key versioning
Seems simpler than it is
- “Don’t store passwords in source!” - Ok, then where?
- Encrypt them! Ok, where is the decryption key?
Some frameworks start to offer solutions for this

44. Microservices
Not really anything new
Can now deploy and redeploy services individually. This can make patching easier!
May need to deal with new things architecturally
- Message queues (rabbitmq, kafka)
- Protobuf/thrift
- East/West traffic vs North/South
Less centralization, authentication + authorization needs to be implemented in each service, correctly
Debate around authenticate at the edge or at the service (I advocate at the service, nobody wants a
%252e%252e%252f auth bypass)
Sprawling library dependencies

45. Poka-yoke

46. Poka-yoke == Mistake proofing

48. Poka-yoke on credentials
Paypal credentials (counter example):
○ Test: id=“AVBgyR….TRne...5nc...” secret=”CO1lMKf…”
○ Production: id=“AZihBxea….EI0z7...” secret=”EKzL2SiYamj..”
Stripe credentials:
○ Test: publishable=“pk_test_VeyW...” secret=”sk_test_AhU…”
○ Production: publishable=“pk_live_kWq6Tt7...” secret=”sk_live_Df...”

49. Boring stack
The “Magpie developer” (https://blog.codinghorror.com/the-magpie-developer/) or Hype
Driven Development (https://blog.daftcode.pl/hype-driven-development-3469fc2e9b22)
Use solid frameworks, modern frameworks do a lot for you well
Language, framework, stack sprawl can be a nightmare. It limits re-use of security
assurance work
Increasing depth of the language knowledge. Every language has ‘gotchas’
Principal least surprise, limit WTFs per minute in code

50. Boring stack
The less technologies you have to manage, the more energy you can spend on
each one.
Modern development relies heavily on frameworks, you may almost spend more
time reviewing frameworks than code written in-house
SASTs rarely support the latest technology:
● Languages
● URL mapping
● Property binding (request parameter -> object)
● ...

51. Building blocks
Ensure developers are working with secure components
Ensure they have
- Safe crypto libraries (ie lidsobdium)
- And know which block mode and constructions they should be using
- Escaping and Input validation
- A sane authentication and authorization framework
Especially important for μ-services

52. Managing application vulnerability
1. Find an issue
2. Extract a pattern from the issue:
a. Apply it to other part of the application
b. Appy it to other applications
c. Create training material for it
3. Write the fix
4. Test the fix
5. Write a test case for the issue

53. Conclusion...

54. Conclusion
Computers are hard
Features that never ship are very secure, not great for business though
Need to work smarter leveraging all of the tools are out disposal
We haven’t even began to talk about securing development environments, supply
chain etc.

55. The end.