DevOps culture creates an opportunity for us to improve application security. Since developers are the ones producing code, integrating components and creating the innovations that fuel our digital economy, they are also the ones who will determine whether or not security is part of development or not. Security professionals must therefore learn to how to talk to developers about how to create a security program that will accelerate development and not slow it down.
Show of hands: how many folks in here build software? Okay, how many of you are trying to go DevOps? How many of you have security requirements? (is that the same group of people?)
This talk is for you. I’m going to talk about why security and DevOps historically haven’t gotten along, how you can get on the same page, and what we can do about it.
DevOps started because of the clash between development, who are incented to change, and ops, who are incented toward stability
The Value Stream
Feedback
Use the Developer example taking feedback from a MPT and making code changes to fix
There’s a similar clash brewing between DevOps and Security. Before you can start to talk about securing DevOps, you have to address this culture clash.
That means developers have to get security conscious, and security folks have to stop looking down their noses at DevOps and figure out how to help it move faster, not stand in the way.
There doesn’t have to be a big disconnect between security and development, but to bridge the gap, security has to stop talking in terms of … security.
More specifically, security has to start framing its mission not in terms of eliminating risk, but in terms of helping developers build better software.
I don’t talk a lot with (non-Veracode) engineering stakeholders about Security. I do talk a lot about quality—that’s a concept that spans out of engineering and into everyday life. And even though every security purist I’ve ever met says it’s an over simplification to talk about security as a “subset” of quality—primarily because of the misalignment of skills and resources that that mindset has brought—I think it’s a useful way to think about why security matters.
If quality is building in resilience that the application can deliver its functional mission even in unusual circumstances, then security does the same thing in downright hostile circumstances. And often the cause of a security problem can be traced to and managed in the same way as the cause of a quality problem—a defect, a bug.
To see what I mean, let’s go to a fun data visualization.
So this visualization is a view of all the biggest data breaches over the last few years (at Information is Beautiful, updated September 4)…
And here’s what it looks like when you exclude appsec related breaches -- breaches that were not attributed to configuration errors, hacks, or poor security. And what was the root cause of those hacks?
Data from SOSS 2017
Much of that which is exploited by attackers in like bugs in code
All code also has security vulnerabilities
“I didn't know that I didn't need to address something a certain way”
Of course there's bugs – most developers are not being enabled with education about secure coding, or with the safety nets to catch when coding errors inadvertently introduce security bugs.
Worse yet, some of these bug categories can be discovered late in the game. Think about Java deserialization vulnerabilities—they weren’t a big deal until someone showed how to exploit them last November. You need to keep thinking about security even after you ship.
And on the security side, we need to recognize that anything that helps developers turn around fixes in code faster helps reduce this type of risk, provided it’s done in a systematic way. We need to stop being gatekeepers and start being enablers.
(TIME FILL) Re #2 and #3, somebody made this point during a DevOps talk at O’Reilly: “Security tests in CI/CD need to be binary. It succeeds or it doesn't.” This led to some discussion where some felt there should be a manual step because different apps have different risks, attack surfaces, and threats. Came to realization we were conflating CI/CD with just CI.
(TIME FILL) Re #4, talk about our Security Champions program.
(TIME FILL) Re #5, talk about security telemetry, piggybacking on tools that DevOps teams are already using, except to trigger on security-related events. Some of the stuff Etsy did under NickG and Zane, as an example.
Instead of penetration testing, why not dynamic?
Instead of dynamic, why not static and SCA?
Don’t not do the slower methods, just don’t gate your release on them
Technology: SAST, guided DAST, APIs
Fail quickly: Just like with QA, have as few security tests that run late in the cycle as possible. You want to automate security testing relatively early in the pipeline. Even better, look at doing it before the code hits the pipeline. Development tools that do security testing as you type have gotten a bad rap in the past for being noisy and inaccurate, but there’s a new generation of those coming that address those issues.
Mature Technology: SAST in the pipeline (runtime concerns)
Emerging technology: “instant SAST” or security unit testing – “as you type” (historical concerns about noise)
There are a bunch of different ways to integrate automated security testing into a pipeline, at least as many as there are to build software. Which one is for you depends on your toolchain and the architecture of your app.
No false alarms: The problem with any automated testing is getting the noise level down. Starting with a static analysis tool that is low noise to begin with helps, but you also need to look at what you will allow to stop your pipeline, vs. that which just becomes backlog.
Technology: Static analysis with low FP
Emerging technology: Interactive Application Security Testing (coverage concerns)
Process: Security mitigation review
Build Security Champions: Part of what you want to think about is how you can reduce the input of flaws into the process in the first place. Look for opportunities to drive learning from findings whether through formal education or on the spot reinforcement. Ultimately security champions aren’t enough by themselves, but you can’t get better over time without them.
Mature technology: eLearning and contextual tutorials.
Process and humans: remediation coaching, instructor led training, embedding security into development teams, security in the “definition of done”
Not enough to think about appsec only before you ship:
New vulnerability categories may emerge
Your org may have applications in production that aren’t deployed through the common pipeline
If you’re attacked, you want to feed that information back into development so you can address the issues quickly.
Emerging technology: RASP (Runtime Application Self Protection) provides monitoring as well as the ability to block common kinds of attacks, and runs in the application or the container so it avoids some of the failings of WAFs.
Mature technologies: WAFs (have to be tuned, not generally in the control of the DevOps team); web application discovery; software composition analysis.
If you’re doing appsec today, you’re probably doing it in the Test stage. DevOps and other methodologies with automated build give us the opportunity to integrate into the pipeline (the Build stage). But you should also think about it before any code gets committed, by embedding security into your process and investigating arming your developers with “as you type” security tools. And you shouldn’t stop thinking about it after you ship either – at a minimum keep a bill of software components so you can quickly react when new vulns are found, and consider web application discovery/rapid baselining and RASP to ensure that you understand your perimeter and that you can be alerted to attacks and buy yourself some time to remediate the underlying vulnerabilities. Or maybe you do a blue green deployment and don’t turn everyone on until the code has passed a scan in production.
There are lots of options.
How do you spearhead the movement to securing DevOps? You want to be thinking about this from the top down as well as the bottom up. Who inside the organization can help plant seeds for incorporating security tools, process, and mindset into DevOps?
In the war room (or boardroom): Who is setting the culture for DevOps? Who defines the goals that engineers are going to ultimately be measured against?
In the trenches: Developers rule the kingdom. Who are the people selecting the tools and operating the tool chain?
(TIME FILL) Talk about VSSL and Stash/Gitlab transition as a “developers rule” example.
Image: https://www.flickr.com/photos/eulothg/4922211016 (CC BY-NC-ND 2.0)
Train beyond your walls, i.e. become more educated on DevOps practices in general and CI/CD. Encourage non-security teams to participate in training on security testing and secure coding.
Image: copyrighted, not for distribution
You can download “Five Principles for Securing DevOps” from Veracode.com: https://info.veracode.com/whitepaper-five-principles-for-securing-devops.html