Much attention is focused to the technology aspects of DevOps, including how to automate security testing. But DevOps is as much a cultural movement as anything else, with a strong focus on feedback loops and continuous improvement. How can organizations implement these aspects of DevOps culture when integrating security, given the massive shortfall in skilled information security personnel? This talk discusses organizational and cultural aspects of DevOps with an emphasis on the role of “security champions”—developers cross-trained in information security basics—in executing a successful DevSecOps transformation.
As noted up front, this is a talk about people and organizational factors. If you think that DevOps begins and ends at the pipeline and the technologies that plug into it, you may want to wait for the recap on Twitter. And this talk is definitely not for you if your job description for your AppSec team includes configuring, tuning, and running code and web app scanners by hand.
For those of you who have had one or more transformation efforts fail due to change management failures, staff shortages, or other organizational issues—and for those who have one or more automated AppSec tools and are wondering how to make them successful—this talk is for you.
I think none of us would be at this webinar if we didn’t think there is transformational power in DevOps. But for a lot of teams facing the DevOps transformation, the potential comes with a lot of anxiety.
After all, we’re talking about collapsing silos. In the real world, when that happens, people die.
This goes double for integrating Security into DevOps. But given that DevOps has already absorbed development, operations, QA and release engineering, why should the security transition be harder?
The reason is culture clash. More than any other organization, Security has been publicly skeptical of the benefits of DevOps. They’ve seen a lot of development processes come and go, and insecure software keeps getting built.
h/t @petecheslock, DevOpsDays Austin
Let’s put a finer point on the skepticism. Security has traditionally been a discipline of controls and gates. But there’s no room for this sort of manual, expert-driven check in DevOps, just as there’s no room for weeks of manual quality assurance testing if you want to ship code several times a day.
That means developers have to get security conscious, and security folks have to stop looking down their noses at DevOps and figure out how to help it move faster, not stand in the way.
As with a lot of other things, Gene Kim’s Three Ways help us to see the problem:
We need to see software delivery as an end to end system that includes security as part of the problem.
Security needs to help create some of the feedback loops from production (and earlier stages of the process), and needs to help Dev interpret and act on them.
And security needs to participate in the culture of continual experimentation and learning.
A big part of the challenge of meeting the need for security to participate in DevOps is math: specifically, for every four people employed in infosec, there are three additional job openings.
You can’t hire an infosec team big enough to define security requirements, perform secure code reviews, interpret security testing results, and handle security alerts for all your applications in your entire portfolio if you’re operating at DevOps speed.
Theory of Constraints: in an end to end flow, identify the constraint, exploit, subordinate, elevate, then repeat
What is the constraint in DevSecOps?
Define requirements --> Develop code --> Review code --> Build code --> Test code --> Handle bugs
(Almost) all have some touch between security and development
First pass: Security is the constraint
Exploit constraint – maximize throughput with processes like security reviews etc.
Subordinate constraint – implicitly done either by minimizing the amount of security changes that happen or adding surge capacity
Elevate constraint – Optimize handoffs, change the process
So to put it another way, once you’ve optimized your Dev and Ops processes, Security becomes the constraint.
Identify
Exploit
Subordinate
(This theory, by the way, is why I say that this talk isn’t for you if you are still running your scanners by hand. If you’re doing that, you haven’t even begun to exploit the constraint of your security team yet.)
Elevate? How do we do that? The definition is “In this step, more substantive changes are implemented to “break” the constraint. These changes may necessitate a significant investment of time and/or money” – and may involve targeted review of lost productive time, tactical actions, design and/or component upgrades, and supplementing the constraint with additional throughput.
But how does that work if you don’t have enough people in the first place?
You do it by making more people who can do the work that security is doing
Enter Security Champions – members of the DevOps team who can perform lower-skill, high frequency jobs to take load off the security team and help DevSecOps to scale. You can think of this role as essentially a new guild, like release engineers or folks who work on unit tests.
Some examples of things security champions can take on (we’ll talk about a few of these):
Peer reviews
Security grooming
Code reviews (very specific topics based on certain security controls)
Data validation
Encoding
Parameterization
Logging
Error handling
Product security incident response – new CVEs based on impact and severity
Provide details of components, how to tell if you’re exposed, etc
Known vulns e.g. Shellshock/Heartbleed/Struts-Shock – triage impact, provide remediation plan
This sounds great, but how will we make this happen?
So how do we do this practically? There’s a lot to talk about here, but I want to focus on how you get started—
-- how to pick the right people for the guild, how do you get the program started, and how do you set up the working relationship between the security champion and the central security team.
We’ll be drawing examples from Veracode’s own experience standing up a security champion function.
Let’s assume you’ve made the case to management. The first step is drafting your team. Let’s talk about how not to pick the right people—
-- You shouldn’t assume that only developers are going to be good security champions. A strong QA resource may be a great candidate.
---- They should just have clear expectations on the time commitment.
---- And make sure to loop the managers into the call for volunteers.
---- And don’t stop with just one per team – you want to make sure you’re not introducing a new single point of failure
-- You should also have clear requirements. These will probably be specific to your culture and process, but some likely common requirements include:
---- Not new to company (ramping on day 2 day)
---- Not too junior (needed an influencer)
---- Not in an existing scrum role (PO, SM)
Now that you have the team in place, you need to give them a certain baseline knowledge. How you do this will differ according to what your security team looks like, but some of the basics are:
1. security fundamentals (2 day instructor led training)
CIA, etc
Trust no one
Security controls must be server-side
Deny by default
Use white lists
Defense in depth
Threat modeling
2. Reinforce with eLearning to provide ongoing training and access
3. Ongoing education with CTF exercises
Use stuff that’s readily available – https://overthewire.org
Set guidelines for common activities –
Product specific grooming guidelines
Update based on findings
Self-service review guidelines (what to teach vs. what can be done by anyone) – automate if possible (pro tip)
Does it need a review
Does it have to be a security team person
Crypto?
Can it be security champion
Dictates basic acceptance criteria
All of these things are great activities. To be really successful, you need to measure what you’re doing and how you’re making progress.
Product security maturity model
Baseline and regular updates
Open Samm and BSIMM useless – built our own
Needed a vision of an ideal security program
Code review certification
eLearning, classroom training, code review ‘test’
First year – mentored transition – failure
SC do first review
Expert validates results
Absence of findings was a problem
Formal exercise with 1:1 meeting to discuss process and findings
Goals for security champions
Goals for teams (against maturity model)
So this is the first step in easing security as a constraint in your process. You’ll probably find a lot of additional places to improve this process.
Last, a quick plug – if you would like to know more about how Veracode thinks about securing DevOps, you may want to look at our white paper, 5 Steps to Achieving Secure DevOps, which includes a few other topics around automation and Appsec in Production.