Presented by Ernest Mueller and James Wickett at RSA 2016
Moving fast is a business imperative that you can’t afford to be in opposition to. Lean, DevOps and Continuous Delivery philosophies hinge on the ability to move fast through collaboration, automation, and aligning with the flow of the organization. Security needs to be able to make the same transformation.
As a concrete example of applying these approaches to security, we will show how an Attack Driven approach to devops increases transparency and visibility throughout the organization and pairs with the high-throughput philosophies of DevOps and Continuous Delivery. We will engage in defensive systems thinking to change the attack landscape in our favor, while working with the way the business functions and not against it.
From this session, you will:
- Understand the Lean, Agile, and DevOps techniques emerging in organizations today
- Be armed with organizational strategies for bridging devops and security
- Take a defensive systems thinking approach to operations (and development)
- Apply the right detection and monitoring with real-world examples
4. @WICKETT // @ERNESTMUELLER // #LEANSECURITY
COMPANIES ARE SPENDING A GREAT
DEAL ON SECURITY, BUT WE READ
OF MASSIVE COMPUTER-RELATED
ATTACKS. CLEARLY SOMETHING IS
WRONG.
THE ROOT OF THE PROBLEM IS
TWOFOLD: WE’RE PROTECTING (AND
SPENDING MONEY ON PROTECTING)
THE WRONG THINGS, AND WE’RE
HURTING PRODUCTIVITY IN THE
PROCESS.
Thinking Security, Steven M. Bellovin 2015
7. @WICKETT // @ERNESTMUELLER // #LEANSECURITY
WHAT IS AGILE?
• INDIVIDUALS AND INTERACTIONS
OVER PROCESSES AND TOOLS
• WORKING SOFTWARE
OVER COMPREHENSIVE DOCUMENTATION
• CUSTOMER COLLABORATION
OVER CONTRACT NEGOTIATION
• RESPONDING TO CHANGE
OVER FOLLOWING A PLAN
SOURCE: THE AGILE MANIFESTO
(HTTP://WWW.AGILEMANIFESTO.ORG/)
8. @WICKETT // @ERNESTMUELLER // #LEANSECURITY
WHY AGILE?
• 45% OF ORGANIZATIONS ARE USING AGILE ON A MAJORITY OF
THEIR TEAMS
ONLY 5% ARE NOT USING IT AT ALL
• AGILE RESULTS:
• ACCELERATE PRODUCT DELIVERY - 59%
• ENHANCE ABILITY TO MANAGE CHANGING PRIORITIES - 56%
• INCREASE PRODUCTIVITY - 53%
• ENHANCE SOFTWARE QUALITY - 46%
• ENHANCE DELIVERY PREDICTABILITY - 44%
SOURCE: VERSIONONE NINTH ANNUAL STATE OF AGILE SURVEY
(HTTPS://WWW.VERSIONONE.COM/PDF/STATE-OF-AGILE-DEVELOPMENT-SURVEY-NINTH.PDF)
11. @WICKETT // @ERNESTMUELLER // #LEANSECURITY
WHAT IS DEVOPS?
DEVOPS IS THE PRACTICE OF OPERATIONS AND
DEVELOPMENT ENGINEERS PARTICIPATING
TOGETHER IN THE ENTIRE SERVICE LIFECYCLE,
FROM DESIGN THROUGH THE DEVELOPMENT
PROCESS TO PRODUCTION SUPPORT.
DEVOPS IS ALSO CHARACTERIZED BY
OPERATIONS STAFF MAKING USE MANY OF THE
SAME TECHNIQUES AS DEVELOPERS FOR THEIR
SYSTEMS WORK.
SOURCE: THE AGILE ADMIN: WHAT IS DEVOPS?
HTTP://THEAGILEADMIN.COM/WHAT-IS-DEVOPS/
12. @WICKETT // @ERNESTMUELLER // #LEANSECURITY
WHY DEVOPS?• BY 2016 “DEVOPS WILL EVOLVE FROM A NICHE TO A MAINSTREAM
STRATEGY EMPLOYED BY 25% OF GLOBAL 2000 ORGANIZATIONS” -
GARTNER, MARCH 2015
• BENEFITS OF DEVOPS:
• NEW SOFTWARE/SERVICES THAT WOULD OTHERWISE NOT BE
POSSIBLE - 21%
• A REDUCTION IN TIME SPENT FIXING AND MAINTAINING APPLICATIONS
- 21%
• INCREASED COLLABORATION BETWEEN DEPARTMENTS - 21%
• AN INCREASE IN REVENUE - 19%
• IMPROVED QUALITY AND PERFORMANCE OF OUR DEPLOYED
APPLICATIONS - 19%
SOURCE: CA RESEARCH REPORT—DEVOPS: THE WORST-KEPT SECRET TO WINNING IN THE
APPLICATION ECONOMY (HTTP://REWRITE.CA.COM/US/ARTICLES/DEVOPS/RESEARCH-REPORT--
DEVOPS-THE-WORST-KEPT-SECRET-TO-WINNING-IN-THE-APPLICATION-ECONOMY.HTML)
13. @WICKETT // @ERNESTMUELLER // #LEANSECURITY
HIGH-PERFORMING IT
ORGANIZATIONS
EXPERIENCE 60X FEWER
FAILURES AND RECOVER
FROM FAILURE 168X FASTER
THAN THEIR LOWER-
PERFORMING PEERS. THEY
ALSO DEPLOY 30X MORE
FREQUENTLY WITH 200X
SHORTER LEAD TIMES.
15. @WICKETT // @ERNESTMUELLER // #LEANSECURITY
LEAN SOFTWARE
DEVELOPMENT
SEVEN PRINCIPLES:
• ELIMINATE
WASTE
• AMPLIFY
LEARNING
• DECIDE AS LATE
AS POSSIBLE
• DELIVER AS FAST
AS POSSIBLE
• EMPOWER THE
TEAM
• BUILD INTEGRITY
IN
• SEE THE WHOLE
AN SOFTWARE DEVELOPMENT: AN AGILE TOOLKIT (2003), MARY AND TOM POPPENDIECK
16. @WICKETT // @ERNESTMUELLER // #LEANSECURITY
LEAN PRODUCT
DEVELOPMENT
• BUILD-MEASURE-LEARN
• BUILD – MINIMUM VIABLE PRODUCT
• MEASURE – THE OUTCOME AND INTERNAL
METRICS
• LEARN – ABOUT YOUR PROBLEM AND YOUR
SOLUTION
• REPEAT – GO DEEPER WHERE IT’S NEEDED
SOURCE: LEAN STARTUP (2011), ERIC RIES
17. @WICKETT // @ERNESTMUELLER // #LEANSECURITY
WHY LEAN?
• BOTH DEVOPS AND AGILE BORROW KEY
CONCEPTS FROM LEAN MANUFACTURING, SO
IT'S ALL ABOUT COMMUNICATION AND
OPENNESS." -INFORMATIONWEEK
18. @WICKETT // @ERNESTMUELLER // #LEANSECURITY
WHAT ARE THE
CHALLENGES THAT
AGILE / DEVOPS /
LEAN POSE TO
INFOSEC?
25. @WICKETT // @ERNESTMUELLER // #LEANSECURITY
“[RISK ASSESSMENT]
INTRODUCES A DANGEROUS
FALLACY: THAT STRUCTURED
INADEQUACY IS ALMOST AS
GOOD AS ADEQUACY AND
THAT UNDERFUNDED
SECURITY EFFORTS PLUS
RISK MANAGEMENT ARE
ABOUT AS GOOD AS
PROPERLY FUNDED SECURITY
WORK”
26.
27. @WICKETT // @ERNESTMUELLER // #LEANSECURITY
A SECURITY MANAGEMENT SYSTEM PROVIDES OPTIMAL
VALUE TO THE ORGANIZATION IF IT:
• ACTIVELY SUPPORTS ACHIEVING THE BUSINESS AND
COMPLIANCE OBJECTIVES OF THE ORGANIZATION
(THE VARIABLE PART)
• IS AN EFFICIENT, AGILE AND INTEGRATED PROCESS,
CAPABLE OF DEALING WITH A DYNAMIC THREAT
ENVIRONMENT
• CONSUMES MINIMAL TIME AND RESOURCES
• RESULTS IN ADEQUATELY MANAGED SECURITY RISK,
IN LINE WITH THE RISK APPETITE OF THE
ORGANIZATION
• PROVIDES ONLY THE NECESSARY, YET ADEQUATE,
USER FRIENDLY, EFFICIENT AND MEASURABLE
SECURITY CONTROLS
SOURCE: JOHAN BAKKER, LEAN SECURITY MANAGEMENT WHITE PAPER
30. @WICKETT // @ERNESTMUELLER // #LEANSECURITY
THE AVERAGE TIME TO
DELIVER CORPORATE IT
PROJECTS HAS INCREASED
FROM ~8.5 MONTHS TO OVER
10 MONTHS IN THE LAST 5
YEARS
Revving up your Corporate RPMs, Fortune Magazine, Feb 1, 2016
31. @WICKETT // @ERNESTMUELLER // #LEANSECURITY
WHY ARE COMPANIES SO SLOW?
THE GROWTH OF CONTROL AND
RISK MANAGEMENT FUNCTIONS
WHICH IS TOO OFTEN POORLY
COORDINATED… [RESULTING IN] A
PROLIFERATION OF NEW TASKS IN
THE AREAS OF COMPLIANCE,
PRIVACY AND DATA PROTECTION.
Revving up your Corporate RPMs, Fortune Magazine, Feb 1, 2016
32. @WICKETT // @ERNESTMUELLER // #LEANSECURITY
THE THREE WASTES
• MUDA - WORK WHICH ABSORBS RESOURCE
BUT ADDS NO VALUE
• MURI - UNREASONABLE WORK THAT IS
IMPOSED ON WORKERS AND MACHINES
• MURA - WORK COMING IN DRIBS AND
DRABS WITH SUDDEN PERIODS OF RUSH
RATHER THAN A CONSTANT OR REGULAR
FLOW, UNEVENNESS.
33. @WICKETT // @ERNESTMUELLER // #LEANSECURITY
SECURITY WASTE
MUDA COMES IN SEVEN FORMS:
• EXCESS INVENTORY - DUMPING YOUR THOUSAND
PAGE PDF OF VULNERABILITIES ON A BUSY TEAM.
PRIORITIZE AND LIMIT WORK IN PROGRESS (WIP)
• OVERPRODUCTION - SECURITY CONTROLS STEMMING
FROM FUD OR MISALIGNMENT WITH BUSINESS NEEDS
(NOT DEMANDED BY ACTUAL CUSTOMERS) - CF.
PHOENIX PROJECT
• EXTRA PROCESSING - FOR EXAMPLE, RELYING ON
COMPLIANCE TESTING RATHER THAN DESIGNING THE
PROCESS TO ELIMINATE PROBLEMS - HELP IT GET
BUILT RIGHT FIRST
34. @WICKETT // @ERNESTMUELLER // #LEANSECURITY
SECURITY WASTE
• HANDOFFS - LEVERAGE THE KNOWLEDGE OF THE TEAMS
DOING THE WORK AND COLLABORATE WITH THEM TO
BUILD SECURITY IN, INSTEAD OF THAT BEING SOME
OTHER TEAM’S JOB
• WAITING - LAG BETWEEN VALUE STEPS WAITING FOR
APPROVALS OR ANALYSES OR TICKET HANDLING - USE
SELF SERVICE AUTOMATION INSTEAD
• TASK SWITCHING - THE THOUSAND PAGE PDF AGAIN -
WORK WITH THEIR WORK INTAKE PROCESS NOT AGAINST
IT
• DEFECTS - FALSE POSITIVES AND FALSE NEGATIVES AND
JUST PLAIN UNIMPORTANT FINDINGS YOU REPORT
CAUSING ZERO-VALUE REWORK
39. @WICKETT // @ERNESTMUELLER // #LEANSECURITY
PERFORMANCE
• BROWSER EXTENSIONS FOR DEVS TO
UNDERSTAND PERFORMANCE PROBLEMS
• RESEARCH SHOWING PERFORMANCE TO
REVENUE CORRELATION
• SEARCHABLE LOGS EMITTING STATSD METRICS
• CONFERENCES COMBINING FRONT END DEVS AND
SYS ADMINS
• COMMITMENT TO INSTRUMENT AND GRAPH ALL
THE THINGS
40. @WICKETT // @ERNESTMUELLER // #LEANSECURITY
SECURITY
• BROWSER EXTENSIONS FOR DEVS TO
UNDERSTAND SECURITY PROBLEMS
• RESEARCH SHOWING SECURITY TO REVENUE
CORRELATION
• SEARCHABLE LOGS EMITTING STATSD METRICS
• CONFERENCES COMBINING DEVS OPS AND
SECURITY
• COMMITMENT TO INSTRUMENT AND GRAPH ALL
THE THINGS
41. @WICKETT // @ERNESTMUELLER // #LEANSECURITY
SEE THE WHOLE
• KEEP MEANINGFUL METRICS, MAKE THOSE
METRICS VISIBLE - IN CONTEXT OF
WORKERS’ TOOLCHAIN
• “LEAST PRIVILEGE” NEEDS TO BE
UNLEARNED SOMEWHAT IN MODERN
ORGANIZATIONS TO ALLOW EFFECTIVE
INFORMATION SHARING
• GET IN BUSINESS OF SHARING AND ADDING
VISIBILITY TO DEV AND TO OPS.
44. @WICKETT // @ERNESTMUELLER // #LEANSECURITY
BUILD INTEGRITY IN
• “CEASE DEPENDENCE ON MASS
INSPECTION TO ACHIEVE QUALITY.
IMPROVE THE PROCESS AND BUILD
QUALITY INTO THE PRODUCT IN THE FIRST
PLACE." — W. EDWARDS DEMING
• INTEGRATE INTO CONTINUOUS
INTEGRATION AND USE TEST DRIVEN
DEVELOPMENT (TDD) TO RECTIFY ISSUES
AT THE LOWEST WASTE POINT
46. @WICKETT // @ERNESTMUELLER // #LEANSECURITY
NEEDED A WAY TO
BE MEAN TO YOUR CODE
EARLIER IN THE
DEVELOPMENT PROCESS
ENTER GAUNTLT…
47. @WICKETT // @ERNESTMUELLER // #LEANSECURITY
@slow @final
Feature: Look for cross site scripting (xss) using arachni
against a URL
Scenario: Using arachni, look for cross site scripting and
verify no issues are found
Given "arachni" is installed
And the following profile:
| name | value |
| url | http://localhost:8008 |
When I launch an "arachni" attack with:
"""
arachni —check=xss* <url>
"""
Then the output should contain "0 issues were detected."
Given
When
Then
What?
AN ATTACK LANGUAGE FOR DEVOPS
53. @WICKETT // @ERNESTMUELLER // #LEANSECURITY
ARE YOU “THAT
GUY?”
• YOU ALREADY KNOW YOU CAN’T
MAKE THINGS SECURE BY
YOURSELF
• YOU NEED EVERYONE ELSE TO
PITCH IN - BUT DOES IT SEEM
LIKE THE THINGS YOU DO JUST
ANGER THEM?
54. @WICKETT // @ERNESTMUELLER // #LEANSECURITY
EMPOWER THE TEAM
• UNDERSTAND HUMAN
MOTIVATION
• NETFLIX AUTOMATION
CREATED SAFE PATHS
AS THE DEFAULT
• REMOVES EMOTIONAL
CHARGE
61. @WICKETT // @ERNESTMUELLER // #LEANSECURITY
ERNEST MUELLER
JAMES WICKETT
@wickett
@ernestmueller
THEAGILEADMIN.COM
Hinweis der Redaktion
Howdy from Austin, Texas. We are James and Ernest and both have worked together doing devops and security and have been friends for the last 12 years. We are both actively involved with DevOps (we run devopsdays austin) and the security community and user groups like OWASP. We blog together at theagileadmin.com.
DevOps changed our life and we and are here to share how understanding the same Lean techniques can improve the effectiveness of your security work.
In the recent book by Steven Bellovin, “Thinking Security,” he points out that as we are failing in our attempts to turn security spend into real security because we’re addressing the wrong things and hurting productivity.
This might be how you feel…. Well, lets get to some solutions.
To talk about Lean, we also want to talk about Agile and DevOps because they’re closely interrelated.
Most of you probably have an understanding of Agile. You have seen it done right, seen it done wrong… There’s a lot more to it, but this is the core manifesto.
Agile has become widespread and has provided clear benefits to software development in organizations.
Next, DevOps!
We had a problem with development and operations being siloed and chasing opposing goals.
DevOps was conceptualized as a way to correct that. “DevOps is the application of Agile methodology to system administration” - The Practice of Cloud System Administration Book
We are no longer having conversations about whether it is or isn't a thing. We are now talking about DevOps adoption and how it’s quickly growing, and is showing the same kind of benefits to adopting organizations.
As we saw earlier in Jez and Nicole’s presentation, companies are getting real ORDERS OF MAGNITUDE value out of devops.
And that brings us to Lean. You have heard other speakers today mention lean, waste, WIP, and the theory of constraints - who’s familiar with Lean (manufacturing, software development, or product development)?
What is Lean? Lean started off by revolutionizing the world of manufacturing (W. Edwards Deming, Toyota Production System) but since then it has been adapted to software development. Its practices include value stream mapping, waste, pull, queueing theory, human motivation, measurement and visualization of metrics, TDD… We’ll go over many of these in the context of improving security work later in the presentation.
Eric Ries also applied lean principles to product development in his book Lean Startup, which characterizes the core loop inside the product development cycle as “Build – Measure – Learn.” Lean is about bringing your effort onto the item with the highest leverage at any given time.
The Puppet State of DevOps Report says: “One can describe DevOps as the pattern that emerges
when you apply these same lean principles to technology.” Lean product, lean software, agile, and devops all come together into a single mutually reinforcing picture for a technology organization.
If you look at every new innovation, whether it’s Lean, devops, cloud, social mobile, etc. as simply a “threat” to security then you’ve adopted a losing mindset out of the gate.
Every single field has to innovate to stay relevant, and InfoSec doesn’t get a pass on that.
We’ll now examine common challenges faced by InfoSec organizations and explain how you might be able to bring Lean to bear on implementing security more effectively in your organization. Each of these is a problem that we had in Ops before we found DevOps.
Each one of these is a perception you have probably heard from someone at some point. While these are not all fair, they are also not completely random and unfounded. The first is that you’re just there to check boxes and don’t do much to make the apps and systems really more secure.
In his book on browser security, Michael Zalewski has a great intro covering the history of information security and he poignantly notes that we decided risk management was able to fill the development and operations gaps we experienced. We became experts of structured inadequacy and wrapping problems with policies and “Accept the risk” statements. This is not value creation.
The foundation of Lean is to understand what the real value add steps are in the creation of your product or service. This is an example value stream mapping representing the value steps between the initiation of a product and delivery to the customer. In Lean Software, this is called “Concept to Cash.”
Do you really understand what value you are creating and where?
Johan Bakker took a stab at what the true value of security management looks like to an organization. Note that it hinges on creating a solution that is custom to your organization. Whatever stock answers you got taught in Security School are not necessarily the value your customers need from you.
Where can you add real security value that improves the value of your organization’s core business?
The second complaint is that security is just a bottleneck to getting “the real work done.”
Fortune Magazine just a few weeks reported that the average time to deliver corporate IT Projects has increased from ~8.5 months to over 10 months in the last 5 years.
And the reason is… Security. Security has resulted in a proliferation of new work that, if badly coordinated, slows everything else down.
Lean focuses a lot on the identification and removal of waste; it’s the very first Lean principle. In today’s business environment time is a critical resource, and to be honest, Security is often guilty of squandering it.
The seven forms of muda can be seen in security operations frequently. These are a couple security-centric examples, but the takeaway is to analyze what you’re doing and identify the areas of waste in it.
Two kinds of muda - type 1 (necessary) and type 2 (unnecessary)
Your net value to the organization is the value you create minus the waste that you generate.
People are tempted to see security as a solution in search of a problem when they don’t see how it fits in to everything.
Security is everyone’s job, right?
In Operations, Performance used to be invisible and we would say performance was everyone’s job… Then we did something about it.
Security has a lot of corollaries to performance problems 5-10 years ago.
To help people actually see and address the problems, performance experts focused on visualizing performance metrics directly in context to workers.
We could do the same thing with security…
This is an example of another Lean principle, “See the whole”; lean software development speaks extensively about metrics collection and visualization’
It’s easy to get forced into reacting after there’s a breach, or even just after a vulnerability has gone live into production.
The more you can create fast feedback loops which detect and remediate security problems continually as part of your customers’ normal work process, the less waste you generate.
I wanted to find a way to be mean to your code in the development process. I knew that attack tooling had to move upstream.
You know - not just a bottleneck due to constraints, but actively messing with us.
Adrian Cockroft from Netflix claimed they had “No process” at AppSec one year. What he really meant is that they have made doing the right thing a part of the systems everyone uses, so the perception is that there’s no process.
Use automation to build integrity in, don’t rely on mass inspection after things have already been done wrong.
Security is a product, like any other. And all products have to make tradeoffs about what they will do and what they won’t do.
If you listen to some folks you can’t “do security” without a $1.2M budget to get the six or seven huge products you need, and that’s the way to achieve perfection. But these tools are not only maybe not a good fit for your needs, but take a lot of time to implement. Rather than add additional waste via long analysis cycles, implement something small and fast, analyze results, and iterate. Often layers of a couple imperfect items yields better security than one “perfect” item.
The whole reason there’s a DevOps track at this conference is that not too many years ago we were in the same situation and had all the exact same criticisms leveled at us. Operations was the at-best-invisible, beancounting bottleneck that was always a day late and a dollar short. But these time-tested principles has helped our entire industry begin to innovate its way out of that rut. Check them out and see if they can help you in the same way.
Thanks for your time! You can find more of our thoughts at theagileadmin.com and we’re both here with companies working on solutions that we think are aligned with this vision of the future of security work.