DevOpsSec: Appling DevOps Principles to Security, DevOpsDays Austin 2012
1. DevOpsSec
Applying DevOps Principles to Security
Nick Galbreath nickg@etsy.com @ngalbreath
DevOpsDays, Austin Texas, April 3, 2012
http://client9.com/20120403 nickg@client9.com
2. Slides! Video!
• Originally presented on April 3, 2012
• Latest Slides! Streaming Video!
http://client9.com/20120403
• Related interview:
http://youtu.be/Afd0u5DGxr8
• Original video stream:
http://www.ustream.tv/recorded/21568549
3. whoami
• Development background
• Lots o’ startups, book, patents,blahblahblah
• Director of Engineering at Etsy covering
• Security, Fraud, Biz Analytics, Email Infra,
Internal Systems, and everything else not
www.etsy.com “Enterprise”
• Second time working with Allspaw!
• “Oh you mean there is a name for this?”
4. Context
My biases for this talk is (Web) Application
Security, not classic Network Security or IT
Security.
9. ...with the
acknowledgement that
• We are working in a complex system
• And in complex systems failure happens
• And failure can happen when
everyone does nothing wrong
• And given this, how can one increase
reward and reduce risk for the business
10. What does this mean for....
People?
Processes (workflow)?
Machines?
11. An Only Slightly
Contrived Example
• I trust MCR to run our network
• I can verify this by looking at our dataporn
• He trusts me that when things go wrong,
the graphs won’t be used to burn him.
• He can verify this by... seeing our Post
Mortems in action (they are open at Etsy)
13. Squeezed from Both Sides
Unreviewed Code going out,
Untrusted Data coming in
DATA UGH CODE
Makes stability and responsibility
“complicated”, even more so if there are
walls between groups.
14. Latent Problems
There are operational problems right now
just not manifested.
There are security problems right now just
not exploited.
15. Cultural Problems
• Both have severe failure causes
• Both Ops and Security have a “say no”
perception
• “Operations” and “Security” are services
groups but frequently not viewed as such
16. Ok, back to the
regularly scheduled
programming
17. DevOpsSec
E 2
Applying DevOps Principles to Security
A K
Nick Galbreath nickg@etsy.com @ngalbreath
DevOpsDays, Austin Texas, April 3, 2012
http://client9.com/20120403 nickg@client9.com
20. How Fast Can You
Deploy or Rebuild
• Your Firewall,VPN, Load Balancer
• Your Operating System, Critical Servers
• Your Database, server, schema, data
• Your Application, patches
• Any other configuration file
in a consistent, sane manner
21. Being able to deploy
quickly is my #1
security feature
This implies a standardized, automated
system and configuration management.
22. I Call Bullshit
Doesn’t the rapid rate of
change in a continuous
integration environment
mean things are less secure?
Well compare this to....
23. We’ll rush that security fix.
It will go out in next release
in about 6 weeks.
former vendor at Etsy
25. It’s ok if we have a few extra
firemen waiting around in
case there is a fire
I’m more concerned we
won’t know there is a
fire until the house is
burnt down
Conversation between Chad Dickerson and Nick Galbreath, Etsy 2011
26. Segmentation Faults
• Why is your server falling over?
• From the same IP address.
• Over and over
Maybe time to patch? Also check
out your server 500 errors.
27. Database Syntax Errors
Almost game over here.
Whose is responsible for these anyways?
Demand zero-tolerance for database syntax errors.
28. SQLi Attacks
The shittest check that works.
Will undercount by at least 10x
ProTip #1: using regexp for sqli is fail
ProTip #2: write a unit test for this.
30. Got that?
Security is not a
binary event.
You are being
attacked constantly.
THIS IS AWESOME
31. Attack Driven Testing
Security testing in dev is hard.
Use actual attacks/probes to guide testing.
• Server 500 errors
• Core Dumps Can you
automate
• CSRF Failures verification?
• XSS Attempts
• Login Failures
• Password Resets
33. assertyour production
environment
Who knew that
writing solid C code is
similar to running a
complex system.
Writing Solid Code
Steve Maguire
1993
34. assert this
• This page is always SSL
• This page requires sign-in
• This page never is publicly available
• Google never crawls this page
• This page is not being routed by the CDN
• This port is never open
38. Post Mortems
• All security issues are “P1” or “P2” (fix
now, or fix by end of week)
• Even for internal applications.
• All get a post-mortem
• Great educational experience and
knowledge transfer
39. Hiring
• Strict no asshole policy.
• Security is a services business.
• “Product Security” is in-house consultancy.
• Can you take people who are interested
and train them? TBD.
http://www.sans.org/
https://www.owasp.org/
40. Extend the Perimeter
• Working on a training program for both in-
house employees, contractors, and external
vendors.
• “Device Tune-Up Day” -- employees bring
their home computers in for a tune-up.