Talk given by James Wickett and Ernest Mueller at the (ISC)2 Secure SDLC event in Austin, TX.

1. RUGGED DEVOPS WILL HELP
YOU BUILD UR CLOUDZ
by @wickett and @ernestmueller

2. OUTLINE
• Us, And Why You Care What We Say
• The Cloud, And How It Is Threatening You
• Rugged, And Its New Approach To Security
• DevOps, And How It Is Driving Collaborative Solutions
• Combining Cloud, Rugged, and DevOps To Solve The Problem
• How We Did Cloud Security With DevOps At NI
• Introducing RuggedDevOps Tool: Gauntlt

3. @wicke'
Senior
DevOps
Engineer
CISSP,
GWAPT,
CCSK,
GSEC,
GCFW
james@wicke'.me
@RuggedDevOps
theagileadmin.com
NI
CONFIDENTIAL

4. @ernestmueller
DevOps Platform Manager
and Release Manager,
Bazaarvoice
ernest.mueller@gmail.com
theagileadmin.com

5. WHAT IS THE CLOUD?

6. THE GRAND UNIFIED THEORY
(ISP -> colo -> MSP) + virtualization
+ HPC + (AJAX + SOA -> REST APIs) =
IaaS
((web site -> web app) -> ASP) +
virtualization + fast ubiquitous
Internet + [RIA browsers && mobile] =
SaaS
IDE/4GLs + (EAI -> SOA) + SaaS +
IaaS = PaaS
[IaaS | PaaS | SaaS ] + [ devops |
open source | noSQL ] = cloud

7. CLOUDINESS
• An outsourced managed service
• providing hosted computing or functionality
• delivered over the Internet
• offering extreme scalability
• by using dynamically provisioned, multitenant, virtualized
systems, storage, and applications
• controlled via REST APIs
• and billed in a utility manner.

8. “Cloud? I’ve been doing that since
1988. It’s just the same old thing
with a new name."
- Technohipster

9. Not new:
virtualization
outsourcing
integration
interwebz
Pretty new:
multitenant
massively scalable
elastic self provisioning
pay as you go
Resulting benefits:
agility
economy of scale
low initial investment
scalable cost/opex
resilience
easy delivery

11. I recognize that my code will be attacked by talented and
persistent adversaries who threaten our physical,
economic, and national security.
I recognize that my
code will be used in
ways I cannot
anticipate, in ways it
was not designed, and
for longer than it was
ever intended.
I am rugged, not because it is easy, but because it is
necessary... and I am up for the challenge.

12. RUGGEDIZATION
THEORY
Building solutions to handle
adversity will cause
unintended, positive benefits
that will provide value that
would have been unrealized
otherwise.

13. No Pain, No Gain

14. RUGGED-ITIES
Maintainability
Availability
Survivability
Defensibility
Security
Longevity
Portability
Reliability

15. WHAT NEEDS TO HAPPEN
• Focus on real security. FUD doesn’t benefit anyone – figuring
out how to “make it happen” – securely – benefits everyone.
• It’ll
take time for compliance standards to get with the times –
but don’t assume the cloud can’t be compliant – some of your
auditors have actually heard of VMs and know what to do
• Organizations have to accept risk to reap rewards.
• Agile has taught orgs the collaborative approach is best
• Lean has taught orgs to experiment and iterate

16. source: Gene Kim, “When IT says No @SXSW 2012”

17. SECURITY SEES...
• They give advice that goes unheeded
• Business decisions made w/o regard of risk
• Irrelevancy in the organization
• Constant bearer of bad news
• Feels ignored by their peers (you know,
those devops guys)
• Inequitable distribution of labor

18. TRADITIONAL SECURITY

19. THE CLOUD RESPONSE

20. THE SEPARATION MODEL

21. DEVOPS

22. SERVICE LIFECYCLE

23. ANTIPATTERN!
Deploying Software Manually

24. ANTIPATTERN!
Deploying to a Production-like
Environment Only after Development is
Complete

25. ANTIPATTERN!
Manual
Configuration
Management of
Production
Environments

26. CONTINUOUS INTEGRATION
• Check In Regularly
• Create an automated and comprehensive test suite
• Keep build and test short and fast
• All tests must pass before moving on
• Never Go Home on a broken build
• Never comment out failing tests

27. CONFIGURATION
MANAGEMENT
• Infrastructure as Code (IaC)
• Model driven deployment
• Version control everything
• Know Your Environment if
you want to make it
defensible

31. RUGGED DEVOPS
BRIDGING SECURITY AND DEVOPS

32. DEVOPS (+SEC)
• Increasedtrend driven by agile development towards tight
collaboration between developers and operations staff
• Be the “security buddy”
• Embed with projects, don’t be a seagull
• By understanding, be understood
• How secure are things usually when people and teams all
work separately?

34. THE 6 R’S RUGGED DEVOPS
• repeatable – no manual steps
• reliable - no DoS here
• reviewable – aka audit
• rapid – fast to build, deploy, restore
• resilient – automated reconfiguration
• reduced - limited attack surface

38. APPLY RUGGED DEVOPS TO
THE CLOUD
• Start with a Rugged DevOps team
• Use a lot of firewalls
• Scan your code
• Source to system
• Threat modeling
• Watch for changes
• Pen Testing

39. BUILD A
RUGGED
DEVOPS
TEAM

40. PEOPLE, PROCESS, TECH

41. PEOPLE AND PROCESS
• Sit near the dev and ops team, better yet, put them
all on the same team
• Track security flaws or bugs in the same bug tracking
system
• Automate whenever possible
• Involve team with vendors
• Measurement over time and clear communication

42. USE
FIREWALLS...
(A LOT OF
THEM)

43. Traditional 3-Tier Web Architecture
Firewall
Web Web Web DMZ 1
Firewall
Middle Tier Middle Tier DMZ 2
Firewall
DB LDAP DMZ 3

44. Cloud Firewalls and DMZ
firewall firewall firewall
Web Web Web DMZ x3
firewall firewall
DMZ x2
Middle Tier Middle Tier
firewall firewall
DB LDAP
DMZ x2

45. firewall firewall firewall
Web Web Web
Repeatable
firewall firewall
Verifiable
Middle Tier Middle Tier Prod/Dev/Test Matching
firewall firewall Controlled
Automated
DB LDAP
firewall firewall firewall firewall firewall firewall
Web Web Web Web Web Web
firewall firewall firewall firewall
Middle Tier Middle Tier Middle Tier Middle Tier
firewall firewall firewall firewall
DB LDAP DB LDAP

46. firewall firewall firewall firewall firewall firewall firewall firewall firewall
Web Web Web Web Web Web Web Web Web
firewall firewall firewall firewall firewall firewall
Middle Tier Middle Tier Middle Tier Middle Tier Middle Tier Middle Tier
firewall firewall firewall firewall firewall firewall
DB LDAP DB LDAP DB LDAP
firewall firewall firewall firewall firewall firewall firewall firewall firewall
Web Web Web Web Web Web Web Web Web
firewall firewall firewall firewall firewall firewall
Middle Tier Middle Tier Middle Tier Middle Tier Middle Tier Middle Tier
firewall firewall firewall firewall firewall firewall
DB LDAP DB LDAP DB LDAP
firewall firewall firewall firewall firewall firewall firewall firewall firewall
Web Web Web Web Web Web Web Web Web
firewall firewall firewall firewall firewall firewall
Middle Tier Middle Tier Middle Tier Middle Tier Middle Tier Middle Tier
firewall firewall firewall firewall firewall firewall
DB LDAP DB LDAP DB LDAP

47. RUGGED BENEFITS
• Control and traffic whitelisting
• Config management
• Reproducible, automated and source controlled
• No accidental data traversal across products or
dev/test/prod tiers
• Dev and Test identical to Prod tier

48. SCAN
THE
CODE

49. • Scans for OWASP Top Ten and more
• Security Scanning as a Service
• Static and Dynamic scanning
• Integrated into development process
• Dynamic and Static scanning

50. SOURCE
TO
SYSTEM

51. AUTOMATED PROVISIONING - PIE
• Programmable Infrastructure Environment (PIE)
• Code can be version controlled
• Make Infrastructure as code
• Defined once, deployed many times
• Eliminate repetitive task and human errors
• Rollback capability

52. • a framework to define, provision, monitor, and control cloud-based
systems
• written in Java, uses SSH as transport, currently supports Amazon
AWS (Linux and Windows) and Microsoft Azure
• takes an XML-based model from source control and creates a full
running system
• to define, provision, monitor, and control cloud-based systems

53. THREAT
MODEL
ME

54. THREAT MODELING
• Understanding the threat profile of
a system
• Provide a basis for secure design
and implementation
• Discover vulnerabilities
• Provide feedback for the
application security life cycle
p. 29 in Threat Modeling, Swiderski, Snyder

55. WATCH
MY
CHANGES

56. HOST INTRUSION
DETECTION SYSTEM
• Watch the file system (using hashing and timestamps)
– /etc/
– /usr/bin
–…
• Change control for applications
• Alert on changes and anomalies
• PIE watchdog

57. PEN
TESTING

58. PENETRATION TESTING
• Use external and internal penetration
testing
• White box testing vs. Black box testing
• Look for automation opportunities
(ruby, python, …)

59. BUT WHAT ABOUT SECURITY
TESTING IN MY
CONTINUOUS INTEGRATION
SYSTEM?

60. PUT YOUR CODE THROUGH
THE GAUNTLT

61. GAUNTLET, N.
AN ATTACK FROM ALL
SIDES

62. custom attacks dirbuster
metasploit sqlmap
fuzzers nessus
w3af nmap
Your web app You

63. GAUNTLT
IS BUILT FOR
CONTINUOUS INTEGRATION

64. GAUNTLT IS

65. AN ALWAYS-ATTACKING
ENVIRONMENT FOR
DEVELOPERS

66. WITH ATTACKS WRITTEN IN
EASY-TO-READ LANGUAGE

67. ACCESSIBLE TO EVERYONE
INVOLVED IN DEV, OPS,
TESTING, SECURITY, ...

68. GAUNTLT INCLUDES

69. WHY GAUNTLT?
SECURITY DOMAIN
KNOWLEDGE IS
GENERALLY A MYSTERY
TO DEV TEAMS

70. GAUNTLT ALLOWS DEV
AND OPS AND SECURITY
TO COMMUNICATE AND
COLLABORATE

71. GAUNTLT JOINS:
THE PHILOSOPHY OF
RUGGED SOFTWARE
&
OUTSIDE-IN
TESTING

72. LETS LOOK INSIDE A COUPLE
OF THESE FILES

73. feature for nmap:
nmap.feature
@gauntlet @run
Feature: Run nmap against a target and pass the value of the hostname from the profile.xml.
Background:
Given nmap is installed
Scenario: Verify server is available on standard web ports
Given the hostname in the profile.xml
When I run nmap against the hostname in the profile on ports 80,443
Then the output should contain:
"""
80/tcp open http
443/tcp open https
"""

74. step definition for nmap:
nmap.rb
Given /^nmap is installed$/ do
steps %{
When I run `which nmap`
Then the output should contain:
"""
nmap
"""
}
end
When /^I run nmap against the hostname in the profile on ports (d+),(d+)$/ do |arg2, arg3|
steps %{
When I run `nmap "#{@hostname}" -p80,443`
}
end

75. lets run gauntlt with
the nmap.feature
against google.com

76. running gauntlt with failing tests
wickett$ gauntlt
@gauntlet @run
Feature: Run nmap against a target and pass the value of the hostname from the profile.xml.
Background: # features/nmap/nmap.feature:5
Given nmap is installed # features/step_definitions/nmap.rb:2
Scenario:Verify server is available on standard web ports # features/nmap/nmap.feature:8
Given the hostname in the profile.xml # features/step_definitions/profile.rb:1
When I run nmap against the hostname in the profile on ports 8080,443 # features/step_definitions/nmap.rb:12
Then the output should contain: # aruba-0.4.11/lib/aruba/cucumber.rb:98
"""
8080/tcp open http
443/tcp open https
"""
...
Failing Scenarios:
cucumber features/nmap/nmap.feature:8 # Scenario:Verify server is available on standard web ports
1 scenario (1 failed)
4 steps (1 failed, 3 passed)
0m0.341s

77. running gauntlt with passing tests
wickett$ gauntlt
@gauntlet @run
Feature: Run nmap against a target and pass the value of the hostname from the profile.xml.
Background: # features/nmap/nmap.feature:5
Given nmap is installed # features/step_definitions/nmap.rb:2
Scenario:Verify server is available on standard web ports # features/nmap/nmap.feature:8
Given the hostname in the profile.xml # features/step_definitions/profile.rb:1
When I run nmap against the hostname in the profile on ports 80,443 # features/step_definitions/nmap.rb:12
Then the output should contain: # aruba-0.4.11/lib/aruba/cucumber.rb:98
"""
80/tcp open http
443/tcp open https
"""
1 scenario (1 passed)
4 steps (4 passed)
0m1.117s

78. WALK VS. RUN
• gauntlt has two modes: walk and run
• meaning fast and slow or smoke and full
• This is done by labels in cucumber
• For
each feature you will get to decide if it is a @walk or a
@run test or both

79. SOME REALIZATIONS
• The core of gauntlt needs to provide a set of
functionality that encourages contributors to write
extensions for their pen testing tools
•A gauntlt DSL (Domain Specific Language) will arise
with words like target, scan, attack, host...
• Smoke tests and validation vs. long running testing
(nightly/weekly)

80. JOIN THE PARTY!!
FORK GAUNTLT ON GITHUB

81. HTTPS://GITHUB.COM/
THEGAUNTLET/GAUNTLT

82. CLOUD & SECURITY
RESOURCES
• Book: Cloud Security and Privacy (Mather, Kumraswamy, Latif)
• Jericho Forum (collaboration.opengroup.org/jericho/)
• Amazon AWS Security Center (aws.amazon.com/security)
• Austin Cloud User Group (acug.cloudug.org)
• Cloud Security Alliance (cloudsecurityalliance.org)
• CSA Austin Chapter (austincloud.org)
• CSA Security Guidance for Critical Areas in Cloud Computing
• ENISA Cloud Computing Risk Assessment

83. CONTACT US!
@ERNESTMUELLER
@WICKETT