Slides from talk at Shmoocon 2013 given by Matt Joyce titled "OpenStack Security Brief".

1. "We used to leak kilobytes,
then megs, then even gigs.
Cloud Computing
Now, we leak EC2 instances.
Someday, we'll leak
entire datacenters."
- @Dymaxion

This term means absolutely nothing.

$variable + vague generic term

2. OpenStack
Security Brief
ShmooCon 2013
http://www.secstack.org/shmoocon2013.ppt

3. Yes, this is me.

4. This is also me.

5. Part I – OpenStack Structure

6. Cloud Computing is a terrible term.
Investopedia defines it as...
... this is why it was referred to as
'Clown Computing' for a very long time.

7. A Better Term :
Elastic Design
Scale horizontally rather than vertically

Distributed services

Standard Orchestration APIs

All States are Ephemeral

8. So.. it's an Open Stack?

Elastic Cloud

Open Source ( Apache License )

Open Standards ( Foundation )

Written in Python

REST APIs

Shared Nothing, Message Oriented

9. Gaming the Foundation
A fun tangent
https://www.music-piracy.com/?p=750

10. OpenStack Membership 2011

11. Top Companies by Commits

12. Votes by Source

13. Components of OpenStack
( Folsom – 2012.2 )
Core Clients Incubated

Nova 
python-novaclient 
Oslo

Swift 
python-swiftclient 
Ceilometer

Keystone 
python-keystoneclient 
python-ceilometerclient

Glance 
python-glanceclient 
HEAT API

Quantum 
python-quantumclient 
python-heatclient

Cinder 
python-cinderclient 
python-openstackclient

Horizon

14. Good Reading
Ken Pepple's Folsom Architecture Post
http://ken.pepple.info/openstack/2012/09/25/openstack-folsom-architecture/

15. http://ken.pepple.info/openstack/2012/09/25/openstack-folsom-architecture/

16. Not getting into hypervisor security.
OpenStack supports many hypervisors.
Some supported hypervisors:

KVM

Xen / XCP

HyperV

VMWare

Physical Provisioning ( in Grizzly )

etc, etc, etc. sky's the limit, bob's your uncle.

18. Keystone – Identity Manager

REST API, Admin API

Service Catalog

Backend to sqlite by default

Supports MySQL, LDAP, Active Directory
( with patches ).

Token generation and shared
authentication endpoint in OpenStack
software.

19. Nova – Elastic Compute ( EC2 )

REST API, Metadata API, EC2 API

Integrates with many hypervisors

Defaults to libvirt

Integrated volume and network
orchestration in Folsom ( deprecated )

Security Groups, Quotas, Zones, Flavors..

Config Drive

Ugliest, oldest, most complex code in
project.

20. Glance – Image Store

REST API

Backed my MySQL

Stores to local volumes

Optionally stores to object storage

21. Quantum – SDN

Replaces nova-network

REST API

Can interact directly with hardware

Pluggable networking extensions

MySQL backend

22. Cinder – Volumes

Replaces nova-volume

REST API

MySQL backend

LVM management on nova-volume nodes

Direct hardware interaction with NAS

Direct interaction with soft block stores

23. Swift – Object Storage ( S3 )

REST API

HA-Proxy Load balancer

Block Manipulation on Nodes

Soft Replication between Nodes

24. Horizon – Web GUI ( Django )

Integrates with REST APIs

Integrates with Client APIs

Uses standard Keystone token
authentication

Django based

Does not use EC2 APIs, solely OpenStack

25. Message Buses

RabbitMQ

ZeroMQ

26. Development Workflows

Continuous Integration

Gerrit

Jenkins

Launchpad

GitHub

Packaging

27. Packaging

Core packages are built from release
tarballs

Client packages are built from pypi tarballs

Git releases are PGP signed

Efforts are being made to ensure all
dependencies are PGP signed properly

Ubuntu / RedHat / SuSE among many
vendors with signed releases

28. Good Reading
China GitHub and Man in the Middle
https://en.greatfire.org/blog/2013/jan/china-github-and-man-middle

29. Part II – Targetting OpenStack

30. Layer 3 Model

31. Layer 2 Model

32. Nested Model

33. The ZeroMQ Message Bus

Fuzzing attacks in 2.1

“ØMQ does not deal with security by
design but concentrates on getting your
bytes over the network as fast as possible.”

The question of encrypting 0mq
communications is difficult in cloud
environments.

Message Signing

34. Good Reading
Status of Secure Messaging
http://lists.openstack.org/pipermail/openstack-dev/2013-
February/005614.html

35. The RabbitMQ Message Bus

Supports SSL

Supports Authentication ( SASL )

Public / Private Queues

No encryption at rest ( who cares? )

Not as horizontally scalable

36. The REST APIs and other HTTP Targets

Backend ( wsgi )

Admin ( wsgi )

Client ( requests )

SDKs ( there are many )

Horizon ( django )

37. Config Drive

CVE-2012-3447

https://blueprints.launchpad.net/nova/+spec/config-drive-v2

Compromise of Compute Hosts WITHOUT hypervisor escape
possible

38. Volumes, Block Storage, and Memory

Volume zeroing is a recurring vulnerability

Volume encryption coming

Shared Memory space presents the
possibility for attackers to sniff memory
allocated to other virtual hosts

DMA access is a continual source of
hypervisor escape attacks

39. Authentication

Auth Tokens – UUID v4 / dev urandom

PKI Certs – Grizzly*

Multifactor Auth – Grizzly*

Token Sizes... Enormous 40bytes to 3k.
Potential for DDOS and Failure in Horizon

Authn/z – Grizzly*

40. Analysis of Past Vulnerabilities

41. Lines of Code per Project

42. Vulnerability Reports by
Company

43. Part III – Defense against the
Dark Arts

44. Intrusion Detection

45. Intrusion Detection

Security APIs ( ceilometer, marconi? ) -
event logging

Precursor Indicators – Homogeneity makes
anomalies easy to spot. Standard methods
as well.

External Reporting

Security Services ( SaaS )

Infrastructure Knowledge ( This Preso )

46. Intrusion Response
You guys know this better than I

Have a plan.

Consumers must have a workflow that is
known and supported for response.

Disclosure of breach and other issues
should be planned for ahead of time.

Don't Panic.

47. Forensics ( Chain of Custody )

Ephemeral Design means interruption is
usually expected as part of SLA

OpenStack has no mechanism for
migrating instances between tenants.

You may want to provide SOC teams
tenant access to monitor compromised
instances.

Instances can be snapshotted and
exported for controlled testing in sandbox.

Logs should be isolated in one way DMZ

48. Reporting to OpenStack

Open a bug in Launchpad and mark it as a
'security bug'. This will make the bug
Private and only accessible to the
Vulnerability Management Team.

If the issue is extremely sensitive, please
send an encrypted email to one of the
Team’s members. Their GPG keys can be
found below, and are also available from
popular public GPG key servers.
http://www.openstack.org/projects/openstack-security/

49. Good Reads on Inc Response
Handling Compromised Components in an IaaS Cloud Installation
Aryan TaheriMonfared (aryan@uninett.no)
Martin G Jaatun (Martin.G.Jaatun@sintef.no)
http://www.journalofcloudcomputing.com/content/1/1/16/abstract

50. Object Storage Pain Points

Overwriting Data is Difficult, no stock
methods.

In event of aggressive evidence collection,
difficulty in identifying physical resources.

Potential loss of data in evidence
collection.

51. TPM + OpenStack = Trusted Pools

52. Zoned by Exposed Surface Area

SaaS is most secure

PaaS less so

IaaS least secure
Duh

53. Good Reading
Trusted Computing Pools
http://wiki.openstack.org/TrustedComputingPools
Putting Trust in OpenStack
http://www.openstack.org/summit/san-diego-2012/openstack-summit-
sessions/presentation/putting-trust-in-openstack

54. Parting thought
Consider public cloud vendors as you would
a Chinese fabrication supply chain.

They are cheap.

They are untrusted.

They are probably going to be around for
the foreseeable future.

55. Good Reading
A multi-level security model for partitioning
workflows over federated clouds
http://www.journalofcloudcomputing.com/content/1/1/15