Presentation from OpenStack 2018 Summit

1. Encrypt Your Volumes with Barbican

2. Agenda
• Introduction
• About OnRamp
• Why Encrypt?
• Key Managers
• How it Works
• Our Encryption Story
• Gotchas & Limitations
• Demo
• Q&A

3. Meet the Speaker
• Previously Built and Managed Lab
Environments for VMware
Certification Training (VCP5/6)
• Experience with OpenStack:
- Started with Icehouse on Ubuntu
(manually)
- Juno on Canonical MaaS + Juju
- Liberty on CentOS / RDO
- Newton on Platform9
• OpenStack Engineer for OnRamp
• Deployed Barbican and support for
Encrypted Volumes for our OpenStack
Based Public Cloud
Duncan Wannamaker
OpenStack Engineer
OnRamp

5. But, OpenStack is Far From Simple, Right?

6. Who Is OnRamp?
OnRamp is a HITRUST-certified data center services company specializing in
high security and compliant hybrid hosting.

7. OnRamp’s Virtual Private Cloud
■ Built on OpenStack®
■ HITRUST-certified
■ Control costs with capped maximum resource usage
■ Open-source APIs enable simple migrations and eliminate vendor lock-in
■ Hardened Linux and Windows Images
■ Retain full control of your private networks, virtual machines, and storage
Offers the ease-of-use of a public cloud, with the security of a private cloud.

8. Why Encrypt?
• Protect Data against Leaks
- Personal Health Information (PHI)
- Credit Card Payment Data (PCI)
- Intellectual Property
• Compliance Requirement
- HIPAA requires encryption for data at rest
and in-transit for PHI
- PCI also requires encryption for data at rest
and in-transit for cardholder data
- In shared hosting environments, each Tenant
must only have access to their own stuff
 Per-Tenant or Per-Volume encryption keys
facilitate this

9. Why Use a Key Manager?
• Security Best Practice
- You don’t leave keys in your car
- You shouldn’t leave your keys next to your encrypted data
• Compliance Requirement
- PCI-DSS Requirements:
 Store keys in the fewest possible locations
 Store secret and private keys used to encrypt/decrypt cardholder data
using a key encryption key at least as strong as the encryption key
itself
- HIPAA- HITRUST Requirements:
 Keys shall be stored separately from encrypted data
 Key manager systems should be physically protected by fewest
number of custodians necessary

10. What is Barbican?
Provides:
• ReST API for Secrets Management
• Pluggable Backends
- Simple Crypto
- PKCS#11 and KMIP (HSM)
• Integration with Nova, Cinder, and
Swift, Neutron, Heat, and many
other OS projects
• Integration with KeyStone for Auth
and RBAC
• Built to Scale
Does Not Provide:
• Graphical User Interface
• Key Splitting for Secure
Import/Export Plain Text Keys using
multiple Key Custodians
• Generation of X.509 Certificates
(Since Pike)
• Volume Encryption

11. What about Volume Encryption?
• LUKS - Linux Unified Key Setup
- Allows for multiple user keys or passwords per volume
 Master Key always stays the same
- Supports CPU Hardware Acceleration (it’s fast!)
- Uses CryptSetup and DM-CRYPT
 Decrypts full volume to a Local Block Device
 Protects iSCSI attached volumes
 Can also protect ephemeral storage if using LVM
• Queens = QEMU Native LUKS support
- QEMU 2.6 and LibVirt 2.2 introduce native LUKS support

12. Transparent Encryption in Nova and Cinder
Thanks to some fantastic work done by Johns Hopkins Applied
Physics Laboratory and others…
• Nova and Cinder Integration exists with Barbican
• Volume Decryption Happens on the Hypervisor instead of
within the Guest OS
- No Agent Required
- Works with Any Operating System
- Works with Bootable Volumes
- Protects Data at Rest and In-Transit to your hypervisor
- Every volume is protected by it’s own unique key

13. How it Works: Creating an Encrypted Volume
1. User Gets a token from Keystone
2. User Asks Cinder to create volume
(using token)
3. Cinder verifies the user token from
keystone
4. Cinder then asks Barbican for a key
5. Barbican checks the Cinder Token
against keystone
6. Barbican creates the secret and
returns secret HREF to Cinder
7. Cinder stores the returned secret
HREF into the volume metadata

14. How it Works: Mounting an Encrypted Volume
Nova Keystone Cinder Barbican
Fetch Secret
HREF from
Cinder
Validate
Request
Return Secret
HREF and
Attach
Volume
Receive
Secret HREF
Fetch Secret
from Barbican
Validate
Token
Validate
Token
Receive
User
Mount
Request
Validate
Request
Validate
Requestor
Return Secret
Mount
Volume using
Secret
1. Nova receives request to mount
volume with token
2. Nova Validates User Token
3. Nova Fetches the Secret HREF
from Cinder
4. Cinder Validates Nova Token
5. Cinder returns Secret HREF to
Nova
6. Nova Fetches Secret from
Barbican
7. Barbican Validates Nova Token
8. Barbican Returns Secret
9. Nova Mounts Volume using
Secret

15. OnRamp Encryption Story
• Followed Documentation
- Wait, these docs are terrible! 
 Thankfully, Barbican Devs on IRC are very helpful! 
• KeyStone:
- Added Endpoints to KeyStone
- Added Barbican Service User
- Added Creator Role
• Installed the Barbican-API:
- Configured Barbican’s database server
- Configured Barbican
- Barbican CLI Access Works!

16. OnRamp Encryption Story
• Cinder Integration Issues:
- Blank Encrypted Volumes were created successfully
- But they could not be created from an image
• Nova Integration Issues:
- Nova was unable to attach any encrypted volumes
- It wasn’t even trying to talk to Barbican

17. Issue 1: Key orders not getting to Barbican
cinder-api
Cloud-Hosted
cinder-volume
On-Premise
SSH VPN barbican
On-Premise
Key Orders
LAN
Encrypted
Volume
Cinder DB
Secret HREF: null
key request
for key 0000…
href = all zeros!
Mount
Failed
no key 0000…
Create
Volume
Mount Request

18. Issue 1 Fix: Key orders not getting to Barbican
cinder-api cinder-volume
forwarder
SSH VPN barbican
Key Order
LAN
Encrypted
Volume
Mount
Cinder DB
Secret HREF: valid
Key Order
Cloud-Hosted On-Premise On-Premise
Secret HREFSecret HREF
Create
Volume
Mount Request
valid href request secretreturn secret

19. Issue 2: Nova Not Talking to Barbican
• Nova error was mentioning a Fixed Key not being defined
• ConfKeyManager (Nova Default for Volume Encryption)
• Single fixed key for all volumes
• Used for testing to substitute in for a real key manager,
such as Barbican! Not for production.
• Setting the api_class in nova.conf would always use
ConfKeyManager and ignore the setting to use barbican
• Submitted Bug:
• https://bugs.launchpad.net/nova/+bug/1704875
• Fixed in Pike or newer!
• Manual Fix for older releases is to comment out lines 27-31
in nova/keymgr/__init__.py

20. Some Gotchas using Encrypted Volumes
• Live Migration does not work and is dangerous
- Volume mounts without decryption on new host which will cause
corruption if this is an active file system
- Fixed in Queens!
 Previous releases use symlinks on hypervisor
 New Method uses QEMU Native LUKS support
• Barbican Doesn’t Start after Reboot (CentOS Specific)
- RDO Packaging Issue
- Create /etc/tmpfiles.d/barbican.conf:
 d /var/run/barbican 0755 barbican barbican –
- Bug report submitted…

21. Some Limitations using Encrypted Volumes
• No mechanism to rotate volume encryption keys
- Manual process of creating a new volume and copying over the
contents
- LUKs supports multiple user keys, so the capability is there
• No UI for key management in horizon
- for securely exporting and importing split-keys
- for managing key ACL’s
- for managing key expiration and revocation

22. Tips for Running in Production
• Make sure your key manager and database are secured in a
locked cabinet with limited physical access
• Use a private barbican instance not accessible to tenants
• Automated database backups
• Use a highly available database cluster such as Galera
• Use multiple barbican-api nodes behind a load balancer
• Use SSL to protect key requests in-transit to hypervisors

23. Demo

24. Questions?

25. Barbican PoC Guide and Resources
• Step by step instructions in GitHub:
https://github.com/dwannamaker-onr/openstack-queens-barbican-guide
• Video walkthrough available:
- YouTube Search: “Queens Barbican PoC”
 Installing Packstack for Queens (Part 1)
 Installing the Barbican-API (Part 2)
• IRC Channel:
- #openstack-barbican on FreeNode

26. Thank you! Stay In Touch:
duncan@randomhack.com
@randomhack
888.667.2660
sales@onramp.com
www.onr.com
www.randomhack.com