Automatically unseal Vault clusters as a Keybase team. We want to automate the unseal of our on-premise Vault clusters. How can we securely distribute Shamir unseal keys to the team so we can unseal our Vault when we are on-call? How did we initialize our production system in a such way that 2 out 4 people are needed to "unseal the Vault"? We are using Keybase.io, and automated Vault on Consul cluster, with an Ansible/Vagrant environment to teach and practice. - Vagrant (tested on Mac) - Consul OSS - Vault OSS - Keybase (vault operator init, vault unseal, KBFS) - Ansible (Brian Shumate's roles, custom roles) - Packer (hardened Centos 7) @bbaassssiiee https://github.com/dockpack/keybase_unseal https://github.com/dockpack/vault_dojo

1. 
Bas Meijer
Software Engineer/DevOps Coach
HUG Amsterdam Co-Organizer
Ansible Ambassador
@bbaassssiiee
08:00 - 08:30 GMT
Friday, February 21
HashiTalks 2020
Friday, February 21 08:00 - 08:30 GMT

2.  @bbaassssiiee
Keybase Auto Unseal - Bas Meijer
HashiTalks 2020

3. 
Purpose
1. Automate provisioning Vault on Consul cluster
2. Securely store the keys to the Vault kingdom
3. Enable team to unseal automatically
@bbaassssiiee 3

4. 
Structure of this presentation
• Vault setup background info
• Start Vault, Initialize, Unseal
• Use the CLI, UI manually
• Automate and/or Secure?
• A Dilemma?
• Open Source Reference Project
@bbaassssiiee 4

5. Copyright © 2019 HashiCorp
Starting the Vault Server
Write a server configuration file
Start the server: vault server -config=<config_file_path>
Initialize the server (generate the unseal keys & an initial token)
Unseal the Vault server
Log in
5

6. Copyright © 2019 HashiCorp
▪ Initialization is the process of configuring the Vault:
‣ Encryption key gets generated
‣ Unseal keys are created
‣ Initial root token is setup
Vault Server Initialization
6

7. Copyright © 2019 HashiCorp
▪ When a Vault server is started, it starts in sealed - doesn't know how to
decrypt the data
▪ Unsealing is the process of constructing the master key necessary to read the
decryption key to decrypt data
▪ Why?
‣ The data stored by Vault is encrypted with encryption key
‣ The encryption key is encrypted with master key
‣ The master key is NOT stored anywhere
Seal / Unseal
17
7
When a Vault server is started, it starts in sealed mode - it doesn't know
how to decrypt the data

8. Copyright © 2019 HashiCorp
Shamir's Secret Sharing
12
Master Key Encryption Key
Protected by a master key
Key Shares
(Unseal keys)
Bob
James
Jennifer
Pam
Tom
A threshold of unseal keys are required to unseal Vault so that
the key to the kingdom won't fall into one person's hand!
Stephan
Kitty
Rudolf
Lars
Marjan
[*]
8

9. Copyright © 2019 HashiCorp 15
Initialize a Vault
Server via CLI
$ vault operator init
Unseal Key 1: oL8fJP4KreJPbZWIgui340j5bNclip9zGVcYIzElsoF1
Unseal Key 2: Ke9VZlGzuVaf4HJB8c9KQR2j8rFTBALV1fD3hjE5pHoY
Unseal Key 3: 4X6Ja/RpMwNabYzklZKxxXVznLQFGgSiVW7Wx8LWOkQn
Unseal Key 4: dhI04g8dIQSXI11BIC6Gtwy/QaJWhVYoFYwKF9UI6axO
Unseal Key 5: IQ2Ls630Sjd/oEQyTmwwpuFEUTiJP4FX2UI3uZMZoa+x
Initial Root Token: s.arHAbYvyeZQH8StLc5OHtbt4
Vault initialized with 5 keys and a key threshold of 3. Please
securely distribute the above keys. When the Vault is re-sealed,
restarted, or stopped, you must provide at least 3 of these keys
to unseal it again.
...
Terminal
9

10. Copyright © 2019 HashiCorp
Initializing Vault (1 of 2)
22
10

11. Copyright © 2019 HashiCorp
Initializing Vault (2 of 2)
23
11

12. Copyright © 2019 HashiCorp
Initializing Vault via UI (2 of 2)
14
12

13. 
---
- name: initialize Hashicorp Vault
delegate_to: "{{ groups.vault_instances[0] }}"
run_once: true
when: vault_status == '501'
no_log: true
environment:
VAULT_ADDR: "https://{{ groups.vault_instances[0] }}:{{ vault_port }}"
VAULT_CACERT: "{{ vault_tls_config_path }}/{{ vault_tls_ca_file }}"
command: |
vault operator init
-key-shares={{ key_shares }}
-key-threshold={{ key_threshold }}
-format=json
register: inited
tags:
- init
...
teamshare.yml
Initializing Vault via Ansible
@bbaassssiiee 13

14. Copyright © 2019 HashiCorp
Starting the Vault Server
Write a server configuration file
Start the server: vault server -config=<config_file_path>
Initialize the server (generate the unseal keys & an initial token)
Unseal the Vault server
Log in
14

15. Copyright © 2019 HashiCorp
Unsealing Vault
24
15

16. Copyright © 2019 HashiCorp 19
Unsealing via CLI
(1 of 3)
$ vault operator unseal
Unseal Key (will be hidden):
Key Value
--- -----
Seal Type shamir
Initialized true
Sealed true
Total Shares 5
Threshold 3
Unseal Progress 1/3
Unseal Nonce 3c0b2c85-6d22-54e4-87ce-249061dd9d1c
Version 1.1.0
HA Enabled false
Terminal
16

17. Copyright © 2019 HashiCorp 20
Unsealing via CLI
(2 of 3)
$ vault operator unseal
Unseal Key (will be hidden):
Key Value
--- -----
Seal Type shamir
Initialized true
Sealed true
Total Shares 5
Threshold 3
Unseal Progress 2/3
Unseal Nonce 3c0b2c85-6d22-54e4-87ce-249061dd9d1c
Version 1.1.0
HA Enabled false
Terminal
17

18. Copyright © 2019 HashiCorp 21
Unsealing via CLI
(3 of 3)
$ vault operator unseal
Unseal Key (will be hidden):
Key Value
--- -----
Seal Type shamir
Initialized true
Sealed false
Total Shares 5
Threshold 3
Version 1.1.0
Cluster Name vault-cluster-ad3f168d
Cluster ID 9fcbb3bc-6d9b-98f5-3f2e-a0cf1040a260
HA Enabled false
Terminal
18

19. Copyright © 2019 HashiCorp 21
Unsealing via CLI
(3 of 3)
$ vault operator unseal
Unseal Key (will be hidden):
Key Value
--- -----
Seal Type shamir
Initialized true
Sealed false
Total Shares 5
Threshold 3
Version 1.1.0
Cluster Name vault-cluster-ad3f168d
Cluster ID 9fcbb3bc-6d9b-98f5-3f2e-a0cf1040a260
HA Enabled false
Terminal
Automate
Unsealing
with
Ansible
---
# teamshare
- name: 'unseal Hashicorp Vault with teamshare unseal'
when: vault_status == '503' and not shamir
environment:
VAULT_ADDR: "https://{{ groups.vault_instances[0] }}:{{ vault_port }}"
VAULT_CACERT: "{{ vault_tls_config_path }}/{{ vault_tls_ca_file }}"
command: "vault operator unseal {{ item }}"
with_items: "{{ unseal_keys_hex }}"
no_log: true
tags:
- unseal
@bbaassssiiee19
Ansible Tip:
Use this action attribute
to avoid leaking sensitive
information into syslog.
no_log: true

20. Copyright © 2019 HashiCorp
Starting the Vault Server
Write a server configuration file
Start the server: vault server -config=<config_file_path>
Initialize the server (generate the unseal keys & an initial token)
Unseal the Vault server
Log in
20

21. Copyright © 2019 HashiCorp
Vault Server Setup Workflow Recap
Write a server configuration file
Start the server: vault server -config=<config_file_path>
Initialize the server (generate the unseal keys & an initial token)
Unseal the Vault server
Log in
Initial Setup
Only the the Vault server
was restarted, or sealed
intentionally
Only when the Vault
server was restarted,
or sealed intentionally
21

22. Copyright © 2019 HashiCorp
Challenge
4
Unsealing process requires a threshold of unseal keys
No single person holds the key to the Vault kingdom!
Unsealing is a manual process and become painful when you have
multiple Vault clusters
22

23. Copyright © 2017 HashiCorp
Auto-Unseal Vault
Cloud based key Master key Encryption key
▪ Instead of using shared keys based on
Shamir's Secret Sharing algorithm, use
the trusted cloud-based encryption key
to protect the master key
▪ Supported cloud services today:
‣ AliCloud KMS
‣ AWS KMS
‣ Azure Key Vault
‣ GCP Cloud KMS
▪ Use Transit secrets engine
Shared keys Master key Encryption key
23

24. Copyright © 2019 HashiCorp
▪ Vault Enterprise integrates with HSM to take advantage of HSMs to provide three
pieces of special functionality:
‣ Master Key Wrapping: Vault protects its master key by transiting it through the
HSM for encryption rather than splitting into key shares
‣ Auto Unsealing: Vault stores its HSM-wrapped master key in storage, allowing for
automatic unsealing
‣ Seal Wrapping to provide FIPS KeyStorage-conforming functionality for Critical
Security Parameters
HSM Support
16
24

25. Copyright © 2019 HashiCorp
▪ Protect encryption key with master key
▪ HSM encryption key protects master key in place of Shamir's Secret Sharing
▪ Communication with HSM via PKCS #11 API to decrypt the master key
Master Key Wrapping and Auto-unseal
17
HSM key Master keys Encrypted
keys
PKCS11
25
PKCS#11

26. 
Constraints
1. On-Premise Infrastructure
2. Independent from Cloud platform
3. Redundancy & Flexibility
4. Open Source/Free
@bbaassssiiee 26

27. 
Searching the internet...
@bbaassssiiee 27

28.  @bbaassssiiee
Integration
★ 6.4k ★ 41.7k ★ 9.8k ★ 19.5k
★ 18.5k★ 15k
28

29. @bbaassssiiee
VagrantPacker Consul Vault
dockpack/centos7 Image Dev Environment Clustered Storage Secrets Management
29

30. Keybase
@bbaassssiiee
• Every account has a public history
• Keybase Team Trust
• User-Friendly PGP Encryption
• Encrypted File System KBFS
• Keybase Command Line
Keybase is secure messaging and file-sharing.
30

31. @bbaassssiiee
Ansible
ansible-galaxy install -r requirements.yml
---
- src: brianshumate.consul
- src: brianshumate.vault
- src: leonallen22.ansible_role_keybase
- src: dockpack.keybase_unseal
• There is a lot of yaml in the galaxy
• Automation after vault operator init
• Automation before vault install
31

32. @bbaassssiiee
Ansible
ansible-vault encrypt /keybase/team/$KEYBASE_TEAM/vault.json
export ANSIBLE_VAULT_PASSWORD_FILE=/keybase/team/$KEYBASE_TEAM/vault.pass
Note: Keybase has a safe place for the ansible-vault password file:
ansible-vault AES encrypted config files
Transparant use in automation
# This is the path where the encrypted JSON is shared.
vault_credentials: "/keybase/team/{{ keybase_team }}/vault.json"
include_vars: "{{ vault_credentials }}"
32

33. 
---
- name: save Vault credentials as pretty JSON
delegate_to: localhost
run_once: true
become: false
no_log: true
when: vault_status == '501'
copy:
dest: "{{ vault_credentials }}"
content: "{{ inited.stdout|from_json|to_nice_json }}"
mode: 0600
register: save_json
tags:
- init
- name: encrypt pretty JSON with ansible-vault
delegate_to: localhost
run_once: true
become: false
no_log: true
when: vault_status == '501'
environment:
# yamllint disable-line rule:line-length
ANSIBLE_VAULT_PASSWORD_FILE: "{{ lookup('env','ANSIBLE_VAULT_PASSWORD_FILE') }}"
command: "ansible-vault encrypt {{ vault_credentials }}"
tags:
- init
...
kbfs.yml
Encrypt^2
@bbaassssiiee 33

34. Keybase Auto Unseal
1.create accounts
2.create team
3.create sub-team for admins
4.add members
5.create vault.pass on KBFS
6.use role in playbook
github.com/dockpack/keybase_unseal
@bbaassssiiee 34

35. 
Keybase
35

36. @bbaassssiiee
---
shamir: true
keybase_team: dockpack.vault
kbt:
- basmeijer
- fbezema
- ksatirli
- ferhaty
export KBT_INDEX=1
export KBT_INDEX=3
export KBT_INDEX=0
export KBT_INDEX=2
Shamir Secrets Keybase Encrypted
Each Team Member sets their
environment variable
Define these groups_vars:
36

37. 
---
- name: initialize Hashicorp Vault
delegate_to: "{{ groups.vault_instances[0] }}"
run_once: true
when: vault_status == '501'
no_log: true
environment:
VAULT_ADDR: "https://{{ groups.vault_instances[0] }}:{{ vault_port }}"
VAULT_CACERT: "{{ vault_tls_config_path }}/{{ vault_tls_ca_file }}"
command: |
vault operator init
-key-shares={{ key_shares }}
-key-threshold={{ key_threshold }}
-format=json
-pgp-keys="keybase:{{ kbt[0] }},keybase:{{ kbt[1] }},keybase:{{ kbt[2] }},keybase:{{ kbt[3] }}"
register: inited
tags:
- init
...
shamir.yml
Initialize Vault with Shamir Secrets Keybase Encrypted
@bbaassssiiee 37

38. 
---
# shamir
- name: 'set kbt_index from env, rang in array of keybase team list kbt.'
when: vault_status == '503' and shamir|bool
delegate_to: localhost
run_once: true
set_fact:
kbt_index: "{{ lookup('env','KBT_INDEX') }}"
tags:
- unseal
- shamir
...
teamshare unseal
@bbaassssiiee
Which Encrypted Shamir Unseal Key is mine?
38

39. 
---
- name: 'decrypt unseal key based on kbt_index'
when: vault_status == '503' and shamir|bool
delegate_to: localhost
become: false
run_once: true
no_log: true
shell: |
set -o pipefail ;
ansible-vault view /keybase/team/{{ keybase_team }}/vault.json
| jq -r .unseal_keys_b64[{{ kbt_index }}]
| base64 --decode
| gpg -d
register: unseal_key
tags:
- unseal
- shamir
- name: 'unseal Hashicorp Vault with tags=unseal'
when: vault_status == '503' and shamir|bool
no_log: true
environment:
VAULT_ADDR: "https://{{ groups.vault_instances[0] }}:{{ vault_port }}"
VAULT_CACERT: "{{ vault_tls_config_path }}/{{ vault_tls_ca_file }}"
command: "vault operator unseal {{ unseal_key.stdout }}"
tags:
- unseal
- shamir
...
shamir unseal
Unseal with my Encrypted Shamir Unseal Key
@bbaassssiiee 39

40. ?
Vault
Consul
Shamir
Keybase
PGP
KBFS
Unseal Key
Ansible
Packer
Vagrant
Cloud
HSM
AWS KMS
Azure Key Vault
Ansible Vault
Keybase Teams Blockchain
IAM
github.com/dockpack/vault_dojo
vimeo.com/391099245
Playbook Role
Root Token
Unsealing
Secure?
Encryption
learn.hashicorp.com
https://github.com/dockpack/keybase_unseal/wiki
@bbaassssiiee 40

41. @bbaassssiiee
Demo
 41