Puppet Camp NYC 2014: Safely storing secrets and credentials in Git for use by Puppet: The BlackBox project (Intermediate) - Thomas A. Limoncelli, Stack Exchange
"Safely Storing Secrets and Credentials in Git for use by
Puppet: The BlackBox Project" presented by Thomas A. Limoncelli, Stack Exchange at Puppet Camp NYC 2014
Similar to Puppet Camp NYC 2014: Safely storing secrets and credentials in Git for use by Puppet: The BlackBox project (Intermediate) - Thomas A. Limoncelli, Stack Exchange
Chromium Sandbox on Linux (BlackHoodie 2018)Patricia Aas
Similar to Puppet Camp NYC 2014: Safely storing secrets and credentials in Git for use by Puppet: The BlackBox project (Intermediate) - Thomas A. Limoncelli, Stack Exchange (20)
Connector Corner: Accelerate revenue generation using UiPath API-centric busi...
Puppet Camp NYC 2014: Safely storing secrets and credentials in Git for use by Puppet: The BlackBox project (Intermediate) - Thomas A. Limoncelli, Stack Exchange
1. The BlackBox project
Safely storing secrets and credentials in Git
for use by Puppet
Tom Limoncelli, SRE, StackExchange.com
Blog: EverythingSysadmin.com
7. ● Laptops get stolen.
● Workstations have guest accounts
● “Circle of Trust” now includes:
○ Everyone with admin access to workstations.
■ Your desktop support people?
○ Everyone with admin access to your git server:
■ Server team, storage team, backup team
○ Everyone you collaborate with that wants read-only
access to Puppet manifests.
8. You have 3 bad options:
1. Deny git access. (Hurts collaboration)
2. Permit git access. (Hurts security)
3. Email individual files. (Hurts… just hurts)
9. Option 4: Encrypt secret parts
● If a file contains secrets, encrypt before
checking into Git.
● Need to edit a secret?
○ Decrypt - Edit - Encrypt
10. What about Puppet master?
● After “git pull”, decrypt all files.
○ Automate this as part of CI.
● Files are unencrypted “at rest”.
● This does not decrease security:
○ No worse than what we were doing before.
○ If you can break into root or puppet on the master,
you’ve already won.
12. Easy, right?
Decrypt:
gpg -q --decrypt -o secret.crt secret.crt.gpg
Encrypt:
gpg --yes --trust-model=always --encrypt
-o secret.crt.gpg $(<keynames) secret.crt
● ...and don’t make any typos when entering the command
● ...and don't accidentally check in the unencrypted version
13. Security is 1% technology plus 99% following
the procedures correctly.
Any process with more than 1 step probably
won't be followed consistently most of the time.
Related reading: "Why Johnny Can't Encrypt: A Usability Evaluation of PGP
5.0”, Alma Whitten", Usenix Security 1999
15. User commands:
Decrypt for editing:
blackbox_edit_start.sh file
Encrypt when done:
blackbox_edit_end.sh file
16. First time a file is encrypted:
Enroll a file into the system:
blackbox_register_new_file.sh file
17. Commands that act on all GPG files:
Decrypt all files: (for use on puppet master)
blackbox_postdeploy.sh
Re-encrypt all files: (after new users added)
blackbox_update_all_files.sh
18. Everyone has their own key
This doesn’t use “symmetric encryption” where
there is one passphrase to decrypt/encrypt all
files.
We maintain a keyring of:
● Each person that should have access.
● A key for the Puppet master.
19. Indoctrinate a new user:
1. New user does this:
● Create GPG key.
● Add their username@host to blackbox-admins.
txt
● git commit -a
(Currently a doc, not a script. Patches gladly accepted.)
20. Indoctrinate a new user:
2. Existing admin does this:
$ gpg --import keyrings/live/pubring.gpg
$ blackbox_update_all_files.sh
$ git commit -a
30. Code is open source as of TODAY
● Entirely written in bash.
● MIT License.
● Download it now:
○ https://github.com/StackExchange/blackbox
31. In the project’s first 9 months:
StackExchange/ServerFault has eliminated
plaintext secrets in our Puppet git repo.
● 7 SREs+Devs sharing the repo securely.
● 50+ files now stored encrypted.
○ Mostly SSL certs and SSH private keys.
● 40+ individual passwords/API keys:
○ Everything from SNMP communities, SaaS API
keys, and many many passwords.
32. Future plans
❏ Open source scripts.
❏ More usability enhancements.
❏ Better setup documentation.
33. Join the open source project
http://github.com/StackExchange/blackbox
34. Q&A
URLs from this talk:
https://github.com/StackExchange/blackbox
EverythingSysadmin.com
35. Shameless plug
Pre-order now! Save 35%
Ships in September.
informit.com/TPOSA
Discount code TPOSA35
Read “rough cuts” today:
safaribooksonline.com
36. Q&A
URLs from this talk:
https://github.com/StackExchange/blackbox
EverythingSysadmin.com
informit.com/TPOSA (code TPOSA35)
37. ● Easier transition. No Puppet code changes
for big files like SSL certs.
● Faster. Zero run-time performance impact
on master.
● eyaml didn’t exist when we started.
Why didn’t we use eyaml?