Ever wondered how to quickly and efficiently roll over all of your 1000 servers’ SSH keys? How to securely manage diskless systems and horizontally / vertically scale their credentials? This talk will introduce a simple approach that combines some hardware support, a little cryptography and SaltStack to help operationalise the management of all the secrets in your cloud

1. Managing server secrets at scale
with SaltStack and a vaultless password manager
Ignat Korchagin
@secumod

2. $ whoami
● Platform engineer at Cloudflare
● Passionate about security and crypto
● Enjoy low level programming

3. Disclaimer

4. So you have a server

5. So you have a server
You need:
● server SSH key

6. So you have a server
You need:
● server SSH key
● configuration management key

7. So you have a server
You need:
● server SSH key
● configuration management key
● disk encryption key

8. So you have a server
You need:
● server SSH key
● configuration management key
● disk encryption key
● some server credentials

9. So you have a server
You need:
● server SSH key
● configuration management key
● disk encryption key
● some server credentials
● probably more…

10. So you have a server
You need:
● server SSH key
● configuration management key
● disk encryption key
● some server credentials
● probably more…
… so at least 5 keys… and that’s per server

11. So you have a datacentre

12. So you have datacentre(s)

13. So you have datacentre(s)

14. Cloudflare network today
120+
Data centers globally
2.5B
Monthly unique visitors
10%
Internet requests
everyday
10MM
Requests/second
websites, apps & APIs
in 150 countries
7M+

15. So you have datacentre(s)

16. Where do keys live?

17. Where do keys live?

18. Where do keys live?

19. Keys in configuration management
#!yaml|gpg
root-password: |
-----BEGIN PGP MESSAGE-----
Version: GnuPG v1
hQIMA221uplZYlMdARAA583Z4o3xZawWzK8yUYJKBEkMQD/i+RRn7A0+h8SEmsov
QkrxgeaCWfIZ5pRpCpVOK1SWZGi0dzWkWe1DeNisawv5X/VUG3d5ej1xtAD4kBTy
AzcnFft7QfIsV8C+jguHYGITU++pFVAgEdGrb09mf6SEDaAGJhOq01BmHccw0Pat
rBH/+gvD155F7sxM/BBQwL25ZjtC+8jUsp1bUcTQVofsy6kTVRNSS4hO4UNtMuMQ
hYf6UAPaJv3PhFXKYYu0tEp2THZVTlUtTjyKAZrNiKyRpC/0exbJjJMqkYmmUG9r
yP1CvubJnmHda2u42981dK3pz5TlLEO4MrBry6vynN0TJfXwn1nt7YMVatiViQb9
UK5NDbjVKBBE6KkN28kSJtsTkCOM7+RztjLdf+7ZWzwxFV5EkM+2SLPIhQFCMjRG
...

20. Keys in configuration management

21. Keys in configuration management
● Bootstrap configuration management

22. Keys in configuration management
● Bootstrap configuration management
● Does not scale for unique keys

23. Keys in configuration management
● Bootstrap configuration management
● Does not scale for unique keys
#!yaml|gpg
root-password: |
{% if grains[‘hostname’] == ‘baredog’ %}
-----BEGIN PGP MESSAGE-----
...
{% elif grains[‘hostname’] == ‘cheesyonion’ %}
-----BEGIN PGP MESSAGE-----
...
{% elif grains[‘hostname’] == … %}

24. Keys on local disk

25. Keys on local disk
#!/bin/bash
# super nitty startup script, fully automated !!!
if [ ! -f /etc/server_key ]; then
dd if=/dev/urandom of=/etc/server_key bs=1 count=32
fi
# don’t forget SSH
if [ ! -f /etc/ssh/id_rsa ]; then
ssh-keygen -f /etc/ssh/id_rsa
fi
● Usually generated by startup scripts

26. Entropy

27. Keys on local disk

28. Keys on local disk
● Not suitable for some key types

29. Keys on local disk
● Not suitable for some key types
○ root passwords

30. Keys on local disk
● Not suitable for some key types
○ root passwords
● Does not play well with disk
encryption

31. Keys on local disk
● Not suitable for some key types
○ root passwords
● Does not play well with disk
encryption
○ decrypt configuration management key

32. Keys on local disk
● Not suitable for some key types
○ root passwords
● Does not play well with disk
encryption
○ decrypt configuration management key
● What about diskless/stateless
systems?

33. Encrypted disks
server
disk 1 disk 2 disk N
encryptstore

34. Unified Extensible Firmware Interface

35. Unified Extensible Firmware Interface
● Aka BIOS 2.0

36. Unified Extensible Firmware Interface
● Aka BIOS 2.0
● Standard pre-OS environment

37. Unified Extensible Firmware Interface
● Aka BIOS 2.0
● Standard pre-OS environment
● Extensible (you can write your own apps)

38. Unified Extensible Firmware Interface
● Aka BIOS 2.0
● Standard pre-OS environment
● Extensible (you can write your own apps)
● Supported by most major OSes

39. Unified Extensible Firmware Interface
● Aka BIOS 2.0
● Standard pre-OS environment
● Extensible (you can write your own apps)
● Supported by most major OSes
● Provides many advanced features

40. Unified Extensible Firmware Interface
● Aka BIOS 2.0
● Standard pre-OS environment
● Extensible (you can write your own apps)
● Supported by most major OSes
● Provides many advanced features
○ UEFI variables

41. UEFI variables
● Backed by flash memory on platform
firmware chip

42. UEFI variables
● Backed by flash memory on platform
firmware chip
● Can store standard and custom
(OEM/user) data

43. UEFI variables
● Backed by flash memory on platform
firmware chip
● Can store standard and custom
(OEM/user) data
● Can be accessed after OS kernel booted

44. UEFI variables
● Backed by flash memory on platform
firmware chip
● Can store standard and custom
(OEM/user) data
● Can be accessed after OS kernel booted
● Have built-in support in Linux

45. UEFI variables in Linux
# not needed for systemd-based Linux distributions
mount -t efivarfs efivarfs /sys/firmware/efi/efivars
# need to prepend data with 4 byte attr and put an “owner” GUID
cat <(printf "x07x00x00x00") <(cat mydata.bin) >
/sys/firmware/efi/efivars/mydata-<some GUID>

46. UEFI variables in Linux
● always available
# not needed for systemd-based Linux distributions
mount -t efivarfs efivarfs /sys/firmware/efi/efivars
# need to prepend data with 4 byte attr and put an “owner” GUID
cat <(printf "x07x00x00x00") <(cat mydata.bin) >
/sys/firmware/efi/efivars/mydata-<some GUID>

47. UEFI variables in Linux
● always available
● can be accessed in early boot stages
# not needed for systemd-based Linux distributions
mount -t efivarfs efivarfs /sys/firmware/efi/efivars
# need to prepend data with 4 byte attr and put an “owner” GUID
cat <(printf "x07x00x00x00") <(cat mydata.bin) >
/sys/firmware/efi/efivars/mydata-<some GUID>

48. UEFI variables in Linux
● always available
● can be accessed in early boot stages
● however, may have limited storage
# not needed for systemd-based Linux distributions
mount -t efivarfs efivarfs /sys/firmware/efi/efivars
# need to prepend data with 4 byte attr and put an “owner” GUID
cat <(printf "x07x00x00x00") <(cat mydata.bin) >
/sys/firmware/efi/efivars/mydata-<some GUID>

49. Keys in cryptography

50. Keys in cryptography
derive

51. Keys in cryptography

52. Key derivation functions
master
key
KDF

53. Key derivation functions
master
key
KDF

54. Key derivation functions
master
key
KDF
string1

55. Key derivation functions
master
key
KDF
string1
key1

56. Key derivation functions
master
key
KDF
string1 string2
key1

57. Key derivation functions
master
key
KDF
string1 string2
key1 key2

58. Generating key pairs
Deterministic
CSPRNG

59. Generating key pairs
Deterministic
CSPRNG
seed1

60. Generating key pairs
Deterministic
CSPRNG
seed1
keypair1

61. Generating key pairs
Deterministic
CSPRNG
seed1 seed2
keypair1

62. Generating key pairs
Deterministic
CSPRNG
seed1 seed2
keypair1 keypair2

63. Introducing gokey tool
master
seed
realm
For example
“ssh”, “saltstack”,
“disk encryption”
etc

64. Introducing gokey tool
HKDF
master
seed
realm
For example
“ssh”, “saltstack”,
“disk encryption”
etc

65. Introducing gokey tool
HKDF
master
seed
realm
Deterministic
CSPRNG
For example
“ssh”, “saltstack”,
“disk encryption”
etc

66. Introducing gokey tool
HKDF
master
seed
realm
Deterministic
CSPRNG
ssh
key
For example
“ssh”, “saltstack”,
“disk encryption”
etc

67. Introducing gokey tool
HKDF
master
seed
realm
Deterministic
CSPRNG
ssh
key
For example
“ssh”, “saltstack”,
“disk encryption”
etc
Deterministic
CSPRNG

68. Introducing gokey tool
HKDF
master
seed
realm
Deterministic
CSPRNG
ssh
key
For example
“ssh”, “saltstack”,
“disk encryption”
etc
saltstack
key
Deterministic
CSPRNG

69. Key management
● Provisioning process ensures a master seed is
generated and stored in UEFI on first boot

70. Key management
● Provisioning process ensures a master seed is
generated and stored in UEFI on first boot
● Startup scripts “recover” (derive from master seed)
configuration management credential (key)

71. Key management
● Provisioning process ensures a master seed is
generated and stored in UEFI on first boot
● Startup scripts “recover” (derive from master seed)
configuration management credential (key)
● Configuration management “recovers” all other keys

72. Key management
root-password: |
{% if grains[‘hostname’] == ‘baredog’ %}
-----BEGIN PGP MESSAGE-----
...
{% elif grains[‘hostname’] == ‘cheesyonion’ %}
-----BEGIN PGP MESSAGE-----

73. Key management
root-password: |
{% if grains[‘hostname’] == ‘baredog’ %}
-----BEGIN PGP MESSAGE-----
...
{% elif grains[‘hostname’] == ‘cheesyonion’ %}
-----BEGIN PGP MESSAGE-----
root-password: {% gokey(‘root-password’) %}
ssh-key: {% gokey(‘ssh’) %}

74. nodekey custom state
def managed(name, type, label, mode = '600'):
keydata = __salt__['cmd.run_all']('gokey -p {seed_pass} -s {seed_file} -skip 4 -t
{key_type} -r {label}'.format(seed_pass=_SEED_PASS, seed_file=seed_file,
key_type=type, label=label), rstrip=False)
if keydata['retcode'] != 0:
ret = {'name': name}
if __opts__['test']:
ret['result'] = None
else:
ret['result'] = False
ret['comment'] = 'gokey command failed'
ret['changes'] = {'retcode': keydata['retcode'], 'stderr': keydata['stderr']}
return ret
return __states__['file.managed'](name=name, mode=mode, show_changes=False,
makedirs=True, contents_newline=False, contents=keydata['stdout'])

75. nodekey custom state
/etc/ssh/ssh_host_ecdsa_key:
nodekey.managed:
- type: ec256
- label: ssh-host-key
/etc/ssh/ssh_host_rsa_key:
nodekey.managed:
- type: rsa2048
- label: ssh-host-key
/etc/luks/luks.key:
nodekey.managed:
- type: raw
- label: luks-key

76. secrets renderer
#!jinja|yaml|secrets
testing:
secret: |
-----BEGIN DERIVE SECRET-----
seed: |
is1FAXoWOR7O3a3rvNqsjlyu5Y3mAwm0Ekwcf…
...
password: hunter2
type: pass
-----END DERIVE SECRET-----

77. secrets renderer
$ salt -L 'foo,bar' pillar.get testing
foo:
----------
secret:
Im$kEJz+8u
bar:
----------
secret:
lAP4a}ztOX

78. secrets renderer
#!jinja|yaml|secrets
testing:
secret: |
-----BEGIN DERIVE SECRET-----
seed: |
is1FAXoWOR7O3a3rvNqsjlyu5Y3mAwm0Ekwcf…
...
password: hunter2
type: pass
-----END DERIVE SECRET-----

79. secrets renderer
#!jinja|yaml|gpg|secrets
testing:
secret: |
-----BEGIN DERIVE SECRET-----
seed: |
<PGP CIPHERTEXT HERE>
password: |
<PGP CIPHERTEXT HERE>
type: pass
-----END DERIVE SECRET-----

80. secrets renderer

81. Adding a service key
server server

82. Adding a service key
/etc/myservice/myservice.key:
nodekey.managed:
- type: ec256
- label: myservice

83. Adding a service key
/etc/myservice/myservice.key:
nodekey.managed:
- type: ec256
- label: myservice
#!jinja|yaml|gpg|secrets
myservice:
privatekey: |
-----BEGIN DERIVE SECRET-----
seed: |
-----BEGIN PGP MESSAGE-----
...
password: |
-----BEGIN PGP MESSAGE-----
...
type: ec256
-----END DERIVE SECRET-----
/etc/myservice/myservice.key:
file.managed:
- mode: 0600
- contents_pillar: |
myservice:privatekey

84. Adding a service key
server server

85. Adding a service key
server server

86. Adding a server
server server

87. Adding a server
server server server

88. Adding a server
server server server

89. Rotate a specific service key
server server server

90. Rotate a specific service key
server server server
ssh-key: {% gokey(‘ssh’) %} ssh-key: {% gokey(‘ssh-v2’) %}

91. Rotate a specific service key
server server server
ssh-key: {% gokey(‘ssh’) %} ssh-key: {% gokey(‘ssh-v2’) %}

92. Rotate all keys on a server
server server server

93. Rotate all keys on a server
server server server
rotate EFI seed

94. Rotate all key on a server
server server server
rotate EFI seed

95. Encrypted disks (previously)
server
disk 1 disk 2 disk N
encryptstore

96. Encrypted disks
server
disk 1 disk 2 disk N
encryptencrypt

97. Conclusions

98. Conclusions
● Decouple key storage from regular storage

99. Conclusions
● Decouple key storage from regular storage
● Decouple key contents from Salt

100. Conclusions
● Decouple key storage from regular storage
● Decouple key contents from Salt
● Efficient key management on stateless
systems with Salt

101. Conclusions
● Decouple key storage from regular storage
● Decouple key contents from Salt
● Efficient key management on stateless
systems with Salt
● Easy to add new or rotate existing keys

102. Conclusions
● Decouple key storage from regular storage
● Decouple key contents from Salt
● Efficient key management on stateless
systems with Salt
● Easy to add new or rotate existing keys
● Better security guarantees

103. Conclusions
● Decouple key storage from regular storage
● Decouple key contents from Salt
● Efficient key management on stateless
systems with Salt
● Easy to add new or rotate existing keys
● Better security guarantees
https://github.com/cloudflare/gokey

104. Thank you