Deep discussion of why IT infrastructure is hard to automate with very specific examples building and rotating an internal PKI infrastructure.

1. The Messy
Underlay Dilemma
Lessons Learned Securing K8s
Rob Hirschfeld, @zehicle

2. Hang on tight!
We’re going deep.
To automated live
encryption key rotation

3. Is Operating
Kubernetes
HARD?
DF
No.
But underlay is hard.
From http://www.slideshare.net/rhirschfeld/
Containers, Orchestration and Security, Oh My!

4. Underlay vs Overlay
Platform
Overlay
Infrastructure
Underlay
Ready State
0
Ready
Prerequisites
1
Prereq
Cluster API &
Control Services
2
Control
Worker Nodes
3
Nodes
Cluster Add-ons
4
Add-Ons
User Applications
5
Apps
Application
Overlay
Underlay = Crust
Overlay = Filling
App = Topping
Underlay components are the
operational integrations and
prerequisites that go into building a
system to before we can install a
platform.

5. Why is Underlay Hard?
It’s Sequential, Multi-node & Environment Specific
Unlike development environments, production cannot overlook integration points
HA/LB PKI DNS SDN IPAM BMC RAID BIOS
Highly Available &
Load Balanced
Public Key
Infrastructure
Domain Name
Servers
Software Defined
Networks
IP Address
Management
Out of Band
Management
Drive Arrays Firmware
Even in cloud only deployments, these critical components for production platforms
and applications require a level of different systems thinking.
Strong underlay builds an IT foundation.

6. Platform &
Infrastructure
Underlay
Ready State
0
Ready
Prerequisites
1
Prereq
Cluster API &
Control Services
2
Control
Worker Nodes
3
Nodes
Cluster Add-ons
4
Add-Ons
User Applications
5
Apps
Application
Overlay
DevOps Is Struggling
Developers don’t want do this
infrastructure specific stuff
Companies are turning to
containers and application
platforms (like Kubernetes) to
abstract the messy underlay.
While platforms hide complexity
from developers, the issues still
need to be addressed by Ops.

7. What makes underlay hard?
Let’s look at Internal PKI

8. Protection via Tunnel Level Security (TLS)
This is pretty complex stuff….
At a very basic level:
1. Send public key to client
2. Client encrypts token with public key
3. Client returns encrypted package
4. Server decrypts token with private key
5. Server uses token to encrypt tunnel
Server
Private
Key
Client
Public
1
2 Token
4 Token
3
5
T
L
S

9. Trusted
3rd
Party
Trust
Anchor
Trust
Chain of Trust in Public Key Infrastructure (PKI)
PKI is doing something amazing!
It establishes asynchronous trust
By relying on strong encryption
And Trust Anchors.
Server
Private
Key
Cert
Auth Client
Public
T
L
S
Digital
Signature
Root

10. Half of all Internet Traffic is encrypted! HTTPS > 50%
That’s great for public traffic where trust
is anchored / embedded into clients
What about the internal traffic?
We want a “narrow trust domain” so
there’s no embedded trust mechanism
and we maintain full control.
We also want to protect both sides.
Public
End Points
East-West Traffic
North-SouthTraffic
Back End Services
End User Clients

11. Server
Client
TrustTrust
Shared Root in Public Key Infrastructure (PKI)
Self-signing keys is not considered secure.
Internal PKI uses a shared root strategy
The private root of the CA must not be
known to in the trust relationship.
Members of the Trust Domain rely on the CA
to verify membership and identity.
External Trust Anchors are not desirable
because we want an exclusive Trust
Domain.
Private
Key
Public
T
L
S
D.
SigRoot
Private
Key
D.
Sig
Public
Root
Site
CA

12. Master
Node 1
Worker
Nodes
Worker
Nodes
Master
Node 2+
“Narrow” Trust Domain Limits Shared Roots
etcd
Kublet Proxy
API
Server
Controller
Controller
etcd
API
Server
Scheduler
Scheduler
User!
Illustration from Slideshare Rob Hirschfeld

13. Master
Node 1
Root
Worker
Nodes
Worker
Nodes
Master
Node 2+
RootRoot Root
Shared Roots Create Trust Zones
etcd
Kublet Proxy
API
Server
Controller
Controller
etcd
API
Server
Scheduler
Scheduler
User!

14. Services
Files
1
2
Config
App
How do we automate this?
Mix of Service and Configuration
1. Run a Root CA Service
2. Create a unique Root
3. Generate Key Pair Certificate for
Server
4. Generate Digital Signature with
Public Key for Client(s)
5. Configure Server with Certificate
6. Configure Client with Signature
Private
Key
Digital
Sig
Public
Site
CA
Root
Server
Certificate
Client
Certificate
3 4
5 6
Public

15. Old Root
New Root
Root Rotation protects Trust Zone - Do it daily?!
By design, root rotation
breaks cluster
communications!
Like an in-place upgrade,
rotation can break APIs.
We need to change the
keys without breaking
communication between
components.
Previous
Cluster
Member
Cluster Trust Zone
Old New New New
Old
New

16. Step 1: Root Rotation without Downtime
Relies on Client to using
support multiple digital
signatures for the server.
Create a new root and
propagate new certificates
in the cluster.
Update the client
configurations to use
either signature.
Old Root
New Root
Previous
Cluster
Member
Cluster Trust Zone
Old
Old
New
Old Old
Old
New New

17. Step 2: Root Rotation without Downtime
Ensure all the desired
clients have new signature.
Replace the server private
key with the new value.
Old keys will no longer
work.
In a daily rotation, leave
both old and new
signatures in place. Old Root
New Root
Previous
Cluster
Member
Cluster Trust Zone
Old
Old
New
Old Old
New New
New

18. Happily,
this is a repeatable pattern
for underlay automation.

19. Questions?
Rob Hirschfeld
@zehicle
RackN.com
Rebar.Digital
Private
Key
Site
CA Root Public