27. What do we need to
achieve Least
Privilege
Orchestration?
28. Mitigating External Attacker
- Externally accessible service ports
are explicitly defined
- Administration endpoints are
authenticated and authorized
29. Mitigating Internal Network
Attacker
- Authentication of both network and
cluster control-plane communication
- Service to service communication is
authorized, with orchestrator managed
ACLs
31. Mitigating Malicious Worker
‣Should only have access to
resources currently in use
‣Push VS Pull
‣No ability to modify or access any
cluster state except their own.
‣Identity is assigned, never
requested
32. Mitigating Malicious Manager
‣Can’t run arbitrary code on workers
‣No access to secret material
‣No ability to spin up unauthorized
nodes/impersonate existing nodes.
‣No ability to read service-to-service
communication
36. Mutual TLS by default
• First node generates a
new self-signed CA.
37. Mutual TLS by default
• First node generates a
new self-signed CA.
• New nodes can get a
certificate issued w/ a
token.
38. Mutual TLS by default
• First node generates a
new self-signed CA.
• New nodes can get a
certificate issued w/ a
token.
• Workers and managers
identified by their
certificate.
39. Mutual TLS by default
• First node generates a new
self-signed CA.
• New nodes can get a
certificate issued w/ a token.
• Workers and managers
identified by their certificate.
• Communications secured
with Mutual TLS.
Group of musicians, with different roles, that play a specific pre-arranged tune in synchronicity
One interesting thing about Classical arts such as Plays, Ballets, and even Symphonies is that they all follow a script
A writer, or composer writes a script that is performed by the creative people under the direction of a director or conductor
Even though Directors and conductors have some latitude to arrange things according to their liking, but they must completely follow certain wishes of the original author
In classical music, the notes performed must be the exact notes that the composer wrote
In plays, the exact wording of the script must be followed even though casting and certain details about the set can change
In ballets, the exact movements of the dancers must be followed.
In each of the examples of the examples that we’ve mentioned: symphonies, ballet and plays, there is a declarative syntax that the authors can represent their intentions in.
These intentions must be strictly followed this is BALLET NOTATION
Composer (Bach), Conductor and a musician
Now that we know what is an orchestrator? What is a least privilege orchestrator then”
Least privilege is the concept of specialization.
For example, if you have a Tuba player, she has to follow strictly the sheet music, and can’t, for example, start playing the drums. It shouldn’t have access to either the drums or the drums sheet music.
- And therefore a least-privilege orchestrator is a an orchestrator that follows this principle in the strictest manner possible.
It comes down what is your attacker model. And we believe we should architect for the most pessimistic attacker model. Following least privilege allows us to think critically about exactly what access each participant in a system should have.
These are the five broad categories of attacker models that we can think about when looking at orchestrator systems
- A common attacker model, is someone completely outside of your system. An example of this attacker model are external firewalls: their exclusive attacker model is an external attacker, and they are completely bypassed by someone with internal network access
- A common attacker model, is someone completely outside of your system. An example of this attacker model are external firewalls: their exclusive attacker model is an external attacker, and they are completely bypassed by someone with internal network access
- A common attacker model, is someone completely outside of your system. An example of this attacker model are external firewalls: their exclusive attacker model is an external attacker, and they are completely bypassed by someone with internal network access
- NSA cat
- A common attacker model, is someone completely outside of your system. An example of this attacker model are external firewalls: their exclusive attacker model is an external attacker, and they are completely bypassed by someone with internal network access
- Malicious Musician AKA malicious worker
- A common attacker model, is someone completely outside of your system. An example of this attacker model are external firewalls: their exclusive attacker model is an external attacker, and they are completely bypassed by someone with internal network access
- Malicious Conductor aka Malicious Manager
- Malicious Conductor aka Malicious Manager
And the fact that we should protect against all of these attacker models, is why we need least-privilege architectures. If any node is compromised, an attacker has exactly the access that the node has at that specific moment in time, no more, no less.
Eclipse attack
What do we need?
It is super important in security design, let’s design architectures that take into account all the attacker models that we just discussed.
From this, we will come up design principles for our idealized least-privilege orchestrator
If a service did not get explicitly defined as needing an internet accessible port, it is closed by default.
Administration endpoints are authenticated, and authorized by default, which means that misconfiguration is unlikely
Secrets, Configs, Networks, etc
Once worker no longer needs resource, remove them to mitigate against future compromise
Simple and understandable through decomposition and state space reduction.
Since any node can trigger an election and terminate the current term at any moment, a Byzantine node can effortlessly starve the whole system by perpetual elections, and consequently subvert Raft’s availability.
Tangaroa and Juno
What do we need?
It is super important in security design, let’s design architectures that take into account all the attacker models that we just discussed.
From this, we will come up design principles for our idealized least-privilege orchestrator
User’s should sign all declarative changes, Workers have to enforce them
All external dependencies must also be signed, specially if they are time-of-use dependent (alpine:latest)
Talk about notary
- Bootstrap: where does the trusted user keys come from in a self-signed, automatically generated swarm. TOFU? Manual configuration on nodes? Hardware trust anchor?
Raft VS Paxos
Tangaroa + blockchain (whtaver the name is) implementations aren’t production ready.
Don’t want malicious manager to take leadership
Add diagram that explains how least-privilege secrets distribution is hard
- Secrets can’t pass managers unencrypted
- Still need the ability of recovering from node failure and rescheduling containers, but can’t add new nodes and self-schedule tasks
- Managers can’t control node identities
Open questions:
Secrets should be accessed through a third-party system or encrypted to each node’s public key?
How do you control identities of new nodes, and ensure that malicious managers aren’t intercepting the requests or creating phantom nodes.
external certificate authority
The current state of orchestrators from a security perspective: It’s great, it’s entertaining, it makes you want to put $5 in the donation bucket. But it really isn’t what you want to run in your production environment. We deserve better. We deserve orchestrators that live up to the name.
Let me tell you about what we’re doing with swarm and how we’re trying to achieve least-privilege orchestrators.
- The first node Generates a CA, and creates a 1-node
- One mode of operation: requires a token, tokens are generated by the server. Short of tweeting the token there is very little you can do to screw it up.
- Everyone has roles. In certificates
- MiTM attacker and Internal Attacker. We also encrypt gossip membership, and have optional data-plane encryption
Customizable certificate rotation periods.
Occurs automatically.
Ensures potentially compromised or leaked certificates are rotated out of use.
Whitelist of currently valid certificates.
- This is the current state of swarm: they are a jazz band. Swarm currently has least-privilege musicians, but we need least-privilege conductors.
And what we want is a classical music orchestra
And we would love your help getting there.