ContainerDays NYC 2016: "The Secure Introduction Problem: Getting Secrets Into Containers" (Jeff Mitchell)

Think Like a Vault Developer:
Secure Introduction & Containers
Jeff Mitchell
HashiCorp
@jefferai

● Understand basic tenets of security, as the Vault team
thinks about them
● Explore properties necessary for secret management and
(initial) secret distribution
Goals

● Understand basic tenets of security, as the Vault team
thinks about them
● Explore properties necessary for secret management and
(initial) secret distribution
Goals
BORING

● Applied security in action: secure introduction, response
wrapping and orchestrators
○ Not a Nomad/Vault talk but will use these as examples
Goals

Non-Goals
● Discussion of crypto theory
○ Crypto properties (algorithm selection, etc.) are important for
applied security
○ Can stay high-level
● Comparison of container/hypervisor/OS security
○ Let’s talk Linux/Docker

Non-Goals
● Deep dive into security rabbit holes
○ Let’s keep paranoia in check

“Security”
● Security is the practice of risk management
○ Deciding which risks can be accepted and which cannot
○ Guarding against violations of norms
● Risk increases with system complexity
○ More points of failure, confusion, ingress and egress

“Threat”
● Anything that can elevate risk is a threat
● Modeling threats informs security policy

“Secret”
● A secret is something that will elevate your risk if exposed
to unauthorized entities
● Undesired consequences are harm
○ Unauthorized data access
○ Identity spoofing
○ Private data egress
○ Regulatory fines

Exposure
● An exposed secret
○ Elevates risk
○ May cause harm
● An exposed secret is a threat

Secrets vs. Identifiers
● Some things that can be disclosed are identifiers
○ Username (identifier) vs. Password (secret)
○ TLS certificate (identifier) vs. TLS key (secret)
○ GitHub user name (identifier) vs. API key (secret)
● Identifiers aren’t completely risk-free
○ Have chosen to ignore that risk

What is “Trust”?
● A trusted entity is one that will not divulge the secrets it
has access to
● Modeling trusted entities is a companion to modeling
threats
● Two concepts

Circle of Trust
Entities we trust with any secret. For this talk:
RAM
root
Secret
Management
ToolEmployees?
CPU
Cloud?

Circle of Trust
Things not in circle of trust:
Persistent
Storage
Random
Users
General
apps/services
NSA
Random
Wi-Fi Hotspot
Your Mother’s
Notepad.txt Employees?
CPU
Cloud?
RAM
root
Secret
Management
Tool
CPU
Cloud?
Employees?

Circle of Trust
Only allowed long-term storage is in circle of trust
Persistent
Storage
Random
Users
General
apps/services
NSA
Random
Wi-Fi Hotspot
Your Mother’s
Notepad.txt Employees?
CPU
Cloud?
RAM
root
Secret
Management
Tool
CPU
Cloud?
Employees?

Chain of Trust
● The set of links (e.g. network hops) that any particular
secret travels through from source entity A to destination
entity B
● Source/destination must be in circle of trust

Links
● Any link in a chain of trust is a potential access
point/interception point
○ Accidental logging
○ Exploitation by attacker
○ Lookup by operator
○ Conspiracy of One/Compromised employee
○ Employee Post-It™

Problem Space
● We can now establish a problem space:
○ Managing secrets in a given environment means establishing trust
chains in the environment
● Links in the chains have associated risk
○ Minimize hops and minimize risk-per-link

Problem Space
● In other words: figuring out how to securely get a secret
from producer to consumer

Problem Space
● In other words: figuring out how to securely get a secret
from producer to consumer
DUH

Problem Space
● Thinking securely means you have to take the roundabout
approach
○ Understanding trust, trust levels, and trust chains along the way is
incredibly important
● Only after understanding trust can you figure out where
and how to minimize risk

Problem Space
● Risk cannot be fully mitigated; must assume any given
secret will eventually be divulged
● Ultimate goal: zero trust
○ Don’t give the opportunity for risks to occur in the first place

Scheduler
Secret Management Tool
Scheduler Agent
Container
Goal: Securely
move secret from
originator to new
container’s RAM

Scheduler
Scheduler Agent
Container
?

Security Properties /
Success Criteria

Secret Protection
● Establishing a chain of trust requires defining the
requirements it must fulfill to keep secrets protected
● Good news: we (often) only need to do this for one secret!
○ First secret authenticates us to allow direct access to more

Secure Introduction
● If you can protect this secret, you can protect any secret
○ ...generally
● This concept is secure introduction (SI)

Secure Introduction
● How do you protect secrets (perform SI)?
● First: establish success criteria based on your
organization’s acceptable risk
○ Any given organization needs a security policy and associated
success criteria that meets their individual needs

Secure Introduction
● Reasons your organization doesn’t need to establish a
security policy:

Secure Introduction
● Reasons your organization doesn’t need to establish a
security policy:
○ None

Success Criteria
● The security properties that must be implemented/met to
implement the accepted security policy
● For secure introduction for this talk:
○ Don’t let them live forever (rotate/expire)
○ Distribute them securely
○ Limit exposure if disclosed
○ Have a break-glass procedure
○ Detect unauthorized access

Rotation/Expiration
● As lifetime increases, chance for exposure → ∞
○ Caches/logs
○ Cracked over time/enough packets (WPA-TKIP, RC4)
○ Debugging

Rotation/Expiration
● Secrets should be rotated “frequently” and have lifetimes
as short as possible
● User passwords vs. machine secrets
○ xkcd’s ______ ______ ______ ______ → bad policies + frequent
rotation = written down user passwords
○ Less frequently/more used = more likely overseen
○ Machines: it’s just data - plan/build for rotation and rotate often

Distribution
● The literal movement along the chain of trust
○ To/From
■ People
■ Machines
● Base level: never plaintext, always covered
○ Encryption
○ Wrapping
○ Etc.

Limit Exposure
● Principle of least privilege
○ DB credentials: only specific tables/operations
○ Login credentials: not root
○ API credentials: minimal function set

Access Detection
● Things have a way of being leaky
● Env vars are a common way to pass in container secrets.
Also:
○ Often logged, sometimes multiple places
○ Easily discoverable by operator
■ Both Docker commands and non-Docker commands can spill the
beans

Access Detection
● Equally as important as protecting a secret is knowing if
an unintended party has intercepted it
● Audit logs are great…
○ ...but do you look at them?
● Active detection (when possible) is even better

Break-Glass
● Compromised?
○ Stop all further access to protected resources
○ Perform forensics
○ Rotate all secrets after re-establishing identities of legitimate
secret-holders
○ …
● Figure this out during the planning process -- not after!

● You do not have to face this alone

● Explosion of FOSS Secret Management (SM) tools
○ KeyWhiz
○ Knox
○ Conjur
○ Many more
Secret Management Tools

● Explosion of FOSS Secret Management (SM) tools
○ KeyWhiz
○ Knox
○ Conjur
○ Many more
○ Vault!

● If you only take away two things from this talk, make sure
they are the following:

● Write your own crypto

● Use it in a Secret Management tool

● Use it in a Secret Management tool
NO!
NO!

● Use a Secret Management tool
● Don’t roll your own

● Discussion of SM tools true for Vault and its capabilities
● Vault has a focus on providing security primitives
○ Enables creation of complex workflows
○ We’ll see how this provides necessary flexibility

● Central:
○ Secure storage (avoid sprawl)
○ Management
○ Auditing (who has seen/accessed what, when)
● Codified, secure access mechanisms
● Secret rotation/revocation/expiration

SM + SI
● Explicit focus on the secure introduction problem
○ Core competency from ground up
○ Necessary tools/capabilities/primitives
● 100k containers can support existing SI paradigms
○ But imagine…
■ Managing 100k LDAP/AD users churning at high rate
■ Generating/dropping 100k Kerberos keytabs
○ ...and you’ve solved SI anyways

SMs <3 Schedulers
● Anyone working with tasks (containers) at scale uses a
scheduler
○ Nomad
○ Mesos
○ Fleet
○ Swarm
○ Kubernetes

SMs <3 Schedulers
● Schedulers are sources of truth...and provide hooks
● SM tools and schedulers can be a magical combination

Scheduler
Scheduler Agent
Container
Vault

Scheduler
Scheduler Agent
Container
SI #1 (Traditional)
Vault

Scheduler
Vault
Scheduler Agent
Container
SI #1 (Traditional):
AWS-EC2 with
Vault Secure Intro Client
Uses: ∞
TTL: 8h (renewable)
Policy: create_app_tokens

Scheduler
Scheduler Agent
Container
SI #2 (New)
Uses: ∞
TTL: 8h (renewable)
Vault

Preconditions
● A Vault token has:
○ Unlimited or limited use-count
○ Limited time-to-live (TTL), possibly renewable
○ Set of authorization policies
■ e.g. use first secret (auth token) to get application secrets
○ Consistent ID in audit logs
○ Token-scoped secure storage tied to token lifetime
■ ‘cubbyhole’
● Claim: These primitives allow excellent secret coverage

Scheduler
Scheduler Agent
Uses: ∞
TTL: 8h (renewable)
Uses: ∞
TTL: 8h (renewable)
Container
Vault

Scheduler
App: db_writer
Security Policy: app_db_rw_pol
Scheduler Agent
Uses: ∞
TTL: 8h (renewable)
Uses: ∞
TTL: 8h (renewable)
Container
Vault

Scheduler
App: db_writer
Create Token:
Policy: app_db_rw_pol
Scheduler Agent
Uses: ∞
TTL: 8h (renewable)
Uses: ∞
TTL: 8h (renewable)
Container
Vault

Scheduler
Create Token:
Uses: 1
TTL: 30s (non-renewable)
Policy: read_cubby
Private storage:
Uses: ∞
TTL: 1h (renewable)
Scheduler Agent
Uses: ∞
TTL: 8h (renewable)
App: db_writer
Uses: ∞
TTL: 8h (renewable)
Container
Vault

Scheduler
Scheduler Agent
Uses: ∞
TTL: 8h (renewable)
ContainerImage: db_writer
TOKEN: (outer token)
Uses: 1
Policy: read_cubby
Private storage:
Uses: ∞
TTL: 1h (renewable)
Uses: ∞
TTL: 8h (renewable)
Vault

Scheduler
Scheduler Agent
Uses: ∞
TTL: 8h (renewable)
Container
Uses: 1
Policy: read_cubby
Private storage:
Uses: ∞
TTL: 1h (renewable)
Uses: ∞
TTL: 8h (renewable)
Unwrap
Vault

Scheduler
Scheduler Agent
Uses: ∞
TTL: 8h (renewable)
Uses: ∞
TTL: 8h (renewable)
Container
Uses: ∞
TTL: 1h (renewable)
Vault

Scheduler
Scheduler Agent
Uses: ∞
TTL: 8h (renewable)
Container
Uses: ∞
TTL: 1h (renewable)
Uses: ∞
TTL: 8h (renewable)
Get DB
Creds
Vault

Scheduler
Scheduler Agent
Uses: ∞
TTL: 8h (renewable)
Uses: ∞
TTL: 8h (renewable)
User: db_user
Password: db_pass
Expiration: 24h
Container
Vault

Scheduler
Scheduler Agent
Uses: ∞
TTL: 8h (renewable)
Container
Uses: ∞
TTL: 1h (renewable)
Uses: ∞
TTL: 8h (renewable)
Get S3
Creds
Vault

Scheduler
Scheduler Agent
Uses: ∞
TTL: 8h (renewable)
Uses: ∞
TTL: 8h (renewable)
ACCESS_KEY: ...
SECRET_KEY: ...
Expiration: 24h
Container
Vault

Scheduler
Scheduler Agent
Uses: ∞
TTL: 8h (renewable)
Container
Uses: ∞
TTL: 1h (renewable)
Uses: ∞
TTL: 8h (renewable)
Get TLS
Cert
Vault

Scheduler
Scheduler Agent
Uses: ∞
TTL: 8h (renewable)
Uses: ∞
TTL: 8h (renewable)
Certificate: ...
Private Key: ...
Issuing CA: ...
Container
Vault

How’d we do?
● Look at previous list:
○ Don’t let them live forever (rotate/expire)
✓ Outer token expires (use limit) with only copy of inner token value
○ Distribute them securely
✓ Inner token covered the entire way
○ Limit exposure if disclosed
✓ Only specific policies granted, can only access specific secrets
○ Have a break-glass procedure
✓ Can lock down access at SM tool
✓ Audit logs ensure we know area of exposure
○ Detect unauthorized access

Access Detection
● Can detect unauthorized access of “real” authentication
token due to time and use limit
● Application:
○ Reads inner token: success
○ Reads storage but no inner token found: log error and recover
(e.g. fail job)
○ Invalid token? Check current time vs. issue time + TTL:
■ Expired? YELLOW FLAG: investigate, but probably just bad timing
■ Not expired? RED FLAG: Unauthorized use

Response Wrapping
● This paradigm is so useful that it’s available for nearly any
Vault call and known as response wrapping
● Mechanism not container- or authentication-specific
○ CM tool drops wrapped secret on EC2 instance
○ File with wrapped value injected into chroot
○ Etc.

Job-Finish Revocation
● Wrapped responses containing tokens also return the
token’s accessor (reference)
● Scheduling agent/system can correlate accessor with job
○ Revoke token / child leases immediately on job finish

Integration
● Nomad integration uses a variation of this scheme but
basic concept is the same
○ User-supplied token at job submission time checked for access to
avoid privilege escalation via job submission
○ Agent can unwrap secrets and mount into ramdisk
○ Etc.

Integration
● HashiStack tools follow the Unix model; we set out to
tackle this problem in a generic way
● Any scheduler can implement Nomad’s approach (or a
variation)
○ Primitives to do so are there in Vault
○ Kubernetes (https://github.com/kelseyhightower/vault-controller)
○ Mesos
(https://github.com/ChannelMeter/vault-gatekeeper-mesos)

Wrap-Up
● Don’t ignore risk, minimize it
○ Understand your circles and chains of trust
● Use Vault, or some other SM tool
● Orchestration tools are great facilitators of secure
introduction

ContainerDays NYC 2016: "The Secure Introduction Problem: Getting Secrets Into Containers" (Jeff Mitchell)

Empfohlen

Empfohlen

Weitere ähnliche Inhalte

Was ist angesagt?

Was ist angesagt? (20)

Andere mochten auch

Andere mochten auch (20)

Ähnlich wie ContainerDays NYC 2016: "The Secure Introduction Problem: Getting Secrets Into Containers" (Jeff Mitchell)

Ähnlich wie ContainerDays NYC 2016: "The Secure Introduction Problem: Getting Secrets Into Containers" (Jeff Mitchell) (20)

Mehr von DynamicInfraDays

Mehr von DynamicInfraDays (16)

Kürzlich hochgeladen

Kürzlich hochgeladen (20)

ContainerDays NYC 2016: "The Secure Introduction Problem: Getting Secrets Into Containers" (Jeff Mitchell)