Bypass Surgery - Akamai's Heartbleed Response Case Study by Brian Sniffen, Chief Security Architect, Akamai Technologies In April of this year, the inevitable happened and Akamai's network was determined to be vulnerable to the Heartbleed bug. The practice of information security is both about preventing vulnerabilities and mitigating vulnerabilities when they're found. In this case study, Akamai Chief Security Architect Brian Sniffen will walk through Akamai's response to the Heartbleed vulnerability and provide insights into the lessons to be learned for improved security processes and incident response. Akamai Edge is the premier event for Internet innovators, tech professionals and online business pioneers who together are forging a Faster Forward World. At Edge, the architects, experts and implementers of the most innovative global online businesses gather face-to-face for an invaluable three days of sharing, learning and together pushing the limits of the Faster Forward World. Learn more at: http://www.akamai.com/edge

2. Bypass Surgery and Other Tales
Brian Sniffen

3. ©2014 AKAMAI | FASTER FORWARDTM
Akamai Security Research & Architecture
• Crypto engineering expertise
• Technical backstop
• Product review
• Akamai Architecture Group seat
• Safety engineering
• Incident management

4. ©2014 AKAMAI | FASTER FORWARDTM
2014: The Year of Vulnerabilities
“You people in InfoSec have become the Product Managers!”
Yes, and we can’t wait to get out of that role.
• Heartbleed
• INRIA-Prosecco Cookies
• Shellshock
2009 2010 2011 2012 2013 2014
1 0 0 1 1 3+
2008
1

5. ©2014 AKAMAI | FASTER FORWARDTM
Akamai Incident Management Principles
• Technical Incident Manager (TIM) coordinates all work
• Incident Executive communicates upwards, shields TIM
• GSS Business Incident Leads manage customer comms
• No single-point harm can cause a Severity 1 Incident
• A hot meal and 6 hours sleep fix more problems than an all-nighter
• If the TIM becomes an SME, get a new TIM

6. ©2014 AKAMAI | FASTER FORWARDTM
We tell ourselves who we wish to be:
• Akamai says thank you.
• Akamai doesn't respond to name calling,
but does respond to the useful technical content.
• Akamai presents itself as a responsible and respected member
of the Internet community.
• Akamai will use this incident to improve both its own security
and the general security of the Internet.
• Akamai can laugh at itself.

7. ©2014 AKAMAI | FASTER FORWARDTM
Heartbleed mail
From: Brian Sniffen
Date: 7 Apr 2014 21:34:08
Subject: Sev 1: Heartbleed
Will,
I'm contacting you because you're the Ghost SME on call. I'm
looking for evidence to refute the statement: "The Heartbleed
bug can’t extract long-term customer private keys from a Ghost;
we put them only in a wired, mmaped page way lower on the
stack."

8. ©2014 AKAMAI | FASTER FORWARDTM
Heartbleed Timeline
April 1: Notice; QA begins
April 4: last Akamai Deployed Systems patched
April 7, 1pm: Public Notice
April 7, 6pm: What did we leak?
April 8, 1am: Working exploit in Akamai lab
April 9–12: Hastily publish Akamai Secure Allocator
April 13, 11pm: Begin cert rotations & revocations

9. ©2014 AKAMAI | FASTER FORWARDTM
“Don’t worry, we restored the old functionality”
April 14, 6am:
“Why is this message in the
old log format?”
A “Manual Change”
had restored an old version.

10. ©2014 AKAMAI | FASTER FORWARDTM
The Akamai Secure Memory Allocator
• 1999 code
• One author, three redactors
• State machine inspired by CLOS “advice” system
Turns out it works fine
Code
Secure
Heap
mmap’d file
Long-term
Allocations
Heap

11. 70% 90% 95% # of certs
©2014 AKAMAI | FASTER FORWARDTM
Cert Revocation Progress
21
Apr
28
Apr
5
May
12
May
19
May
26
May
2
Jun
9
Jun
16
Jun
23
Jun
30
Jun

12. ©2014 AKAMAI | FASTER FORWARDTM
Learning from Heartbleed
Nobody’s paying for OpenSSL!
Practice in mass, fast, patching
Practice in releasing helpful patches
Simplicity promotes safety.

13. ©2014 AKAMAI | FASTER FORWARDTM
Shellshock Timeline
Sep 23, 12pm: Notice from Florian Weimer, Debian Security
Sep 23, 9pm: Manual change: replace bash wish dash;
Patches started
Sep 24, 5am: WAF rule in place
SSH command=
systems made safe
Sep 24, 12pm: Public Notice
Sep 25: “Kobrin Patch” to remove dangerous feature
Sep 28: bash mostly replaced with dash on deployed network

14. ©2014 AKAMAI | FASTER FORWARDTM
Bash patches
Pre-release:
• Embargoed patch: 195 lines, 7 files (1/6 CVEs)
• Kobrin patch: 2 lines, 1 file (6/6 CVEs)
Post-release:
• NetBSD patch: 3 lines, 2 files (6/6 CVEs)
• Fixed patch: 164 lines, 11 files (6/6 CVEs)
• Apple patch: unpublished (exposure unclear)

15. ©2014 AKAMAI | FASTER FORWARDTM
SSH command= limits
ssh-ed25519
AAAAC3NzaC1lZDI1NTE5AAAAIFnHfYyS5onAN72oFpaopm+/yKbRy/
TCwpt7Tmw3lk0P bts@tereva.evenmere.org-2014-03-19
command="/a/bin/akamai_run suspend" ssh-ed25519
AAAAB3NzaC1yc2EAAAADAQABAAABAQDKVmNk8leXjKkWZUHQjJIT
zrX+n1aa1xfBwK9Yp42q bsniffen@akamai.com-example-2014-10-01
V=“() { :;}; /bin/bash” ssh example.com :

16. ©2014 AKAMAI | FASTER FORWARDTM
Akamai Shellshock exposures
Client ssh Authgate Server 1
Server 2
Perforce Server 3
https ssh
Web Kerberos
CGI

17. ©2014 AKAMAI | FASTER FORWARDTM
Solaris 10
“We don’t have any Solaris 10 admins”

18. ©2014 AKAMAI | FASTER FORWARDTM
Who’s looking?
13000 IPs probing per day

19. ©2014 AKAMAI | FASTER FORWARDTM
Learnings from Shellshock
Nobody’s paying for Bash.
And it was written in the 1980s!
Simplicity promotes safety.

20. ©2014 AKAMAI | FASTER FORWARDTM
The New Normal
• Two or three internet-wide patching incidents per year
• Enterprise-wide compliance takes months
• Trust less code.
• Trust code less.
• Treat upstream code like you wrote it?
• Homework: set up 24/7 contacts and Security contacts