The Heartbleed vulnerability was an information disclosure bug in OpenSSL unveiled to the world in April 2014. This talk will describe the impact of this bug on the Internet and CloudFlare's part in contributing to the research and education of the public about this issue.

1. Heartbleed
Nick Sullivan (@grittygrease)
Friday, August 8, 2014

2. Overview
• What is Heartbleed?
• The Heartbleed Challenge
• How certificate revocation is broken
and endangered our network
2

3. Application Layer - CloudFlare
• DNS (TCP & UDP port 53)
• HTTP (TCP port 80)
• HTTPS (TCP port 443) - powered by OpenSSL
3

4. Heartbleed
• A bug so bad it has its own website
and branding
4

5. What is it?
• A bug in OpenSSL 1.0.1
!
• Changelog
• Major changes between OpenSSL 1.0.0h and OpenSSL 1.0.1 [14 Mar 2012]:
• TLS/DTLS heartbeat support.
5

6. What is a heartbeat?
• Heartbeat: a keepalive extension to TLS
!
• Client sends (length, challenge)
• Server sends (length, challenge)
6

7. What was the bug?
• Classic buffer over-read
• Attacker sends length value
that is too long
• Server returns attacker supplied
amount of memory (up to 64kB)
7

8. 8

9. Why was it so dangerous?
• One request gets attacker server data
• Typically not logged — doesn’t leave a trace
• Valuable information
• Random decrypted data
• Login session cookies
• SSL/TLS private keys (???)
9

10. Who was vulnerable?
10

11. Who was vulnerable?
• Any server running OpenSSL
• Apache and nginx use OpenSSL by default: 65% of all active sites
!
!
• 0.8% of the top 200,000 still vulnerable (May 2, 2014)
11

12. Who was impacted?
• Almost everybody
12

13. Who discovered it?
• Neel Mehta at Google
• Codenomicon
!
• Sometime in March 2014 or earlier
13

14. Disclosure — keeping it secret
• CloudFlare, Google, Akamai, Facebook, others were notified early
• Why: large web-facing networks with the largest impact(?)
!
• Encrypted communication
• Source code visibility restricted to need-to-know
• Secure software upgrade
14

15. Big Questions
• Were private keys at risk?
• Do I have to revoke all my certificates?
15

16. The CloudFlare Heartbleed challenge
• Can someone really steal private keys from nginx?
• Code said probably not
• Temporary variables cleaned up
• Private key allocated earlier
!
• We set up a challenge on https://www.cloudflarechallenge.com/heartbleed
16

17. The CloudFlare Heartbleed challenge
!
!
!
• Results: solved in under 10 hours
• Private keys are vulnerable
• Server had 200Mbps of “mystery” outbound traffic
17

18. 18

19. RSA
• Two prime numbers P & Q
• Public key, including P x Q
• Finding P or Q can get you the private key
19

20. How it was solved
• Take every 128byte block
• Attempt to divide into public RSA key
!
• Coppersmith’s attack (only requires partial prime factor)
20

21. How it was solved
• Why was the private key on the heap?
• There was a second bug in OpenSSL
!
• The prime factor was used in the computation in a temporary variable
• Temporary variables were not cleaned during a resize
21

22. Challenge aftermath
• Undeniable key compromise potential
• All certificates need to be revoked and re-keyed
!
• CloudFlare revoked over 100,000 certificates
22

23. How revocation works
• Certificate Revocation Lists (CRLs)
• Online Certificate Status Protocol (OCSP)
• CRLSets (Google Chrome proprietary)
23

24. Revoking 100,000 SSL certificates in 24 hours
24

25. Revoking 100,000 SSL certificates in 24 hours
25

26. Revoking 100,000 SSL certificates in 24 hours
• CRL for GlobalSign grew from 22KB to 4.7MB
• CloudFlare provides caching for these CRLs
• We started seeing 30Gbps extra baseline traffic
• Repeated waves of 100Gbps every three hours (24 hours below)
26

27. Revoking 100,000 SSL certificates in 24 hours
• The issue: CRL was being downloaded by web browsers
• New CRL was being published on a 1-3 hour basis
• Internet Explorer 7/8 downloads CRLs, 9/10 OCSP with CRL fallback
• OS X: OCSP with CRL fallback
• No delta updates, we downloaded the whole thing
27

28. Revoking 100,000 SSL certificates in 24 hours
• Intra-machine links were being congested
• Had to modify cache strategy
• Moved from one cache box per rack to caching on all boxes
• Update cache headers to increase browser cache time
• Asked CA to give CloudFlare their own intermediate certificate
28

29. Revocation is broken
• None of 100,000+ certificates were in Chrome’s CRLSets
• CRL growth can’t scale
• Too many cases when OCSP hard fail
29

30. Revocation is broken
• Most efficient revocation code ever:
30

31. Revocation solutions?
• Shorter certificate expiration periods
• CRL lists not necessary after expiration
• OCSP Must-staple
• Server performs OCSP check and sends to client when connecting
31

32. Conclusion
• Bug in pervasive server software
• Huge unexpected impact on Internet security
• Crowdsourcing works
• Revocation shown to be problematic
32

33. Heartbleed
Nick Sullivan (@grittygrease)
Friday, August 8, 2014