Just like drinking is not a game in Finland; neither is browser bug hunting - it’s serious business! Browser bugs have been supporting Atte Kettunen (@attekett) traditional Finnish way of living since late 2011 and he’s going to tell you all about how he has been living the dream browser bug hunting - focusing on one of the most secure browser around, Google Chrome!
He’ll tell you a tale of his experiences with bounty programs and how those have evolved since he started way back (vendors can show the love too!) and how he’s managed to survive in the harsh environment of browser bug hunting. He’ll impart some important bug hunting social skills by showing you how and how NOT to step on the others guys toes - very competitive cottage industry is browser bug hunting. ;)
Atte is also going to share with you how and why he selected his current target feature *(still full of bugs!), how he built his fuzzer-module(s) and the results produced. We’ll all walk a mile in a bug hunters shoes together and take a peek at the tool sets, as well as the infrastructures that are used to find browser bugs by individuals and vendors!
3. Atte Kettunen
Started at OUSPG in summer 2011
First security bug from Chrome 2011-12
Since then
~100 Vulns
~60 Rewards
39 CVEs
4. Browser Bug Bounty Programs
Mozilla since 2004
- Sec-High/Critical $3,000
Google since 2010
- Typical security bugs $1,000-$3,133.7
- Possibility for bonus rewards
● PoC, exploit, awesomeness
(Microsoft 2013 June 25 - July 25)
5. Browser Bug Bounty Programs
Easy to get started - Lots of bugs o/
Helpful vendor security teams and supportive
responses to first bug submissions
Supportive (secretive/competitive)
community of other bounty hunters
6. Where the bugs are
● Use-after-free
○ DOM
○ CSS
○ Rendering
● Buffer-overflow
○ Media formats
○ Parsers
○ Decoders
○ Coordinates
9. Some bug - Regression - Chrome
==3213== ERROR: AddressSanitizer heap-buffer-overflow
on address 0x7f50cd6ffcf8 at pc 0x7f50dd159dde bp
0x7fff3e0accd0 sp 0x7fff3e0accc8
READ of size 2 at 0x7f50cd6ffcf8 thread T0
#0 0x7f.de in WebCore::CSSParser::lex(void*) ???:0
#1 0x7f.78 in cssyyparse(void*) ???:0
#2 0x7f.40 in WebCore::CSSParser::parseDeclaration()
.
Repro-file:
<a style=top:-1px>
13. Hunting for living
Three golden rules:
1. Stay green - Features
2. Stay green - Competition
3. Stay green
14. Hunting for living
Three golden rules:
1. Stay green - Features
2. Stay green - Competition
3. Stay green - Tools
15. Hunting for living
1. Stay green - Features
● New features are published all the time
○ New code o/
● Some changes are not highlighted
○ Minor updates to JavaScript API support etc.
● Old bugs fixed
○ New code o/
● Old features can change
○ Prefixes disappear(-webkit,-moz),
○ Features can get disabled
16. Hunting for living
1. Stay Green - Features
● Firefox Aurora - Release note: "Partial support for
Web Audio, targeted at web developers for testing"
(May 17, 2013)
17. Hunting for living
2. Stay green - Competition
● Tools
○ Different approach -> Different bugs?
● Targets
○ Find new minefields
● Platforms
○ Different code on different platforms
18. Hunting for living
2. Stay green - Competition
@cevans: "@j00ru has melted polar ice with
his PDF fuzzing on 9k cores."
19. Hunting for living
3. Stay green - Tools
● Instrumentations
○ New instrumentation -> detect new issues
● Build environments
○ Broken builds @#!¤#...
● Fuzzers
○ New techniques
20. Hunting for living
3. Stay green - Tools
<Q>: WTF??? On Chromium startup:
==25254== ERROR: AddressSanitizer: global-buffer-overflow on address
0x000011d3dde5 at pc 0x5ab21a bp 0x7fff00659450 sp 0x7fff00659428
READ of size 10 at 0x000011d3dde5 thread T0
#0 0x5ab219 in __interceptor_memcmp _asan_rtl_
#1 0xa1edc08 in fillInUnixFile .../sqlite/amalgamation/sqlite3.c:28654
#2 0xa1efe7c in unixOpen .../sqlite/amalgamation/sqlite3.c:29294
<A>:
Diff of /trunk/tools/build/scripts/slave/runtest.py:
+ # Avoid aggressive memcmp checks until http://crbug.com/178677 is fixed.
+ os.environ['ASAN_OPTIONS'] = 'strict_memcmp=0'
22. AddressSanitizer
● Clang compiler plugin
● Adds instrumentation to check memory
access at runtime
● Similar to Valgrind
● Only 2x slowdown
● Created at Google
● Used by Google & Mozilla
● Linux & OS X
●
http://www.chromium.org/developers/testing/addresssanitizer
23. AddressSanitizer
● Awesome with use-after-frees
● Very good for buffer-overflows and out of
bounds access
● Good but confused with type confusions
24. AddressSanitizer
==6==ERROR: AddressSanitizer: heap-use-after-free on address
0x6070000268d0 at pc 0x7f845771029f bp 0x7fff...2a0 sp 0x7fffc7eea298
READ of size 8 at 0x6070000268d0 thread T0 (chrome)
#0 0x7f845771029e (... /asan-linux-release-209136/chrome+0x96f229e)
#1 0x7f84576aacea (... /asan-linux-release-209136/chrome+0x968ccea)
#2 0x7f8451ce00f3 (... /asan-linux-release-209136/chrome+0x3cc20f3)
.
0x6070000268d0 is located 64 bytes inside of 72-byte region
[0x607000026890,0x6070000268d8)
freed by thread T19 (AudioOutputDevi) here:
#0 0x7f844f58e101 (... /asan-linux-release-209136/chrome+0x1570101)
#1 0x7f845887b5ec (... /asan-linux-release-209136/chrome+0xa85d5ec)
.
25. AddressSanitizer
==6==ERROR: AddressSanitizer: heap-use-after-free on address
0x6070000268d0 at pc 0x7f845771029f bp 0x7fff...2a0 sp 0x7fffc7eea298
READ of size 8 at 0x6070000268d0 thread T0 (chrome)
#0 0x7f845771029e in WebCore::WaveShaperDSPKernel::
lazyInitializeOversampling(...) .../WebKit/Source/wtf/OwnPtr.h:138
#1 0x7.a in WebCore::WaveShaperProcessor::setOversample(...) ...
/WebKit/Source/modules/webaudio/WaveShaperProcessor.cpp:70
.
0x6070000268d0 is located 64 bytes inside of 72-byte region
[0x607000026890,0x6070000268d8)
freed by thread T19 (AudioOutputDevi) here:
#0 0x7.1 in operator delete(void*) _asan_rtl_
#1 0x7.c in WebCore::AudioDSPKernelProcessor::uninitialize()
src/third_party/WebKit/Source/wtf/OwnPtrCommon.h:47
.
26. SyzyASan
●
Used to instrument binaries
●
Redirects heap-related calls to own runtime library
●
Currently only heap-instrumentation
●
Chrome/Chromium only atm.
●
About 3x Slowdown
●
Windows only
●
https://code.google.com/p/sawbuck/wiki/SyzyASanDesignDocument
27. SyzyASan
SyzyASAN error: heap-buffer-overflow on address
0x0379D1A7 (stack_id=0x44CB69D7)
READ of size 8 at 0x0379D000
#0 0x000068ef23be in (unknown)
#1 0x000068f387f4 in (unknown)
#2 0x000068eeb486 in (unknown)
#3 0x000068e8add7 in (unknown)
.
.
.
30. Page-Heap
● Heap allocation monitoring for Windows
● No feedback - Only crash :(
● “Works” on Chrome/Chromium
● env: CHROME_ALLOCATOR="winheap"
● Enable Chrome error reporting ->
minidumps
● Firewall Chrome( No free 0-days for Google ;) )
● Debugging tools x86
32. Fuzzers
● Dumb fuzzing
○ Yes, still works
○ Yes, you can still find bugs with bit-flipping of
image-files
● Smart fuzzing
○ Finds bugs fast but runs out of bugs faster. :(
34. Fuzzers
Smart fuzzing
● W3C/MDN(/MSDN)
● Again stay green
● Most of the JavaScript APIs in
browsers are really similar
● Some of the public tools have the logic
in them already
● W3C spec + grep + sed = $$$
42. Hardware/Infrastructure
ClusterFuzz aka. CF
● Google fuzzing cluster
● 2012 ○ 6000 Chrome instances
○ 50m+ test cases per day
○ Plans for quadrupling at that time
● ASAN, multiple fuzzers, minimization,
regression ranges, verify fixes, dupes &
dupes & dupes...
43. ClusterFuzz
“cluster-fuzz is a soulless bug hunting machine.
It has no want or need for your gratitude. It
lives only to feed on bugs.”
44. My stuff
● 12 machines running 24/7
● ~50 cores, ~133.7GB of RAM
● approx. 20m test cases per day
● 19 file-formats
● git, scp, auto-update, auto-minimize
● Radamsa and ...
45. NodeFuzz
●
●
●
●
Browser fuzzer harness
Written in JavaScript ( Node.js )
Linux, Windows, OS X
Test case generators and instrumentations
loaded as modules
● Uses WebSockets for test case injection to
browser
● Stable - https://code.google.com/p/ouspg/downloads/list
● Trunkish - https://github.com/attekett/NodeFuzz
47. NodeFuzz - module - WebAudio
● Fairly new JS API (Chrome 2011, FF
2013)
● "The API has been designed to allow modular
routing.(UAF) Basic audio operations are performed
by audio nodes that are linked together to form an
audio routing graphs.(UAF/BOF) Inside a same
context, several sources are supported, with different
kind of channel layout.(UAF/BOF) This modular
design allows for great flexibility and for the creation
of complex audio functions and of dynamic effects.
(BOF)" - MDN