SlideShare ist ein Scribd-Unternehmen logo
1 von 63
Downloaden Sie, um offline zu lesen
Showing How Security Has (And Hasn't) Improved, After Ten Years Of Trying Dan Kaminsky, Chief Scientist, DKH Inc. Michael Eddington, Partner, Déjà vu Security Adam Cecchitti, Partner, Déjà vu Security
1990’s:  Security == Cryptography and Java 2000’s:  Bad guys show up, start owning everything Got particularly bad in 2003 We learn how software actually fails 2010’s:  The decade of defense OK.  We’ve gotten software to fail. How do we make software not fail? Have we gotten any better? History (As I See It)
Major efforts going on in the corporate/defensive world to try to get some sense of Return On Investment / “What Actually Works” in defense Everybody’s.  Getting.  Owned. Something’s not working. Difference between attack and defense Attack doesn’t work, it’s easy to tell. Defense doesn’t work, it’s…not so easy. What sort of metrics would be helpful? The Push For Metrics
“How hard is it for an attacker to corrupt a computer exposing this particular codebase?” Security is about systems, not about individual codebases or even just computers, but bear with me Can We Get Metrics ForSoftware Quality?
We know more things are getting compromised Is this because attackers are getting better? Or is this because code is getting worse? Hypothesis:  Software quality has improved over the last ten years. Corrolary:  Code is not getting worse; attackers are getting better A Fundamental Question
Standard dataset usually pored through for quality metrics: CVE CVE:  Common Vulnerabilities and Exposures Information on 48,425 vulnerabilities found (and presumably fixed) over the last ten years or so This is lots of data! Or is it? What Sort Of Experiments Could We Try?
Fundamental bias Conflation:  Software quality with finder interest Interest not only in finding the bugs, but reporting the bugs Major human element (at minimum, difficult/impossible to measure as the exclusive metric) Not that much finder interest that still reports back to vendors We have a latency problem The Problem with CVE
Could we take the source code from a large number of projects, and look at the number of “bad things” in the source code? Sort of. First issue:  Good luck getting source code to targets Could be bypassed if companies published their static analysis rates This would actually be pretty cool, hint hint Second issue: In practice, difficult to distinguish stylistic concerns from actual vulnerabilities Much bigger problem Perhaps Static Analysis?
Fuzzing:  The use of a large number of generally corrupt and machine generated test cases to attempt to find failure modes in software. Is it possible to use the results of large scale fuzzing as a rough metric for software quality? “Fuzzmarking” Possibly.  There’s even some precedent BreakingPoint’s Resiliency Score against networks and network devices What About Fuzzing?
Target Formats Office Documents:  .doc, .xls., .ppt Portable Documents:  .pdf These are the formats with the highest complexity and the greatest exposure Also, empirically they’ve been the ones that are getting us owned the most There are two sorts of studies that can be run against these formats Lets Start With The Highest Risk Stuff
Office Documents Microsoft Word/Excel/Powerpoint OpenOffice Writer/Calc/Impress Portable Documents Adobe Acrobat Foxit Reader GhostScript …but there’s something way more interesting we can do. Cross Sectional Studies:Which Parsers Are (Maybe) Safest?
We’ve been doing security for ten years. How much has software improved in those ten years? Has it improved? If so, when? We’ve taken software circa 2000, 2003, 2007, and 2010, and run synchronized fuzz tests across all of them This is (potentially) a window into bugs that have never seen the light of a CVE So what does the data look like? Longitudinal Studies:When
OK, not really. 175,334 crashes in ~44 days. Details? Lets talk first about process, procedure, and how you’re going to screw this up if you try to do this. 188,000 Crashes In 44 Days
PROCESS
Scavenge sample files from Google Internet Archive Looking for files near the year 2000 This did not work out so well… Obtain Sample Files 1
Pre-generate all fuzzed files using Peach 100,000 files per format (doc, xls, ppt, pdf) Office: Understand OLE Storage Fuzz the streams in doc/xls/ppt PDF: Decompress streams (Eventually) Generate Fuzzed Files 2
Peach Farm Hosted fuzzing service from Déjà vu Security You provide the target, we produce the crashes For this talk… 88 Cores of fuzzing fun 5 Targets 4 File formats >250 GB of logs 3
Office 2010/2007 Thousands of Registry Keys (File recovery) Office/OpenOffice Tens of thousands/Gigs of temp files OpenOffice “Quick Start” keeps OO in memory PDF Compressed/Non-Compressed Running of the Fuzzers 4
Look correct? Was every iteration a crash? #fail No crashes? Complete set? Runs crash Machines die Logs misplaced Verify Logs 5
Collect interesting data Mine data into SQL Easy to query for data sets Mine Logs 6
How Not To Fuzz Fuzzing “How hard could opening a program and reading a file be?”  Run the same 100k files across 18programs.  Every file, every program, every time. Must open the program  Must open, read, and parse the file.  Must record the result.
Personality  Some software has very low self esteem and needs constant attention. “Look what I parsed!”   Each program ends up with it own personality based on how many deterrents there are to fuzzing. Ranging from whiner to aggressive to needy.  Occasionally 30k / 50k runs we’d have to “sweet talk” the exe back into running. Some things were predictable and others randomish.  Opening the Program
Personality traits From the program  Auto Updaters    : Version 6.2 –> 6.5   Auto Upgraders  : Version 6 -> 7 Bundle ware / Nag ware  Recovery of crashed files “Safe modes”  Importing Error reporting / Crash Analysis software From the OS  Auto Update Anti Virus  Scheduled scans / reboots.  Startup  / OnRun / Preloaders
Click here to punch clippy From the Program  Pop Ups  Registration  Surveys  Ads  Assistants
Files from the internet why not DLLs? Dynamic downloads Add Ons Fonts / international support  Foxit from 2007 doesn’t have default JPEG support Download and install at runtimes  Puts the program out of sync with the rest “Static” downloads  Old documents linked to things all over the web. Parts of the fuzzed doc / pdf are just gone.  Makes for less interesting fuzzing
Parsing the file A few “docs” we crawled were really HTML, JPEG, TIFF, PDF, JavaScript Ended up with a lot of integrity checking of file  Crawler and verifier is around 3k lines of C# code Normally we wouldn’t care a crash is a crash to us, but we wanted to make sure our file set was valid.
Magic Number check  Caused false positive  Searched for JavaScript and HTML tags Basic length checks Failed downloads / truncated uploads Hash of file Resume a downloaded crawl that failed.   Filename -> guid Many files on the internet share names. “Homework1.doc”, “Expenses.xls” Things to check for
Data integrity becomes very important Collection of crashes and mining of the data requires more meta data the more complex you want to be.  Tag the results directories with  Program name, year, date, suite, file type, version, machine, iterations, and a guid. When in doubt add another guid. No for real.   This will barely be enough to manually reconstruct the run in the event things go wrong.  Recording the results
175,334 Crashes Across 26 Codebases Word/Excel/Powerpoint 2003/2007/2010 Acrobat/GhostView 2003/2007/2010 Foxit2007/2010 One crash about every six seconds One crash about every ten iterations So What? Data Analysis(Naïve And Totally Misleading Edition)
You don’t fix the same bug over and over You also don’t fix bugs in the order they appear Severity matters How do you operationalize bug classification? You don’t fix crashes.  You fix bugs.
!exploitable:  Microsoft debugger extension that: A) “Bucketizes” the crash – allowing it to be matched with similar crashes B) Attempts to measure the severity of the crash Yes, this can be done manually, but: Manual analysis doesn’t scale Told you !exploitable was important
2536 Unique Major/Minor Hashes 1149 in Office 1125 in OpenOffice 181 in Ghostview 70 in Foxit 10 in Acrobat Unique Crashes
But that’s Major/Minor Most “minor hashes” represent the same underlying bug under a single major Depends on the product…75-95% chance of minors representing the same bug Lets be conservative -- only consider Major 942 Unique Major Hashes 440 in Office 396 in Open Office 68 in Ghostview 32 in Foxit 5 in Acrobat Slicing Things Down
942 Unique Major Hashes 150 EXPLOITABLE (15%) 188 PROBABLY_EXPLOITABLE (19%) 16 PROBABLY_NOT_EXPLOITABLE (1.6%) 588 UNKNOWN (62%) What About Severity?
This is a cross-sectional study Is severity equally distributed across the various parsers? This is a longitudinal study Is severity equally distributed across the various versions? Now, lets merge Severity with our other metrics…
Office vs. StarOffice 2003/7/10(Exploitable/Probably Exploitable)
Ghostview vs. Foxit vs. Reader 2003/7/10 (E/PE)
Fuzzmarkingis implying an across the board effect – code shipped in 2003 was objectively less secure than code shipped in 2010 At least, in parsers for the highly targeted file formats doc/xls/ppt/pdf We would be surprised if this effect showed up in parsers for file formats that do not cross security boundaries (i.e. aren’t targeted) (But aren’t using XML/JSON) Systematic Fuzzing Seems To Reflect Improving Code Quality
The fundamental tension We know that some file formats “invite trouble” Lengths Explicit (especially 32 bit) Overlapping Implicit  Jump tables Diffs (“fast saving”) .txt is going to end up being a safer format than .doc or .pdf But what about .doc vs. .pdf vs. .rtf? Can One Compare File Formats With Fuzzmarking?
Absolute comparisons across file formats suffer conflation between raw danger of the format and quality of the fuzz tests Are we getting more crashes because the format has more things to find? Or are we just better at finding what’s there? Danger!
In our case, it’s very obvious why we have more Office crashes than PDF crashes We’re relatively document aware in doc/xls/ppt We’re just flipping bits on PDF Our PDF fuzzer isn’t even decompressing yet  We were surprised to see any crashes! (And we didn’t see any E/PE in Acrobat) What about Office Format vs. Office Format? At least we know where we stand in this case
At First Glance, Does Seem Like .doc has more bugs than others Total Crashes Per Type Total Unique Majors Per Type
When you filter to E/PE, the formats get equal…
Only Simultaneously Viewing Severity/Version/Type Helps Office Doc/XLS/PPT E/PE OpenOffice Doc/XLS/PPT E/PE
What’s going on? No serious correlation between Office and OpenOffice on .doc risk What’s not going on? .doc, .xls, and .ppt can’t actually have any security differential You can reach any parser from any We’re not going to notice that without really smart template generation Analysis
588 UNKNOWN crashes… 289 TaintedDataControlsBranchSelection (49%) 95 ReadAV (16%) 74 TaintedDataPassedToFunction (13%) 58 TaintedDataReturnedFromFunction (10%) 28 PossibleStackCorruption (5%) 27 0xe06d7363 (5%) 16 StackOverflow (3%) 1 Invalid Handle (0.1%) What About The UNKNOWN cases?
Office vs. StarOffice 2003/7/10(UNKNOWN)
1) There’s still improvement, but not as sharp It’s almost like Exploitable / Probably Exploitable bugs are more likely to be fixed than Unknown bugs 2) There’s a far higher floor, even in 2010 From The Unknown Data…
Ghostview vs. Foxit vs. Reader 2003/7/10 (UNKNOWN)
We’re still seeing improvement PDF had an interesting artifact in E/PE – 2003 was better than 2007 This artifact is repeated in U – why? More features in 2007 parsers More corruption resistance in 2007 parsers Template preference for 2007/2010 parsers What About PDF?
Nate Lawson showed:  Never just average everything together.  Always look at the distributions. Chunking down from 174K crashes to 940 unique vulns throws away a lot of data It matters:  Does a given bug take 10 rounds to find?  Or 100,000? Another Possible Metric:Bug Rarity Profiles
Number of Crashes Per Unique Bug(A “Spectral Fingerprint” For Fuzzmarking) Office 2003 OpenOffice Writer 2003
How do you correlate the same bug across Word and Writer? The stack traces are different Obviously the Major/Minor hashes are different But the fuzz file is the same Naïve Numbers 110,637 unique files created crashes 65989 (59%) crashed only one target 44648 (41%) crashed more than one target What About Bugs That Hit More Than One Target?
Can we visualize this? Put the filename in the center Treat the left side as “Office Word” Treat the right side as “OpenOffice Writer” Treat distance from the center as distance from now Color by severity Sort by recentness of vuln * severity Make each pattern unique Can We Do Better With Viz?
Office Word 2003/2007/2010 v. OpenOffice 2003/2007/2010 [0]
Exploitable in Writer 2003, Unknown in Word 2003/2007/2010
Easy Dereferencing Back To Major Hashes Unknown in Word 2007, PE in Writer 2010 Major Hashes
Other Neat Profiles (About 2000 Total) “Gap Years” – Unknown in SW2003, PE in SW2010 “Welcome To The Club”
Targeting an area already being swept by defenders Code may look better than it actually is, because defenders are securing stuff exposed by these very same methods This is ultimately the issue with all benchmarks The only question is whether these bugs are representative of bugs that need to get fixed Since these are the easiest flaws for an attacker to find, maybe this is OK Potential Issues
Bitflippingonly works well against file formats that are tightly packed Most new formats are all textual Also require complex grammars This is a first attempt, we intentionally didn’t want to pack too much intelligence in Code coverage Automatic grammar extraction Automatic segmentation of file formats Manual generation of files Integration with memory tracing What about the fuzzer itself?
Bugs aren’t rare Bugs aren’t hard to find Cross Sectional Findings Major document platforms may have bugs, but so do their competitors Longitudinal Findings Everybody’s code is getting better Nobody’s code is perfect Run the latest version of everything! Operational Conclusions
Has software quality improved over the last ten years? Conclusion:  For the set of formats tested, we find an unambiguous reduction in the number of failures, particularly when those failures show signs of being security-impacting. What next? Better fuzzing Releasing of data! Conclusion
We are immediately releasing for analysis the summary data from the fuzz run There’s lots of interesting meat to chew on / visualize Go to www.fuzzmark.com for more We are not dropping 0day on anyone Will provide the vendors with all the test cases they want If you want the test cases, or even the stack traces, they’ll have to give us permission  Oh Yeah, Data!

Weitere ähnliche Inhalte

Was ist angesagt?

Dmk sb2010 web_defense
Dmk sb2010 web_defenseDmk sb2010 web_defense
Dmk sb2010 web_defenseDan Kaminsky
 
Man vs Internet - Current challenges and future tendencies of establishing tr...
Man vs Internet - Current challenges and future tendencies of establishing tr...Man vs Internet - Current challenges and future tendencies of establishing tr...
Man vs Internet - Current challenges and future tendencies of establishing tr...Luis Grangeia
 
Domain Key Infrastructure (From Black Hat USA)
Domain Key Infrastructure (From Black Hat USA)Domain Key Infrastructure (From Black Hat USA)
Domain Key Infrastructure (From Black Hat USA)Dan Kaminsky
 
Mere Paas Teensy Hai (Nikhil Mittal)
Mere Paas Teensy Hai (Nikhil Mittal)Mere Paas Teensy Hai (Nikhil Mittal)
Mere Paas Teensy Hai (Nikhil Mittal)ClubHack
 
Hacking the future with USB HID
Hacking the future with USB HIDHacking the future with USB HID
Hacking the future with USB HIDNikhil Mittal
 
A Gentle introduction to microservices
A Gentle introduction to microservicesA Gentle introduction to microservices
A Gentle introduction to microservicesGianluca Padovani
 
Teensy Programming for Everyone
Teensy Programming for EveryoneTeensy Programming for Everyone
Teensy Programming for EveryoneNikhil Mittal
 
Enter The back|track Linux Dragon
Enter The back|track Linux DragonEnter The back|track Linux Dragon
Enter The back|track Linux DragonAndrew Kozma
 
BSides London 2017 - Hunt Or Be Hunted
BSides London 2017 - Hunt Or Be HuntedBSides London 2017 - Hunt Or Be Hunted
BSides London 2017 - Hunt Or Be HuntedAlex Davies
 
SSL: Past, Present and Future
SSL: Past, Present and FutureSSL: Past, Present and Future
SSL: Past, Present and FutureLuis Grangeia
 
From a web application to a distributed system
From a web application to a distributed systemFrom a web application to a distributed system
From a web application to a distributed systemGianluca Padovani
 
Five Cliches of Online Game Development
Five Cliches of Online Game DevelopmentFive Cliches of Online Game Development
Five Cliches of Online Game Developmentiandundore
 
More fun using Kautilya
More fun using KautilyaMore fun using Kautilya
More fun using KautilyaNikhil Mittal
 

Was ist angesagt? (20)

Dmk shmoo2007
Dmk shmoo2007Dmk shmoo2007
Dmk shmoo2007
 
Dmk sb2010 web_defense
Dmk sb2010 web_defenseDmk sb2010 web_defense
Dmk sb2010 web_defense
 
Confidence web
Confidence webConfidence web
Confidence web
 
Man vs Internet - Current challenges and future tendencies of establishing tr...
Man vs Internet - Current challenges and future tendencies of establishing tr...Man vs Internet - Current challenges and future tendencies of establishing tr...
Man vs Internet - Current challenges and future tendencies of establishing tr...
 
Domain Key Infrastructure (From Black Hat USA)
Domain Key Infrastructure (From Black Hat USA)Domain Key Infrastructure (From Black Hat USA)
Domain Key Infrastructure (From Black Hat USA)
 
Dmk bo2 k8_bh_fed
Dmk bo2 k8_bh_fedDmk bo2 k8_bh_fed
Dmk bo2 k8_bh_fed
 
Luis Grangeia IBWAS
Luis Grangeia IBWASLuis Grangeia IBWAS
Luis Grangeia IBWAS
 
Dmk bo2 k8
Dmk bo2 k8Dmk bo2 k8
Dmk bo2 k8
 
Dmk bo2 k8_ccc
Dmk bo2 k8_cccDmk bo2 k8_ccc
Dmk bo2 k8_ccc
 
Dmk blackops2006
Dmk blackops2006Dmk blackops2006
Dmk blackops2006
 
Mere Paas Teensy Hai (Nikhil Mittal)
Mere Paas Teensy Hai (Nikhil Mittal)Mere Paas Teensy Hai (Nikhil Mittal)
Mere Paas Teensy Hai (Nikhil Mittal)
 
Hacking the future with USB HID
Hacking the future with USB HIDHacking the future with USB HID
Hacking the future with USB HID
 
A Gentle introduction to microservices
A Gentle introduction to microservicesA Gentle introduction to microservices
A Gentle introduction to microservices
 
Teensy Programming for Everyone
Teensy Programming for EveryoneTeensy Programming for Everyone
Teensy Programming for Everyone
 
Enter The back|track Linux Dragon
Enter The back|track Linux DragonEnter The back|track Linux Dragon
Enter The back|track Linux Dragon
 
BSides London 2017 - Hunt Or Be Hunted
BSides London 2017 - Hunt Or Be HuntedBSides London 2017 - Hunt Or Be Hunted
BSides London 2017 - Hunt Or Be Hunted
 
SSL: Past, Present and Future
SSL: Past, Present and FutureSSL: Past, Present and Future
SSL: Past, Present and Future
 
From a web application to a distributed system
From a web application to a distributed systemFrom a web application to a distributed system
From a web application to a distributed system
 
Five Cliches of Online Game Development
Five Cliches of Online Game DevelopmentFive Cliches of Online Game Development
Five Cliches of Online Game Development
 
More fun using Kautilya
More fun using KautilyaMore fun using Kautilya
More fun using Kautilya
 

Ähnlich wie Showing How Security Has (And Hasn't) Improved, After Ten Years Of Trying

From 🤦 to 🐿️
From 🤦 to 🐿️From 🤦 to 🐿️
From 🤦 to 🐿️Ori Pekelman
 
Whittaker How To Break Software Security - SoftTest Ireland
Whittaker How To Break Software Security - SoftTest IrelandWhittaker How To Break Software Security - SoftTest Ireland
Whittaker How To Break Software Security - SoftTest IrelandDavid O'Dowd
 
Fixing security by fixing software development
Fixing security by fixing software developmentFixing security by fixing software development
Fixing security by fixing software developmentNick Galbreath
 
Chaos Engineering Without Observability ... Is Just Chaos
Chaos Engineering Without Observability ... Is Just ChaosChaos Engineering Without Observability ... Is Just Chaos
Chaos Engineering Without Observability ... Is Just ChaosCharity Majors
 
Derby con 2014
Derby con 2014Derby con 2014
Derby con 2014TonikJDK
 
BsidesMCR_2016-what-can-infosec-learn-from-devops
BsidesMCR_2016-what-can-infosec-learn-from-devopsBsidesMCR_2016-what-can-infosec-learn-from-devops
BsidesMCR_2016-what-can-infosec-learn-from-devopsJames '​-- Mckinlay
 
The Anatomy of Java Vulnerabilities (Devoxx UK 2017)
The Anatomy of Java Vulnerabilities (Devoxx UK 2017)The Anatomy of Java Vulnerabilities (Devoxx UK 2017)
The Anatomy of Java Vulnerabilities (Devoxx UK 2017)Steve Poole
 
Truly Secure: The Steps a Security Practitioner Took to Build a Secure Public...
Truly Secure: The Steps a Security Practitioner Took to Build a Secure Public...Truly Secure: The Steps a Security Practitioner Took to Build a Secure Public...
Truly Secure: The Steps a Security Practitioner Took to Build a Secure Public...John Kinsella
 
Testingfor Sw Security
Testingfor Sw SecurityTestingfor Sw Security
Testingfor Sw Securityankitmehta21
 
DevSecOps and the CI/CD Pipeline
 DevSecOps and the CI/CD Pipeline DevSecOps and the CI/CD Pipeline
DevSecOps and the CI/CD PipelineJames Wickett
 
The Emergent Cloud Security Toolchain for CI/CD
The Emergent Cloud Security Toolchain for CI/CDThe Emergent Cloud Security Toolchain for CI/CD
The Emergent Cloud Security Toolchain for CI/CDJames Wickett
 
Security vulnerabilities for grown ups - GOTOcon 2012
Security vulnerabilities for grown ups - GOTOcon 2012Security vulnerabilities for grown ups - GOTOcon 2012
Security vulnerabilities for grown ups - GOTOcon 2012Vitaly Osipov
 
5-Ways-to-Revolutionize-Your-Software-Testing
5-Ways-to-Revolutionize-Your-Software-Testing5-Ways-to-Revolutionize-Your-Software-Testing
5-Ways-to-Revolutionize-Your-Software-TestingMary Clemons
 
From DevOps to NoOps how not to get Equifaxed Apidays
From DevOps to NoOps how not to get Equifaxed ApidaysFrom DevOps to NoOps how not to get Equifaxed Apidays
From DevOps to NoOps how not to get Equifaxed ApidaysOri Pekelman
 
RSA 2015 Blending the Automated and the Manual: Making Application Vulnerabil...
RSA 2015 Blending the Automated and the Manual: Making Application Vulnerabil...RSA 2015 Blending the Automated and the Manual: Making Application Vulnerabil...
RSA 2015 Blending the Automated and the Manual: Making Application Vulnerabil...Denim Group
 

Ähnlich wie Showing How Security Has (And Hasn't) Improved, After Ten Years Of Trying (20)

Broken by design (Danny Fullerton)
Broken by design (Danny Fullerton)Broken by design (Danny Fullerton)
Broken by design (Danny Fullerton)
 
App locker
App lockerApp locker
App locker
 
From 🤦 to 🐿️
From 🤦 to 🐿️From 🤦 to 🐿️
From 🤦 to 🐿️
 
Whittaker How To Break Software Security - SoftTest Ireland
Whittaker How To Break Software Security - SoftTest IrelandWhittaker How To Break Software Security - SoftTest Ireland
Whittaker How To Break Software Security - SoftTest Ireland
 
Avc prot 2012b_en
Avc prot 2012b_enAvc prot 2012b_en
Avc prot 2012b_en
 
Fixing security by fixing software development
Fixing security by fixing software developmentFixing security by fixing software development
Fixing security by fixing software development
 
Chaos Engineering Without Observability ... Is Just Chaos
Chaos Engineering Without Observability ... Is Just ChaosChaos Engineering Without Observability ... Is Just Chaos
Chaos Engineering Without Observability ... Is Just Chaos
 
Derby con 2014
Derby con 2014Derby con 2014
Derby con 2014
 
BsidesMCR_2016-what-can-infosec-learn-from-devops
BsidesMCR_2016-what-can-infosec-learn-from-devopsBsidesMCR_2016-what-can-infosec-learn-from-devops
BsidesMCR_2016-what-can-infosec-learn-from-devops
 
Avc per 201206_en
Avc per 201206_enAvc per 201206_en
Avc per 201206_en
 
The Anatomy of Java Vulnerabilities (Devoxx UK 2017)
The Anatomy of Java Vulnerabilities (Devoxx UK 2017)The Anatomy of Java Vulnerabilities (Devoxx UK 2017)
The Anatomy of Java Vulnerabilities (Devoxx UK 2017)
 
Truly Secure: The Steps a Security Practitioner Took to Build a Secure Public...
Truly Secure: The Steps a Security Practitioner Took to Build a Secure Public...Truly Secure: The Steps a Security Practitioner Took to Build a Secure Public...
Truly Secure: The Steps a Security Practitioner Took to Build a Secure Public...
 
Testingfor Sw Security
Testingfor Sw SecurityTestingfor Sw Security
Testingfor Sw Security
 
DevSecOps and the CI/CD Pipeline
 DevSecOps and the CI/CD Pipeline DevSecOps and the CI/CD Pipeline
DevSecOps and the CI/CD Pipeline
 
The Emergent Cloud Security Toolchain for CI/CD
The Emergent Cloud Security Toolchain for CI/CDThe Emergent Cloud Security Toolchain for CI/CD
The Emergent Cloud Security Toolchain for CI/CD
 
Security vulnerabilities for grown ups - GOTOcon 2012
Security vulnerabilities for grown ups - GOTOcon 2012Security vulnerabilities for grown ups - GOTOcon 2012
Security vulnerabilities for grown ups - GOTOcon 2012
 
5-Ways-to-Revolutionize-Your-Software-Testing
5-Ways-to-Revolutionize-Your-Software-Testing5-Ways-to-Revolutionize-Your-Software-Testing
5-Ways-to-Revolutionize-Your-Software-Testing
 
nullcon 2011 - Fuzzing with Complexities
nullcon 2011 - Fuzzing with Complexitiesnullcon 2011 - Fuzzing with Complexities
nullcon 2011 - Fuzzing with Complexities
 
From DevOps to NoOps how not to get Equifaxed Apidays
From DevOps to NoOps how not to get Equifaxed ApidaysFrom DevOps to NoOps how not to get Equifaxed Apidays
From DevOps to NoOps how not to get Equifaxed Apidays
 
RSA 2015 Blending the Automated and the Manual: Making Application Vulnerabil...
RSA 2015 Blending the Automated and the Manual: Making Application Vulnerabil...RSA 2015 Blending the Automated and the Manual: Making Application Vulnerabil...
RSA 2015 Blending the Automated and the Manual: Making Application Vulnerabil...
 

Mehr von Dan Kaminsky

Mehr von Dan Kaminsky (16)

Chicken
ChickenChicken
Chicken
 
I Want These * Bugs Off My * Internet
I Want These * Bugs Off My * InternetI Want These * Bugs Off My * Internet
I Want These * Bugs Off My * Internet
 
Chicken Chicken Chicken Chicken
Chicken Chicken Chicken ChickenChicken Chicken Chicken Chicken
Chicken Chicken Chicken Chicken
 
Some Thoughts On Bitcoin
Some Thoughts On BitcoinSome Thoughts On Bitcoin
Some Thoughts On Bitcoin
 
Interpolique
InterpoliqueInterpolique
Interpolique
 
232 md5-considered-harmful-slides
232 md5-considered-harmful-slides232 md5-considered-harmful-slides
232 md5-considered-harmful-slides
 
Interpolique
InterpoliqueInterpolique
Interpolique
 
Bh us-02-kaminsky-blackops
Bh us-02-kaminsky-blackopsBh us-02-kaminsky-blackops
Bh us-02-kaminsky-blackops
 
Bh eu 05-kaminsky
Bh eu 05-kaminskyBh eu 05-kaminsky
Bh eu 05-kaminsky
 
Bh eu 05-kaminsky
Bh eu 05-kaminskyBh eu 05-kaminsky
Bh eu 05-kaminsky
 
Dmk neut toor
Dmk neut toorDmk neut toor
Dmk neut toor
 
Dmk audioviz
Dmk audiovizDmk audioviz
Dmk audioviz
 
Bo2004
Bo2004Bo2004
Bo2004
 
Gwc3
Gwc3Gwc3
Gwc3
 
Advanced open ssh
Advanced open sshAdvanced open ssh
Advanced open ssh
 
Bh fed-03-kaminsky
Bh fed-03-kaminskyBh fed-03-kaminsky
Bh fed-03-kaminsky
 

Kürzlich hochgeladen

React JS; all concepts. Contains React Features, JSX, functional & Class comp...
React JS; all concepts. Contains React Features, JSX, functional & Class comp...React JS; all concepts. Contains React Features, JSX, functional & Class comp...
React JS; all concepts. Contains React Features, JSX, functional & Class comp...Karmanjay Verma
 
Abdul Kader Baba- Managing Cybersecurity Risks and Compliance Requirements i...
Abdul Kader Baba- Managing Cybersecurity Risks  and Compliance Requirements i...Abdul Kader Baba- Managing Cybersecurity Risks  and Compliance Requirements i...
Abdul Kader Baba- Managing Cybersecurity Risks and Compliance Requirements i...itnewsafrica
 
The State of Passkeys with FIDO Alliance.pptx
The State of Passkeys with FIDO Alliance.pptxThe State of Passkeys with FIDO Alliance.pptx
The State of Passkeys with FIDO Alliance.pptxLoriGlavin3
 
Potential of AI (Generative AI) in Business: Learnings and Insights
Potential of AI (Generative AI) in Business: Learnings and InsightsPotential of AI (Generative AI) in Business: Learnings and Insights
Potential of AI (Generative AI) in Business: Learnings and InsightsRavi Sanghani
 
A Framework for Development in the AI Age
A Framework for Development in the AI AgeA Framework for Development in the AI Age
A Framework for Development in the AI AgeCprime
 
React Native vs Ionic - The Best Mobile App Framework
React Native vs Ionic - The Best Mobile App FrameworkReact Native vs Ionic - The Best Mobile App Framework
React Native vs Ionic - The Best Mobile App FrameworkPixlogix Infotech
 
Decarbonising Buildings: Making a net-zero built environment a reality
Decarbonising Buildings: Making a net-zero built environment a realityDecarbonising Buildings: Making a net-zero built environment a reality
Decarbonising Buildings: Making a net-zero built environment a realityIES VE
 
How to Effectively Monitor SD-WAN and SASE Environments with ThousandEyes
How to Effectively Monitor SD-WAN and SASE Environments with ThousandEyesHow to Effectively Monitor SD-WAN and SASE Environments with ThousandEyes
How to Effectively Monitor SD-WAN and SASE Environments with ThousandEyesThousandEyes
 
The Fit for Passkeys for Employee and Consumer Sign-ins: FIDO Paris Seminar.pptx
The Fit for Passkeys for Employee and Consumer Sign-ins: FIDO Paris Seminar.pptxThe Fit for Passkeys for Employee and Consumer Sign-ins: FIDO Paris Seminar.pptx
The Fit for Passkeys for Employee and Consumer Sign-ins: FIDO Paris Seminar.pptxLoriGlavin3
 
Zeshan Sattar- Assessing the skill requirements and industry expectations for...
Zeshan Sattar- Assessing the skill requirements and industry expectations for...Zeshan Sattar- Assessing the skill requirements and industry expectations for...
Zeshan Sattar- Assessing the skill requirements and industry expectations for...itnewsafrica
 
The Future Roadmap for the Composable Data Stack - Wes McKinney - Data Counci...
The Future Roadmap for the Composable Data Stack - Wes McKinney - Data Counci...The Future Roadmap for the Composable Data Stack - Wes McKinney - Data Counci...
The Future Roadmap for the Composable Data Stack - Wes McKinney - Data Counci...Wes McKinney
 
Unleashing Real-time Insights with ClickHouse_ Navigating the Landscape in 20...
Unleashing Real-time Insights with ClickHouse_ Navigating the Landscape in 20...Unleashing Real-time Insights with ClickHouse_ Navigating the Landscape in 20...
Unleashing Real-time Insights with ClickHouse_ Navigating the Landscape in 20...Alkin Tezuysal
 
A Glance At The Java Performance Toolbox
A Glance At The Java Performance ToolboxA Glance At The Java Performance Toolbox
A Glance At The Java Performance ToolboxAna-Maria Mihalceanu
 
Use of FIDO in the Payments and Identity Landscape: FIDO Paris Seminar.pptx
Use of FIDO in the Payments and Identity Landscape: FIDO Paris Seminar.pptxUse of FIDO in the Payments and Identity Landscape: FIDO Paris Seminar.pptx
Use of FIDO in the Payments and Identity Landscape: FIDO Paris Seminar.pptxLoriGlavin3
 
2024 April Patch Tuesday
2024 April Patch Tuesday2024 April Patch Tuesday
2024 April Patch TuesdayIvanti
 
Digital Identity is Under Attack: FIDO Paris Seminar.pptx
Digital Identity is Under Attack: FIDO Paris Seminar.pptxDigital Identity is Under Attack: FIDO Paris Seminar.pptx
Digital Identity is Under Attack: FIDO Paris Seminar.pptxLoriGlavin3
 
Microservices, Docker deploy and Microservices source code in C#
Microservices, Docker deploy and Microservices source code in C#Microservices, Docker deploy and Microservices source code in C#
Microservices, Docker deploy and Microservices source code in C#Karmanjay Verma
 
Testing tools and AI - ideas what to try with some tool examples
Testing tools and AI - ideas what to try with some tool examplesTesting tools and AI - ideas what to try with some tool examples
Testing tools and AI - ideas what to try with some tool examplesKari Kakkonen
 
Moving Beyond Passwords: FIDO Paris Seminar.pdf
Moving Beyond Passwords: FIDO Paris Seminar.pdfMoving Beyond Passwords: FIDO Paris Seminar.pdf
Moving Beyond Passwords: FIDO Paris Seminar.pdfLoriGlavin3
 
Tampa BSides - The No BS SOC (slides from April 6, 2024 talk)
Tampa BSides - The No BS SOC (slides from April 6, 2024 talk)Tampa BSides - The No BS SOC (slides from April 6, 2024 talk)
Tampa BSides - The No BS SOC (slides from April 6, 2024 talk)Mark Simos
 

Kürzlich hochgeladen (20)

React JS; all concepts. Contains React Features, JSX, functional & Class comp...
React JS; all concepts. Contains React Features, JSX, functional & Class comp...React JS; all concepts. Contains React Features, JSX, functional & Class comp...
React JS; all concepts. Contains React Features, JSX, functional & Class comp...
 
Abdul Kader Baba- Managing Cybersecurity Risks and Compliance Requirements i...
Abdul Kader Baba- Managing Cybersecurity Risks  and Compliance Requirements i...Abdul Kader Baba- Managing Cybersecurity Risks  and Compliance Requirements i...
Abdul Kader Baba- Managing Cybersecurity Risks and Compliance Requirements i...
 
The State of Passkeys with FIDO Alliance.pptx
The State of Passkeys with FIDO Alliance.pptxThe State of Passkeys with FIDO Alliance.pptx
The State of Passkeys with FIDO Alliance.pptx
 
Potential of AI (Generative AI) in Business: Learnings and Insights
Potential of AI (Generative AI) in Business: Learnings and InsightsPotential of AI (Generative AI) in Business: Learnings and Insights
Potential of AI (Generative AI) in Business: Learnings and Insights
 
A Framework for Development in the AI Age
A Framework for Development in the AI AgeA Framework for Development in the AI Age
A Framework for Development in the AI Age
 
React Native vs Ionic - The Best Mobile App Framework
React Native vs Ionic - The Best Mobile App FrameworkReact Native vs Ionic - The Best Mobile App Framework
React Native vs Ionic - The Best Mobile App Framework
 
Decarbonising Buildings: Making a net-zero built environment a reality
Decarbonising Buildings: Making a net-zero built environment a realityDecarbonising Buildings: Making a net-zero built environment a reality
Decarbonising Buildings: Making a net-zero built environment a reality
 
How to Effectively Monitor SD-WAN and SASE Environments with ThousandEyes
How to Effectively Monitor SD-WAN and SASE Environments with ThousandEyesHow to Effectively Monitor SD-WAN and SASE Environments with ThousandEyes
How to Effectively Monitor SD-WAN and SASE Environments with ThousandEyes
 
The Fit for Passkeys for Employee and Consumer Sign-ins: FIDO Paris Seminar.pptx
The Fit for Passkeys for Employee and Consumer Sign-ins: FIDO Paris Seminar.pptxThe Fit for Passkeys for Employee and Consumer Sign-ins: FIDO Paris Seminar.pptx
The Fit for Passkeys for Employee and Consumer Sign-ins: FIDO Paris Seminar.pptx
 
Zeshan Sattar- Assessing the skill requirements and industry expectations for...
Zeshan Sattar- Assessing the skill requirements and industry expectations for...Zeshan Sattar- Assessing the skill requirements and industry expectations for...
Zeshan Sattar- Assessing the skill requirements and industry expectations for...
 
The Future Roadmap for the Composable Data Stack - Wes McKinney - Data Counci...
The Future Roadmap for the Composable Data Stack - Wes McKinney - Data Counci...The Future Roadmap for the Composable Data Stack - Wes McKinney - Data Counci...
The Future Roadmap for the Composable Data Stack - Wes McKinney - Data Counci...
 
Unleashing Real-time Insights with ClickHouse_ Navigating the Landscape in 20...
Unleashing Real-time Insights with ClickHouse_ Navigating the Landscape in 20...Unleashing Real-time Insights with ClickHouse_ Navigating the Landscape in 20...
Unleashing Real-time Insights with ClickHouse_ Navigating the Landscape in 20...
 
A Glance At The Java Performance Toolbox
A Glance At The Java Performance ToolboxA Glance At The Java Performance Toolbox
A Glance At The Java Performance Toolbox
 
Use of FIDO in the Payments and Identity Landscape: FIDO Paris Seminar.pptx
Use of FIDO in the Payments and Identity Landscape: FIDO Paris Seminar.pptxUse of FIDO in the Payments and Identity Landscape: FIDO Paris Seminar.pptx
Use of FIDO in the Payments and Identity Landscape: FIDO Paris Seminar.pptx
 
2024 April Patch Tuesday
2024 April Patch Tuesday2024 April Patch Tuesday
2024 April Patch Tuesday
 
Digital Identity is Under Attack: FIDO Paris Seminar.pptx
Digital Identity is Under Attack: FIDO Paris Seminar.pptxDigital Identity is Under Attack: FIDO Paris Seminar.pptx
Digital Identity is Under Attack: FIDO Paris Seminar.pptx
 
Microservices, Docker deploy and Microservices source code in C#
Microservices, Docker deploy and Microservices source code in C#Microservices, Docker deploy and Microservices source code in C#
Microservices, Docker deploy and Microservices source code in C#
 
Testing tools and AI - ideas what to try with some tool examples
Testing tools and AI - ideas what to try with some tool examplesTesting tools and AI - ideas what to try with some tool examples
Testing tools and AI - ideas what to try with some tool examples
 
Moving Beyond Passwords: FIDO Paris Seminar.pdf
Moving Beyond Passwords: FIDO Paris Seminar.pdfMoving Beyond Passwords: FIDO Paris Seminar.pdf
Moving Beyond Passwords: FIDO Paris Seminar.pdf
 
Tampa BSides - The No BS SOC (slides from April 6, 2024 talk)
Tampa BSides - The No BS SOC (slides from April 6, 2024 talk)Tampa BSides - The No BS SOC (slides from April 6, 2024 talk)
Tampa BSides - The No BS SOC (slides from April 6, 2024 talk)
 

Showing How Security Has (And Hasn't) Improved, After Ten Years Of Trying

  • 1. Showing How Security Has (And Hasn't) Improved, After Ten Years Of Trying Dan Kaminsky, Chief Scientist, DKH Inc. Michael Eddington, Partner, Déjà vu Security Adam Cecchitti, Partner, Déjà vu Security
  • 2. 1990’s: Security == Cryptography and Java 2000’s: Bad guys show up, start owning everything Got particularly bad in 2003 We learn how software actually fails 2010’s: The decade of defense OK. We’ve gotten software to fail. How do we make software not fail? Have we gotten any better? History (As I See It)
  • 3. Major efforts going on in the corporate/defensive world to try to get some sense of Return On Investment / “What Actually Works” in defense Everybody’s. Getting. Owned. Something’s not working. Difference between attack and defense Attack doesn’t work, it’s easy to tell. Defense doesn’t work, it’s…not so easy. What sort of metrics would be helpful? The Push For Metrics
  • 4. “How hard is it for an attacker to corrupt a computer exposing this particular codebase?” Security is about systems, not about individual codebases or even just computers, but bear with me Can We Get Metrics ForSoftware Quality?
  • 5. We know more things are getting compromised Is this because attackers are getting better? Or is this because code is getting worse? Hypothesis: Software quality has improved over the last ten years. Corrolary: Code is not getting worse; attackers are getting better A Fundamental Question
  • 6. Standard dataset usually pored through for quality metrics: CVE CVE: Common Vulnerabilities and Exposures Information on 48,425 vulnerabilities found (and presumably fixed) over the last ten years or so This is lots of data! Or is it? What Sort Of Experiments Could We Try?
  • 7. Fundamental bias Conflation: Software quality with finder interest Interest not only in finding the bugs, but reporting the bugs Major human element (at minimum, difficult/impossible to measure as the exclusive metric) Not that much finder interest that still reports back to vendors We have a latency problem The Problem with CVE
  • 8. Could we take the source code from a large number of projects, and look at the number of “bad things” in the source code? Sort of. First issue: Good luck getting source code to targets Could be bypassed if companies published their static analysis rates This would actually be pretty cool, hint hint Second issue: In practice, difficult to distinguish stylistic concerns from actual vulnerabilities Much bigger problem Perhaps Static Analysis?
  • 9. Fuzzing: The use of a large number of generally corrupt and machine generated test cases to attempt to find failure modes in software. Is it possible to use the results of large scale fuzzing as a rough metric for software quality? “Fuzzmarking” Possibly. There’s even some precedent BreakingPoint’s Resiliency Score against networks and network devices What About Fuzzing?
  • 10. Target Formats Office Documents: .doc, .xls., .ppt Portable Documents: .pdf These are the formats with the highest complexity and the greatest exposure Also, empirically they’ve been the ones that are getting us owned the most There are two sorts of studies that can be run against these formats Lets Start With The Highest Risk Stuff
  • 11. Office Documents Microsoft Word/Excel/Powerpoint OpenOffice Writer/Calc/Impress Portable Documents Adobe Acrobat Foxit Reader GhostScript …but there’s something way more interesting we can do. Cross Sectional Studies:Which Parsers Are (Maybe) Safest?
  • 12. We’ve been doing security for ten years. How much has software improved in those ten years? Has it improved? If so, when? We’ve taken software circa 2000, 2003, 2007, and 2010, and run synchronized fuzz tests across all of them This is (potentially) a window into bugs that have never seen the light of a CVE So what does the data look like? Longitudinal Studies:When
  • 13. OK, not really. 175,334 crashes in ~44 days. Details? Lets talk first about process, procedure, and how you’re going to screw this up if you try to do this. 188,000 Crashes In 44 Days
  • 15. Scavenge sample files from Google Internet Archive Looking for files near the year 2000 This did not work out so well… Obtain Sample Files 1
  • 16. Pre-generate all fuzzed files using Peach 100,000 files per format (doc, xls, ppt, pdf) Office: Understand OLE Storage Fuzz the streams in doc/xls/ppt PDF: Decompress streams (Eventually) Generate Fuzzed Files 2
  • 17. Peach Farm Hosted fuzzing service from Déjà vu Security You provide the target, we produce the crashes For this talk… 88 Cores of fuzzing fun 5 Targets 4 File formats >250 GB of logs 3
  • 18. Office 2010/2007 Thousands of Registry Keys (File recovery) Office/OpenOffice Tens of thousands/Gigs of temp files OpenOffice “Quick Start” keeps OO in memory PDF Compressed/Non-Compressed Running of the Fuzzers 4
  • 19. Look correct? Was every iteration a crash? #fail No crashes? Complete set? Runs crash Machines die Logs misplaced Verify Logs 5
  • 20. Collect interesting data Mine data into SQL Easy to query for data sets Mine Logs 6
  • 21. How Not To Fuzz Fuzzing “How hard could opening a program and reading a file be?” Run the same 100k files across 18programs. Every file, every program, every time. Must open the program Must open, read, and parse the file. Must record the result.
  • 22. Personality Some software has very low self esteem and needs constant attention. “Look what I parsed!” Each program ends up with it own personality based on how many deterrents there are to fuzzing. Ranging from whiner to aggressive to needy. Occasionally 30k / 50k runs we’d have to “sweet talk” the exe back into running. Some things were predictable and others randomish. Opening the Program
  • 23. Personality traits From the program Auto Updaters : Version 6.2 –> 6.5 Auto Upgraders : Version 6 -> 7 Bundle ware / Nag ware Recovery of crashed files “Safe modes” Importing Error reporting / Crash Analysis software From the OS Auto Update Anti Virus Scheduled scans / reboots. Startup / OnRun / Preloaders
  • 24. Click here to punch clippy From the Program Pop Ups Registration Surveys Ads Assistants
  • 25. Files from the internet why not DLLs? Dynamic downloads Add Ons Fonts / international support Foxit from 2007 doesn’t have default JPEG support Download and install at runtimes Puts the program out of sync with the rest “Static” downloads Old documents linked to things all over the web. Parts of the fuzzed doc / pdf are just gone. Makes for less interesting fuzzing
  • 26. Parsing the file A few “docs” we crawled were really HTML, JPEG, TIFF, PDF, JavaScript Ended up with a lot of integrity checking of file Crawler and verifier is around 3k lines of C# code Normally we wouldn’t care a crash is a crash to us, but we wanted to make sure our file set was valid.
  • 27. Magic Number check Caused false positive Searched for JavaScript and HTML tags Basic length checks Failed downloads / truncated uploads Hash of file Resume a downloaded crawl that failed. Filename -> guid Many files on the internet share names. “Homework1.doc”, “Expenses.xls” Things to check for
  • 28. Data integrity becomes very important Collection of crashes and mining of the data requires more meta data the more complex you want to be. Tag the results directories with Program name, year, date, suite, file type, version, machine, iterations, and a guid. When in doubt add another guid. No for real. This will barely be enough to manually reconstruct the run in the event things go wrong. Recording the results
  • 29. 175,334 Crashes Across 26 Codebases Word/Excel/Powerpoint 2003/2007/2010 Acrobat/GhostView 2003/2007/2010 Foxit2007/2010 One crash about every six seconds One crash about every ten iterations So What? Data Analysis(Naïve And Totally Misleading Edition)
  • 30. You don’t fix the same bug over and over You also don’t fix bugs in the order they appear Severity matters How do you operationalize bug classification? You don’t fix crashes. You fix bugs.
  • 31. !exploitable: Microsoft debugger extension that: A) “Bucketizes” the crash – allowing it to be matched with similar crashes B) Attempts to measure the severity of the crash Yes, this can be done manually, but: Manual analysis doesn’t scale Told you !exploitable was important
  • 32. 2536 Unique Major/Minor Hashes 1149 in Office 1125 in OpenOffice 181 in Ghostview 70 in Foxit 10 in Acrobat Unique Crashes
  • 33. But that’s Major/Minor Most “minor hashes” represent the same underlying bug under a single major Depends on the product…75-95% chance of minors representing the same bug Lets be conservative -- only consider Major 942 Unique Major Hashes 440 in Office 396 in Open Office 68 in Ghostview 32 in Foxit 5 in Acrobat Slicing Things Down
  • 34. 942 Unique Major Hashes 150 EXPLOITABLE (15%) 188 PROBABLY_EXPLOITABLE (19%) 16 PROBABLY_NOT_EXPLOITABLE (1.6%) 588 UNKNOWN (62%) What About Severity?
  • 35. This is a cross-sectional study Is severity equally distributed across the various parsers? This is a longitudinal study Is severity equally distributed across the various versions? Now, lets merge Severity with our other metrics…
  • 36. Office vs. StarOffice 2003/7/10(Exploitable/Probably Exploitable)
  • 37. Ghostview vs. Foxit vs. Reader 2003/7/10 (E/PE)
  • 38. Fuzzmarkingis implying an across the board effect – code shipped in 2003 was objectively less secure than code shipped in 2010 At least, in parsers for the highly targeted file formats doc/xls/ppt/pdf We would be surprised if this effect showed up in parsers for file formats that do not cross security boundaries (i.e. aren’t targeted) (But aren’t using XML/JSON) Systematic Fuzzing Seems To Reflect Improving Code Quality
  • 39. The fundamental tension We know that some file formats “invite trouble” Lengths Explicit (especially 32 bit) Overlapping Implicit Jump tables Diffs (“fast saving”) .txt is going to end up being a safer format than .doc or .pdf But what about .doc vs. .pdf vs. .rtf? Can One Compare File Formats With Fuzzmarking?
  • 40. Absolute comparisons across file formats suffer conflation between raw danger of the format and quality of the fuzz tests Are we getting more crashes because the format has more things to find? Or are we just better at finding what’s there? Danger!
  • 41. In our case, it’s very obvious why we have more Office crashes than PDF crashes We’re relatively document aware in doc/xls/ppt We’re just flipping bits on PDF Our PDF fuzzer isn’t even decompressing yet We were surprised to see any crashes! (And we didn’t see any E/PE in Acrobat) What about Office Format vs. Office Format? At least we know where we stand in this case
  • 42. At First Glance, Does Seem Like .doc has more bugs than others Total Crashes Per Type Total Unique Majors Per Type
  • 43. When you filter to E/PE, the formats get equal…
  • 44. Only Simultaneously Viewing Severity/Version/Type Helps Office Doc/XLS/PPT E/PE OpenOffice Doc/XLS/PPT E/PE
  • 45. What’s going on? No serious correlation between Office and OpenOffice on .doc risk What’s not going on? .doc, .xls, and .ppt can’t actually have any security differential You can reach any parser from any We’re not going to notice that without really smart template generation Analysis
  • 46. 588 UNKNOWN crashes… 289 TaintedDataControlsBranchSelection (49%) 95 ReadAV (16%) 74 TaintedDataPassedToFunction (13%) 58 TaintedDataReturnedFromFunction (10%) 28 PossibleStackCorruption (5%) 27 0xe06d7363 (5%) 16 StackOverflow (3%) 1 Invalid Handle (0.1%) What About The UNKNOWN cases?
  • 47. Office vs. StarOffice 2003/7/10(UNKNOWN)
  • 48. 1) There’s still improvement, but not as sharp It’s almost like Exploitable / Probably Exploitable bugs are more likely to be fixed than Unknown bugs 2) There’s a far higher floor, even in 2010 From The Unknown Data…
  • 49. Ghostview vs. Foxit vs. Reader 2003/7/10 (UNKNOWN)
  • 50. We’re still seeing improvement PDF had an interesting artifact in E/PE – 2003 was better than 2007 This artifact is repeated in U – why? More features in 2007 parsers More corruption resistance in 2007 parsers Template preference for 2007/2010 parsers What About PDF?
  • 51. Nate Lawson showed: Never just average everything together. Always look at the distributions. Chunking down from 174K crashes to 940 unique vulns throws away a lot of data It matters: Does a given bug take 10 rounds to find? Or 100,000? Another Possible Metric:Bug Rarity Profiles
  • 52. Number of Crashes Per Unique Bug(A “Spectral Fingerprint” For Fuzzmarking) Office 2003 OpenOffice Writer 2003
  • 53. How do you correlate the same bug across Word and Writer? The stack traces are different Obviously the Major/Minor hashes are different But the fuzz file is the same Naïve Numbers 110,637 unique files created crashes 65989 (59%) crashed only one target 44648 (41%) crashed more than one target What About Bugs That Hit More Than One Target?
  • 54. Can we visualize this? Put the filename in the center Treat the left side as “Office Word” Treat the right side as “OpenOffice Writer” Treat distance from the center as distance from now Color by severity Sort by recentness of vuln * severity Make each pattern unique Can We Do Better With Viz?
  • 55. Office Word 2003/2007/2010 v. OpenOffice 2003/2007/2010 [0]
  • 56. Exploitable in Writer 2003, Unknown in Word 2003/2007/2010
  • 57. Easy Dereferencing Back To Major Hashes Unknown in Word 2007, PE in Writer 2010 Major Hashes
  • 58. Other Neat Profiles (About 2000 Total) “Gap Years” – Unknown in SW2003, PE in SW2010 “Welcome To The Club”
  • 59. Targeting an area already being swept by defenders Code may look better than it actually is, because defenders are securing stuff exposed by these very same methods This is ultimately the issue with all benchmarks The only question is whether these bugs are representative of bugs that need to get fixed Since these are the easiest flaws for an attacker to find, maybe this is OK Potential Issues
  • 60. Bitflippingonly works well against file formats that are tightly packed Most new formats are all textual Also require complex grammars This is a first attempt, we intentionally didn’t want to pack too much intelligence in Code coverage Automatic grammar extraction Automatic segmentation of file formats Manual generation of files Integration with memory tracing What about the fuzzer itself?
  • 61. Bugs aren’t rare Bugs aren’t hard to find Cross Sectional Findings Major document platforms may have bugs, but so do their competitors Longitudinal Findings Everybody’s code is getting better Nobody’s code is perfect Run the latest version of everything! Operational Conclusions
  • 62. Has software quality improved over the last ten years? Conclusion: For the set of formats tested, we find an unambiguous reduction in the number of failures, particularly when those failures show signs of being security-impacting. What next? Better fuzzing Releasing of data! Conclusion
  • 63. We are immediately releasing for analysis the summary data from the fuzz run There’s lots of interesting meat to chew on / visualize Go to www.fuzzmark.com for more We are not dropping 0day on anyone Will provide the vendors with all the test cases they want If you want the test cases, or even the stack traces, they’ll have to give us permission  Oh Yeah, Data!
  • 64. Thanks for all the fish! Déjà vu Security Adam Cecchetti adam@dejavusecurity.com Mike Eddington mike@dejavusecurity.com http://dejavusecurity.com http://peachfuzzer.com Dan Kaminsky dan@doxpara.com http://dankaminsky.com http://tinyurl.com/cansecfuzz DKH Inc.