Effective Prioritization Through Exploit Prediction
•
2 gefällt mir•178 views
The document discusses using machine learning to predict which vulnerabilities are most likely to receive exploits in the future. It describes collecting data on vulnerabilities from CVE databases and observations of exploits and breaches. This data is then used to build a supervised classification model to predict the likelihood of future exploits. The model aims to help prioritize remediation of vulnerabilities that pose the greatest risks. Key challenges addressed include the increasing volume of vulnerabilities and shortening windows between disclosure and exploits.
Empfohlen
Empfohlen
Weitere ähnliche Inhalte
Was ist angesagt?
Was ist angesagt? (20)
Ähnlich wie Effective Prioritization Through Exploit Prediction
Ähnlich wie Effective Prioritization Through Exploit Prediction (20)
Kürzlich hochgeladen
Kürzlich hochgeladen (20)
Effective Prioritization Through Exploit Prediction
- 1. Michael Roytman Jonathan Cran @mroytman @jcran Black Hat 2018 Effective Prioritization Through Exploit Prediction Fixing more of what matters, and less of what doesn’t
- 2. 2
- 3. 3 Complete Remediation is Infeasible Complexity Abound Multiple patch releases from major vendors, including microcode updates Incompatible Antivirus or Endpoint protection Massive Array of Devices Affected Affects Printers, Thermostats, Door Locks, Cameras, Phones, etc Intel’s Nehalem and Westmere (released in 2008 and 2010) affected Not Just Patches Code “should be recompiled with the /Qspectre switch enabled”
- 4. 4 The Modern Stack is COMPLEX Intel / ARM / AMD CPU Hypervisor Java Management Agent Docker Operating System .NET Operating System (Container) Node 3rd party libs Your App Patch Me! Idea Credit: @samnewman Python Ruby PHP App Server / Web Server / etc
- 6. Exploit Release Dates - Tied to CVE Publish
- 8. How to find and fix before an event?
- 9. 9 “Remember the Recall” Infosec is largely a search problem: 1. We are data rich and signal poor. 2. Multi-stage testing cost-effectively increases both precision and recall. 3. Analyst time is the capacity constraint for most security problems We must aim to create signal for our analysts.
- 10. 10 CVSS (alone) Isn’t it
- 11. 11 Events Are What Matters 2018: 36 new CVEs with events
- 12. 12 Targets Not Created Equal
- 13. 13 What Matters for Scoring Is anyone actively targeted? Could we detect success? How much effort is required? What is the attacker payoff? Does a valid attack path exist? score = $CVSS_SCORE score += A if recent_breaches_exist? SCORE += B if exploits_exist? SCORE += C if popular_target? SCORE += D if exploit_will_exist?
- 14. How do we know if we are working on the right stuff?
- 15. 15 Attack + Defense Detect & Respond Predict & Prevent
- 16. 16 Measuring Remediation Strategies Coverage: Of the vulns we fixed, did we pick all (100%) of the correct ones? Efficiency: Of the ones we ended up fixing, did fix any that didn’t matter?
- 17. 17 Coverage & Efficiency, Explained OURS NEIGHBORS ROBOT MOWED Coverage =~ 80% Efficiency =~ 60% EFFICIENCY: Out of all the grass mowed, how much of the grass should have been cut COVERAGE: How much of the grass we wanted to cut was actually cut? wasted effort (inefficiency ) not covered
- 18. 18 Coverage & Efficiency In Practice CVES with known exploits or events CVEs with no known exploit or event Coverage How many vulnerabilities did we prioritize of those that ended up with a known exploit or event Efficiency (green in the red area green + blue) =~ 9.28% Of all the vulnerabilities we prioritized, how many ended up with a kown exploit or event
- 19. 19 Coverage & Efficiency In Practice CVES with known exploits or events CVEs with no known exploit or event Total Prioritized CVEs All CVEs Vulnerabilities prioritized with known exploits or events CVEs prioritized with no known exploits or events Coverage (green / red) How many vulnerabilities did we prioritize of those that ended up with a known exploit or event Efficiency (green / green + blue) Of all the vulnerabilities we prioritized, how many ended up with a known exploit or event
- 20. 20 Coverage / Efficiency Tradeoff ● There exists a natural tradeoff between coverage and efficiency. ● We are operating with incomplete information at any given moment. ● Why would you want <100% efficiency? ○ Abundance of caution (if you can afford it!) ● Why would you want <100% coverage? ○ New campaign can spin up or an older one can spin down. The world is not static. Continuous review and adjustment provides the best result.
- 22. 22 Current Attacker Velocity Average Days from Publish to Exploit (639 / 8%): 19.68 Days Average Days from Publish to Event (36 / 0.5%): 27.36 Days Shortest Window: Adobe Reader (zero days) Longest Window: IE Edge (months)
- 23. 23 Recent Popular Targets (2018) Apache Struts 2.3.x - CVE-2017-5638, CVE-2017-9791, CVE-2017-9805 Joomla! 3.7.1 - CVE-2017-8917 Oracle WebLogic 10.3.6, 12.1.x, 12.2.x - CVE-2017-10271 Jenkins 2.56 - CVE-2017-1000353 Microsoft SMBv1 (ETERNALBLUE) - CVE-2017-0143/4/5 MASTER IPCAMERA (hardcoded password) - CVE-2018-5723 Drupal (Drupalgeddon) - CVE-2018-7600 Adobe Flash - CVE-2018-4878
- 24. Increasing Risk Factoring in Velocity Created Discovery Disclosure Public Exploit Code Released Exploitation Detected In the Wild Detection Generate d
- 25. Exploit Release Dates - Tied to CVE Publish
- 26. The Case for Prediction
- 27. Enter The Exploit Prediction Model
- 28. 28 Future of Data Past Q: “A new vulnerability was just released. Do we scramble? A:
- 29. 29 “Prediction is very difficult, especially about the future” -Niels Bohr
- 30. Data Sources: CVE Enrichment Projects
- 31. Data Sources: Exploit Code & Observations
- 32. 32 What IS Machine Learning? • Methods for automatically learning and recognizing complex patterns from data • A set of tools for understanding data by buildings models from data • measure success on coverage and efficiency
- 33. 33 Type of Algorithms Do you have labeled data? Supervised Unsupervised What do you want to predict? Classification Regression Category NoYes Quantity
- 34. 34 We are current really good at: • “Of my current 300 million vulnerabilities, which ones should I remediate first?” • “Old ones with stable, weaponized exploits, known breaches, high risk meter scores”
- 36. 36 Asking the right questions: • Classification: output is qualitative • prediction: “Will this vulnerability have an exploit written for it?” (== cause more risk later)
- 37. 37 Input variables Numeric cvss_base cvss_temporal kenna_score breach_count cpe count cpe_vendor_count cpe_product_count cve age days_to_first_exploit first_exploit_age days_to_first_breach first_breach_age vulnerability_count open_vulnerability_count closed_vulnerability_count reference_count Binary any_exploits in metasploit in_exploitdb in_elliot any_breaches popular_target recent_breaches rce description_contains_in_the_wild description_contains_buffer_overflow description_contains_man_in_the_middle exploited_in_7 exploited_in_14 exploited_in_30 exploited_in_60 exploited_in_90 breached_in_/ breached_in_14 breached_in_30 breached_in_60 breached_in_90 library_vuln fix_exists Categorical access_vector access_complexity authentication confidentiality_impact integrity_impact availability_impact wasc_id cwe_id fix_type Text cve summary exploit_authors exploit_platforms exploit types exploit_ports cpe_vendors cpe_products reference_types
- 38. 38 Predictive - The Expectations Distribution is not uniform. 77% of dataset is not exploited 1. Accuracy of 77% would be bad Precision matters more than Recall 1. No one would use this model absent actual exploit available data. 2. False Negatives matter less than false positives - wasted effort. We are not modeling when something will be exploited, just IF 1. Could be tomorrow or in 6 months. Re-run the model every day.
- 39. 39 Measuring performance of a predictive model The ideal1 10 Precision Recall Returns relevant documents buy misses many useful ones too Returns most relevant documents but includes lots of junk
- 43. 43 “Most Likely”
- 44. 44 Characteristics of Predicted CVEs ● Common Phrases ○ “Arbitrary Code Execution ○ “Command Injection” ○ “Remote Attackers” ● Vulnerability Classes ○ SQL Injection (CWE-89) ○ Buffer Overflow (CWE-119) ○ Improper Imput Validation (CWE-20)
- 45. 45 The Work Averse Attacker “An attacker massively deploys only one exploit per software version. The only exception we find is for Internet Explorer; the exception is characterised by a very low cost to create an additional exploit, where it is sufficient to essentially copy and paste code from the old exploit, with only few modifications, to obtain the new one.” -The Work-Averse Cyber Attacker Model: Theory and Evidence From Two Million Attack Signatures by Luca Allodi, Fabio Massacci, Julian Williams
- 46. 46 ● CVE-2016-10372 - cpe:/o:eir:d1000_modem_firmware: ○ https://www.rapid7.com/db/modules/exploit/linux/http/tr064_ntpserver_cmdinject ● CVE-2017-18046 - cpe:/o:dasannetworks:h640x_firmware:2.77p1-1124 ○ https://blogs.securiteam.com/index.php/archives/3552 ● CVE-2017-8116 - cpe:/o:teltonika:rut900_firmware:00.03.265 ○ https://labs.nettitude.com/blog/cve-2017-8116-teltonika-router-unauthenticated-re mote-code-execution/ ● CVE-2017-16228 - cpe:/a:dulwich_project:dulwich:0.18.4 ○ [no exploit exists] ● CVE-2017-17946 - cpe:/a:novosoft:handy_password:4.9.3 ○ [no exploit exists] Machine Learning Has Side Benefits
- 48. 48 ● CVE-2016-10372 - cpe:/o:eir:d1000_modem_firmware: ○ https://www.rapid7.com/db/modules/exploit/linux/http/tr064_ntpserver_cmdinject ● CVE-2017-18046 - cpe:/o:dasannetworks:h640x_firmware:2.77p1-1124 ○ https://blogs.securiteam.com/index.php/archives/3552 ● CVE-2017-8116 - cpe:/o:teltonika:rut900_firmware:00.03.265 ○ https://labs.nettitude.com/blog/cve-2017-8116-teltonika-router-unauthenticated-re mote-code-execution/ ● CVE-2017-16228 - cpe:/a:dulwich_project:dulwich:0.18.4 ○ https://twitter.com/jcran/status/1026533985630007296 ● CVE-2017-17946 - cpe:/a:novosoft:handy_password:4.9.3 ○ [no exploit exists] Machine Learning Has Side Benefits
- 49. 49 Interesting, Highly Scored CVEs ● CVE-2016-10717 - MalwareBytes - bypass whitelisting ● CVE-2016-1417 - Snort - DLL hijacking ● CVE-2018-3605 - Trend Micro Control Manager SQLi (RCE) ● CVE-2018-4944 - Adobe Reader Type Confusion ● CVE-2016-7272 - Windows Icon File Integer Overflow
- 50. 50 Constraints on the Future Any new rating system must be: ● Simple (in every sense of the word) ● Explainable (cause and effect understandable) ● Defensible (science!) ● an Improvement And every data source is on the table...
- 51. The future
- 52. 52 Lesson: Less is More New variables aren’t adding much overall
- 53. 53 Lesson: Probability is our friend confusing ^ 78% of vulns are < 1% ● While initially confusing, probability offers a very intuitive measure ● Most vulnerabilities are predicted to have < 1% probability of exploitation 2,400+ vulnerabilities are predicted > 10% ● How can we validate probabilistic estimates?
- 54. 54 Lesson: Probability is our friend confusing ^ ~450 vulnerabilities (what we say) (what we see) Dashed line is “calibrated”
- 55. 55 Real World You can deal with: 44,000,000 Alerts or Fix 299 Vulnerabilities
- 56. 56 Takeaways Volume, complexity and speed of both vulnerabilities and threats are modern vulnerability management challenges Coverage and efficiency allow us to measure vuln management strategies For all the new vulnerabilities you’ve seen this week… is it truly critical? Will it be attacked in the future? Future threats should be addressed, but only after immediate / existing threats