SlideShare ist ein Scribd-Unternehmen logo
1 von 60
Downloaden Sie, um offline zu lesen
Finding Needles in Haystacks
          (the size of countries)
                            Michael Baker
                            @cloudjunky
                        Ruxcon - October 2012




Sunday, 21 October 12
Acknowledgements

                        David Turnbull @dsturnbull
                        Gerald Kaszuba @gakman
                        Packetpig Committers




Sunday, 21 October 12
Two Rules



Sunday, 21 October 12
The Landscape.



Sunday, 21 October 12
Exhibit A
                        CVE-2011-3192 - “Apache Killer”
                        auxiliary/dos/http/apache_range_dos 2011-08-19
                        normal Apache Range header DoS (Apache Killer)
                        Snort 1:19825
                        /Ranges*x3As*bytes=([dx2D]+x2C){50}/Hsmi
                        /Ranges*x3As*bytes=([dx2D]+[x2Cs]*){50}/
                        Hsmi




Sunday, 21 October 12
Prevention Fails.



Sunday, 21 October 12
Detection is the key.



Sunday, 21 October 12
NSM - “focused on providing an
                intrusion analyst with the best possible
                 information in the shortest amount of
                            time” - NSMWiki




Sunday, 21 October 12
Network Security Monitoring
                        Advocates focus on detection and that
                        prevention will fail.
                        Believes in inventoried and defensible
                        networks.
                        Build entropy from alert (attack)
                        information.
                        Provide analysts with accurate information
                        as fast as possible.


Sunday, 21 October 12
Tools collect.



Sunday, 21 October 12
People analyze.



Sunday, 21 October 12
Network Security Monitoring
                        Squil
                        Argus
                        Flowgrep
                        Snort and Suricata
                        Bro
                        Network Miner
                        Netwitness


Sunday, 21 October 12
It’s all about Context.



Sunday, 21 October 12
Context
                        Enriched information, not just IP Addresses.
                        Additional intelligence on attackers.
                        Allow you to perform detective work
                        What if? Branch analysis and exploring data.
                        Providing full fidelity and full context
                        quickly.



Sunday, 21 October 12
Full Packet Capture
                        Complete record of all network data.
                        Provides the highest fidelity to analysts.
                        Only way to really understand subtle,
                        targeted attacks.
                        Play, pause and rewind your network.
                        No need to have a specific logging setup.



Sunday, 21 October 12
NSM + FPC
                        > % OPTIONS


Sunday, 21 October 12
bit.ly/RdrI6M

Sunday, 21 October 12
“The difficulty shifts from traffic
                 collection to traffic analysis. If you can
                  store hundreds of gigabytes of traffic
                 per day, how do you make sense of it?”
                            - Richard Bejtlich




Sunday, 21 October 12
Big Data is a collection of data sets so large and
                  complex that it becomes difficult to process
                using on-hand database management tools. The
                 challenges include capture, curation, storage,
                   search, sharing, analysis, and visualization.
                                   - Wikipedia




Sunday, 21 October 12
Big Data

                        Cloud - Elastic compute and Cheap Storage
                        Map Reduce - parallel computation
                        Pig, Hive - avoid writing M/R
                        NoSQL - Cassandra and Mongo




Sunday, 21 October 12
Map Reduce




Sunday, 21 October 12
Sunday, 21 October 12
Big Data Scale
                        I want to ask a 2.5TB question
                          Process 2.5TB, 8 hours, 4 Compute units.
                          Process 2.5TB, 4 hours , 8 Compute units.
                          Process 2.5TB, 2 hours, 16 Compute units.
                          Process 2.5TB, 1 hour, 32 Compute units.
                          Process 2.5 TB, 30 minutes, 64 Compute units.
                          Process 2.5 TB , 15 minutes, 128 Compute units.
                        Scale my compute to answer my question.


Sunday, 21 October 12
Big Data Scale
                                    Complex Job (Approx 2.5TB)
         500
                        480
                                                                      Minutes
         375



         250
                              240


         125
                                          120

                                                     60
             0                                                   30
                                                                         15
                        4     8           16         32          64     128



Sunday, 21 October 12
History
                        Google Map Reduce Whitepaper (2004)
                        Google File System Whitepaper (2003)
                        Hadoop is an Apache Project for M/R (2007)
                        Hadoop File System is a distributed file system
                        for Hadoop nodes (2007)
                        Pig is a data analysis language to ease the
                        creation of Map / Reduce jobs that run on
                        Hadoop Clusters (2008)



Sunday, 21 October 12
@packetpig
                        @packetpig = Packets (FPC) + Pig
                        Pig uses a data flow language called Pig
                        Latin.
                        Executes Map/Reduce Jobs over Hadoop
                        Clusters.
                        Works identically on-premise or in the
                        cloud (Amazon’s EMR)



Sunday, 21 October 12
Features
                        Full access to IP packets at scale.
                        Threat Analysis (Snort)
                        Traffic Analysis.
                        Flow-based deep packet inspection.
                        Geo-Location
                        Passive OS Detection (p0f)
                        File Dissection


Sunday, 21 October 12
Sunday, 21 October 12
Finding Zero Days



Sunday, 21 October 12
Worth a coffee JD?
                        Motivation
                        Time window
                        Attacker
                        Attack type
                        Target
                        Obfuscated
                        Anonymised


Sunday, 21 October 12
Attacker Information



Sunday, 21 October 12
File Extraction



Sunday, 21 October 12
Big Data
                        Security Analytics


Sunday, 21 October 12
Anscombe’s Quartet
                                  I                    II                    III                  IV
                            x          y         x           y         x            y       x           y
                           0.0        8.04      10.0        9.14     10.0          7.46    8.0         6.58

                           8.0        6.95       8.0        8.14      8.0          6.77    8.0         5.76

                           13.0       7.58      13.0        8.74     13.0          12.74   8.0         7.71

                           9.0        8.81       9.0        8.77      9.0          7.11    8.0         8.84

                           11.0       8.33      11.0        9.26      11.0         7.81    8.0         8.47

                           14.0       9.96      14.0        8.10     14.0          8.84    8.0         7.04

                           6.0        7.24       6.0        6.13      6.0          6.08    8.0         5.25

                           4.0        4.26       4.0        3.10      4.0          5.39    19.0        12.50

                           12.0       10.84     12.0        9.13     12.0          8.15    8.0         5.56

                           7.0        4.82       7.0        7.26      7.0          6.42    8.0         7.91

                           5.0        5.68       5.0        4.74      5.0          5.73    8.0         6.89


                        Source: http://en.wikipedia.org/wiki/Anscombe%27s_quartet

Sunday, 21 October 12
Anscombe’s Quartet




                        Source: http://visual.ly/anscombes-quartet

Sunday, 21 October 12
Big Data Security Analytics
                        Visualization       Prediction and
                                            Probability
                        Fidelity
                                            Intelligence sharing
                        Interaction
                                            Statistical Analysis
                        Outlier Detection
                                            Feature Extraction
                        Attacker Profiling
                                            Machine Learning
                        Enrichment

                        Transform



Sunday, 21 October 12
This is not SIEM.



Sunday, 21 October 12
Not SIEM
                        Full Fidelity
                        Explore and explain the data (evidence).
                        Play, Pause and Rewind.
                        Blink and you miss it technology.
                        No aggregation.
                        No parsers or complex integration.
                        Clear intelligence.


Sunday, 21 October 12
Visualisation



Sunday, 21 October 12
Full HD
                        Play, Pause, Rewind


Sunday, 21 October 12
Outlier Detection



Sunday, 21 October 12
Classification



Sunday, 21 October 12
Sunday, 21 October 12
Novelty and Outliers



Sunday, 21 October 12
Sunday, 21 October 12
Entropy and Covert
                             Channels


Sunday, 21 October 12
Enrichment



Sunday, 21 October 12
Geocoding



Sunday, 21 October 12
TOR



Sunday, 21 October 12
Torrent Triangulation



Sunday, 21 October 12
Transformation



Sunday, 21 October 12
Network Graphs and
                           Relationships


Sunday, 21 October 12
Intelligence and Metric
                    Sharing


Sunday, 21 October 12
Indicators of Compromise
                        OpenIOC and CyBOX
                          Open Indicators of Compromise (XML)
                          Host and Network Indicators of
                          Compromise
                        Fork a github repository
                          Execute Packetpig scripts that find bad
                          things and visualise them


Sunday, 21 October 12
DNS and Malware



Sunday, 21 October 12
www.weatherzone.com.au
                                            www.watoday.com.au
                                        www.tweednews.com.au
                                             www.triplem.com.au
                                       www.tradingroom.com.au
                                        www.tradingpost.com.au
                                  www.themorningbulletin.com.au
                                          www.theherald.com.au
                                       www.thechronicle.com.au
                                             www.theage.com.au
                                 www.sunshinecoastdaily.com.au
                                                www.stayz.com.au
                                           www.smhshop.com.au
                                   www.smartedition.smh.com.au
                                                 www.rsvp.com.au
                                                    www.qt.com.au
                                           www.portnews.com.au
                                 www.northerndailyleader.com.au
                                        www.magic1278.com.au
                                        www.investsmart.com.au
                                        www.goodguides.com.au
                                                   www.fox.com.au
                        dns$V1




                                     www.fairfaxsyndication.com
                                       www.fairfaxevents.com.au
                                               www.facebook.com
                                           www.adcentre.com.au
                                                 www.3aw.com.au
                                                        twitter.com
                                             tributes.smh.com.au
                                              tradingroom.com.au
                                     subscriptions.fairfax.com.au
                                                  smhshop.com.au
                                                magic1278.com.au
                                                    m.smh.com.au
                                               investsmart.com.au
                                               goodguides.com.au
                                           fairfaxsyndication.com
                                             fairfaxevents.com.au
                                     dsa.f2.com.au.edgesuite.net
                                        classifieds.fairfax.com.au
                                                 apndigital.com.au
                                                      apm.com.au
                                                  adcentre.com.au
                                              a1040.g.akamai.net
                                                      3aw.com.au


                                                                      2000     4000   6000   8000   10000
                                                                      dns$V2



Sunday, 21 October 12
Analytics or
                        Surveillance



Sunday, 21 October 12
bit.ly/TzcSq8

Sunday, 21 October 12
Questions?
                        @packetpig @packetloop




Sunday, 21 October 12
Thank you!
                        http://blog.packetloop.com




Sunday, 21 October 12

Weitere ähnliche Inhalte

Ähnlich wie Ruxcon Finding Needles in Haystacks (the size of countries)

SMART TOOLS: DISSECT, DIGEST AND DELIVER BIG DATA from Structure:Data 2012
SMART TOOLS: DISSECT, DIGEST AND DELIVER BIG DATA from Structure:Data 2012SMART TOOLS: DISSECT, DIGEST AND DELIVER BIG DATA from Structure:Data 2012
SMART TOOLS: DISSECT, DIGEST AND DELIVER BIG DATA from Structure:Data 2012Gigaom
 
Big Data - architectural concerns for the new age
Big Data - architectural concerns for the new ageBig Data - architectural concerns for the new age
Big Data - architectural concerns for the new ageDebasish Ghosh
 
OpenStack-Design-Summit-HA-Pairs-Are-Not-The-Only-Answer copy.pdf
OpenStack-Design-Summit-HA-Pairs-Are-Not-The-Only-Answer copy.pdfOpenStack-Design-Summit-HA-Pairs-Are-Not-The-Only-Answer copy.pdf
OpenStack-Design-Summit-HA-Pairs-Are-Not-The-Only-Answer copy.pdfOpenStack Foundation
 
OpenStack Summit :: Redundancy Doesn't Always Mean "HA" or "Cluster"
OpenStack Summit :: Redundancy Doesn't Always Mean "HA" or "Cluster"OpenStack Summit :: Redundancy Doesn't Always Mean "HA" or "Cluster"
OpenStack Summit :: Redundancy Doesn't Always Mean "HA" or "Cluster"Randy Bias
 
A Morning with MongoDB Barcelona: Introduction
A Morning with MongoDB Barcelona: IntroductionA Morning with MongoDB Barcelona: Introduction
A Morning with MongoDB Barcelona: IntroductionMongoDB
 
Bio-IT for Core Facility Managers
Bio-IT for Core Facility ManagersBio-IT for Core Facility Managers
Bio-IT for Core Facility ManagersChris Dagdigian
 
Peeling back your Network Layers with Security Onion
Peeling back your Network Layers with Security OnionPeeling back your Network Layers with Security Onion
Peeling back your Network Layers with Security OnionMark Hillick
 
An NSA Big Graph experiment
An NSA Big Graph experimentAn NSA Big Graph experiment
An NSA Big Graph experimentTrieu Nguyen
 
Building A Scalable Open Source Storage Solution
Building A Scalable Open Source Storage SolutionBuilding A Scalable Open Source Storage Solution
Building A Scalable Open Source Storage SolutionPhil Cryer
 
Municipal Government Meets NoSQL
Municipal Government Meets NoSQLMunicipal Government Meets NoSQL
Municipal Government Meets NoSQLMongoDB
 
15 minute presentation about Thesis
15 minute presentation about Thesis15 minute presentation about Thesis
15 minute presentation about ThesisSven Meys
 
A Morning with MongoDB Barcelona: Use Cases and Roadmap
A Morning with MongoDB Barcelona: Use Cases and RoadmapA Morning with MongoDB Barcelona: Use Cases and Roadmap
A Morning with MongoDB Barcelona: Use Cases and RoadmapMongoDB
 
Cloud Foundry Bootcamp
Cloud Foundry BootcampCloud Foundry Bootcamp
Cloud Foundry BootcampAlvaro Videla
 
MongoDB Hadoop and Humongous Data
MongoDB Hadoop and Humongous DataMongoDB Hadoop and Humongous Data
MongoDB Hadoop and Humongous DataMongoDB
 
Data Hiding Techniques
Data Hiding TechniquesData Hiding Techniques
Data Hiding Techniquesprashant3535
 
Java one2011 brisk-and_high_order_bits_from_cassandra_and_hadoop
Java one2011 brisk-and_high_order_bits_from_cassandra_and_hadoopJava one2011 brisk-and_high_order_bits_from_cassandra_and_hadoop
Java one2011 brisk-and_high_order_bits_from_cassandra_and_hadoopsrisatish ambati
 
Beam PHP2012 Workshops: The Cloud
Beam PHP2012 Workshops: The CloudBeam PHP2012 Workshops: The Cloud
Beam PHP2012 Workshops: The CloudJames Dunmore
 
Cloudstack talk
Cloudstack talkCloudstack talk
Cloudstack talkbodepd
 

Ähnlich wie Ruxcon Finding Needles in Haystacks (the size of countries) (20)

SMART TOOLS: DISSECT, DIGEST AND DELIVER BIG DATA from Structure:Data 2012
SMART TOOLS: DISSECT, DIGEST AND DELIVER BIG DATA from Structure:Data 2012SMART TOOLS: DISSECT, DIGEST AND DELIVER BIG DATA from Structure:Data 2012
SMART TOOLS: DISSECT, DIGEST AND DELIVER BIG DATA from Structure:Data 2012
 
Big Data - architectural concerns for the new age
Big Data - architectural concerns for the new ageBig Data - architectural concerns for the new age
Big Data - architectural concerns for the new age
 
OpenStack-Design-Summit-HA-Pairs-Are-Not-The-Only-Answer copy.pdf
OpenStack-Design-Summit-HA-Pairs-Are-Not-The-Only-Answer copy.pdfOpenStack-Design-Summit-HA-Pairs-Are-Not-The-Only-Answer copy.pdf
OpenStack-Design-Summit-HA-Pairs-Are-Not-The-Only-Answer copy.pdf
 
OpenStack Summit :: Redundancy Doesn't Always Mean "HA" or "Cluster"
OpenStack Summit :: Redundancy Doesn't Always Mean "HA" or "Cluster"OpenStack Summit :: Redundancy Doesn't Always Mean "HA" or "Cluster"
OpenStack Summit :: Redundancy Doesn't Always Mean "HA" or "Cluster"
 
A Morning with MongoDB Barcelona: Introduction
A Morning with MongoDB Barcelona: IntroductionA Morning with MongoDB Barcelona: Introduction
A Morning with MongoDB Barcelona: Introduction
 
Bio-IT for Core Facility Managers
Bio-IT for Core Facility ManagersBio-IT for Core Facility Managers
Bio-IT for Core Facility Managers
 
Spark 2013-04-17
Spark 2013-04-17Spark 2013-04-17
Spark 2013-04-17
 
Peeling back your Network Layers with Security Onion
Peeling back your Network Layers with Security OnionPeeling back your Network Layers with Security Onion
Peeling back your Network Layers with Security Onion
 
An NSA Big Graph experiment
An NSA Big Graph experimentAn NSA Big Graph experiment
An NSA Big Graph experiment
 
Building A Scalable Open Source Storage Solution
Building A Scalable Open Source Storage SolutionBuilding A Scalable Open Source Storage Solution
Building A Scalable Open Source Storage Solution
 
Municipal Government Meets NoSQL
Municipal Government Meets NoSQLMunicipal Government Meets NoSQL
Municipal Government Meets NoSQL
 
Xtreme Deployment
Xtreme DeploymentXtreme Deployment
Xtreme Deployment
 
15 minute presentation about Thesis
15 minute presentation about Thesis15 minute presentation about Thesis
15 minute presentation about Thesis
 
A Morning with MongoDB Barcelona: Use Cases and Roadmap
A Morning with MongoDB Barcelona: Use Cases and RoadmapA Morning with MongoDB Barcelona: Use Cases and Roadmap
A Morning with MongoDB Barcelona: Use Cases and Roadmap
 
Cloud Foundry Bootcamp
Cloud Foundry BootcampCloud Foundry Bootcamp
Cloud Foundry Bootcamp
 
MongoDB Hadoop and Humongous Data
MongoDB Hadoop and Humongous DataMongoDB Hadoop and Humongous Data
MongoDB Hadoop and Humongous Data
 
Data Hiding Techniques
Data Hiding TechniquesData Hiding Techniques
Data Hiding Techniques
 
Java one2011 brisk-and_high_order_bits_from_cassandra_and_hadoop
Java one2011 brisk-and_high_order_bits_from_cassandra_and_hadoopJava one2011 brisk-and_high_order_bits_from_cassandra_and_hadoop
Java one2011 brisk-and_high_order_bits_from_cassandra_and_hadoop
 
Beam PHP2012 Workshops: The Cloud
Beam PHP2012 Workshops: The CloudBeam PHP2012 Workshops: The Cloud
Beam PHP2012 Workshops: The Cloud
 
Cloudstack talk
Cloudstack talkCloudstack talk
Cloudstack talk
 

Kürzlich hochgeladen

Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve DecarbonizationUsing IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve DecarbonizationIES VE
 
Secure your environment with UiPath and CyberArk technologies - Session 1
Secure your environment with UiPath and CyberArk technologies - Session 1Secure your environment with UiPath and CyberArk technologies - Session 1
Secure your environment with UiPath and CyberArk technologies - Session 1DianaGray10
 
VoIP Service and Marketing using Odoo and Asterisk PBX
VoIP Service and Marketing using Odoo and Asterisk PBXVoIP Service and Marketing using Odoo and Asterisk PBX
VoIP Service and Marketing using Odoo and Asterisk PBXTarek Kalaji
 
Machine Learning Model Validation (Aijun Zhang 2024).pdf
Machine Learning Model Validation (Aijun Zhang 2024).pdfMachine Learning Model Validation (Aijun Zhang 2024).pdf
Machine Learning Model Validation (Aijun Zhang 2024).pdfAijun Zhang
 
Introduction to Matsuo Laboratory (ENG).pptx
Introduction to Matsuo Laboratory (ENG).pptxIntroduction to Matsuo Laboratory (ENG).pptx
Introduction to Matsuo Laboratory (ENG).pptxMatsuo Lab
 
Basic Building Blocks of Internet of Things.
Basic Building Blocks of Internet of Things.Basic Building Blocks of Internet of Things.
Basic Building Blocks of Internet of Things.YounusS2
 
IaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdf
IaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdfIaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdf
IaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdfDaniel Santiago Silva Capera
 
Igniting Next Level Productivity with AI-Infused Data Integration Workflows
Igniting Next Level Productivity with AI-Infused Data Integration WorkflowsIgniting Next Level Productivity with AI-Infused Data Integration Workflows
Igniting Next Level Productivity with AI-Infused Data Integration WorkflowsSafe Software
 
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdfUiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdfDianaGray10
 
Building AI-Driven Apps Using Semantic Kernel.pptx
Building AI-Driven Apps Using Semantic Kernel.pptxBuilding AI-Driven Apps Using Semantic Kernel.pptx
Building AI-Driven Apps Using Semantic Kernel.pptxUdaiappa Ramachandran
 
UiPath Community: AI for UiPath Automation Developers
UiPath Community: AI for UiPath Automation DevelopersUiPath Community: AI for UiPath Automation Developers
UiPath Community: AI for UiPath Automation DevelopersUiPathCommunity
 
Videogame localization & technology_ how to enhance the power of translation.pdf
Videogame localization & technology_ how to enhance the power of translation.pdfVideogame localization & technology_ how to enhance the power of translation.pdf
Videogame localization & technology_ how to enhance the power of translation.pdfinfogdgmi
 
Cybersecurity Workshop #1.pptx
Cybersecurity Workshop #1.pptxCybersecurity Workshop #1.pptx
Cybersecurity Workshop #1.pptxGDSC PJATK
 
Salesforce Miami User Group Event - 1st Quarter 2024
Salesforce Miami User Group Event - 1st Quarter 2024Salesforce Miami User Group Event - 1st Quarter 2024
Salesforce Miami User Group Event - 1st Quarter 2024SkyPlanner
 
OpenShift Commons Paris - Choose Your Own Observability Adventure
OpenShift Commons Paris - Choose Your Own Observability AdventureOpenShift Commons Paris - Choose Your Own Observability Adventure
OpenShift Commons Paris - Choose Your Own Observability AdventureEric D. Schabell
 
NIST Cybersecurity Framework (CSF) 2.0 Workshop
NIST Cybersecurity Framework (CSF) 2.0 WorkshopNIST Cybersecurity Framework (CSF) 2.0 Workshop
NIST Cybersecurity Framework (CSF) 2.0 WorkshopBachir Benyammi
 
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...UbiTrack UK
 
Designing A Time bound resource download URL
Designing A Time bound resource download URLDesigning A Time bound resource download URL
Designing A Time bound resource download URLRuncy Oommen
 

Kürzlich hochgeladen (20)

Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve DecarbonizationUsing IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
 
Secure your environment with UiPath and CyberArk technologies - Session 1
Secure your environment with UiPath and CyberArk technologies - Session 1Secure your environment with UiPath and CyberArk technologies - Session 1
Secure your environment with UiPath and CyberArk technologies - Session 1
 
VoIP Service and Marketing using Odoo and Asterisk PBX
VoIP Service and Marketing using Odoo and Asterisk PBXVoIP Service and Marketing using Odoo and Asterisk PBX
VoIP Service and Marketing using Odoo and Asterisk PBX
 
Machine Learning Model Validation (Aijun Zhang 2024).pdf
Machine Learning Model Validation (Aijun Zhang 2024).pdfMachine Learning Model Validation (Aijun Zhang 2024).pdf
Machine Learning Model Validation (Aijun Zhang 2024).pdf
 
Introduction to Matsuo Laboratory (ENG).pptx
Introduction to Matsuo Laboratory (ENG).pptxIntroduction to Matsuo Laboratory (ENG).pptx
Introduction to Matsuo Laboratory (ENG).pptx
 
Basic Building Blocks of Internet of Things.
Basic Building Blocks of Internet of Things.Basic Building Blocks of Internet of Things.
Basic Building Blocks of Internet of Things.
 
IaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdf
IaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdfIaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdf
IaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdf
 
Igniting Next Level Productivity with AI-Infused Data Integration Workflows
Igniting Next Level Productivity with AI-Infused Data Integration WorkflowsIgniting Next Level Productivity with AI-Infused Data Integration Workflows
Igniting Next Level Productivity with AI-Infused Data Integration Workflows
 
201610817 - edge part1
201610817 - edge part1201610817 - edge part1
201610817 - edge part1
 
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdfUiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
 
Building AI-Driven Apps Using Semantic Kernel.pptx
Building AI-Driven Apps Using Semantic Kernel.pptxBuilding AI-Driven Apps Using Semantic Kernel.pptx
Building AI-Driven Apps Using Semantic Kernel.pptx
 
UiPath Community: AI for UiPath Automation Developers
UiPath Community: AI for UiPath Automation DevelopersUiPath Community: AI for UiPath Automation Developers
UiPath Community: AI for UiPath Automation Developers
 
Videogame localization & technology_ how to enhance the power of translation.pdf
Videogame localization & technology_ how to enhance the power of translation.pdfVideogame localization & technology_ how to enhance the power of translation.pdf
Videogame localization & technology_ how to enhance the power of translation.pdf
 
Cybersecurity Workshop #1.pptx
Cybersecurity Workshop #1.pptxCybersecurity Workshop #1.pptx
Cybersecurity Workshop #1.pptx
 
Salesforce Miami User Group Event - 1st Quarter 2024
Salesforce Miami User Group Event - 1st Quarter 2024Salesforce Miami User Group Event - 1st Quarter 2024
Salesforce Miami User Group Event - 1st Quarter 2024
 
OpenShift Commons Paris - Choose Your Own Observability Adventure
OpenShift Commons Paris - Choose Your Own Observability AdventureOpenShift Commons Paris - Choose Your Own Observability Adventure
OpenShift Commons Paris - Choose Your Own Observability Adventure
 
20150722 - AGV
20150722 - AGV20150722 - AGV
20150722 - AGV
 
NIST Cybersecurity Framework (CSF) 2.0 Workshop
NIST Cybersecurity Framework (CSF) 2.0 WorkshopNIST Cybersecurity Framework (CSF) 2.0 Workshop
NIST Cybersecurity Framework (CSF) 2.0 Workshop
 
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...
 
Designing A Time bound resource download URL
Designing A Time bound resource download URLDesigning A Time bound resource download URL
Designing A Time bound resource download URL
 

Ruxcon Finding Needles in Haystacks (the size of countries)

  • 1. Finding Needles in Haystacks (the size of countries) Michael Baker @cloudjunky Ruxcon - October 2012 Sunday, 21 October 12
  • 2. Acknowledgements David Turnbull @dsturnbull Gerald Kaszuba @gakman Packetpig Committers Sunday, 21 October 12
  • 3. Two Rules Sunday, 21 October 12
  • 5. Exhibit A CVE-2011-3192 - “Apache Killer” auxiliary/dos/http/apache_range_dos 2011-08-19 normal Apache Range header DoS (Apache Killer) Snort 1:19825 /Ranges*x3As*bytes=([dx2D]+x2C){50}/Hsmi /Ranges*x3As*bytes=([dx2D]+[x2Cs]*){50}/ Hsmi Sunday, 21 October 12
  • 7. Detection is the key. Sunday, 21 October 12
  • 8. NSM - “focused on providing an intrusion analyst with the best possible information in the shortest amount of time” - NSMWiki Sunday, 21 October 12
  • 9. Network Security Monitoring Advocates focus on detection and that prevention will fail. Believes in inventoried and defensible networks. Build entropy from alert (attack) information. Provide analysts with accurate information as fast as possible. Sunday, 21 October 12
  • 12. Network Security Monitoring Squil Argus Flowgrep Snort and Suricata Bro Network Miner Netwitness Sunday, 21 October 12
  • 13. It’s all about Context. Sunday, 21 October 12
  • 14. Context Enriched information, not just IP Addresses. Additional intelligence on attackers. Allow you to perform detective work What if? Branch analysis and exploring data. Providing full fidelity and full context quickly. Sunday, 21 October 12
  • 15. Full Packet Capture Complete record of all network data. Provides the highest fidelity to analysts. Only way to really understand subtle, targeted attacks. Play, pause and rewind your network. No need to have a specific logging setup. Sunday, 21 October 12
  • 16. NSM + FPC > % OPTIONS Sunday, 21 October 12
  • 18. “The difficulty shifts from traffic collection to traffic analysis. If you can store hundreds of gigabytes of traffic per day, how do you make sense of it?” - Richard Bejtlich Sunday, 21 October 12
  • 19. Big Data is a collection of data sets so large and complex that it becomes difficult to process using on-hand database management tools. The challenges include capture, curation, storage, search, sharing, analysis, and visualization. - Wikipedia Sunday, 21 October 12
  • 20. Big Data Cloud - Elastic compute and Cheap Storage Map Reduce - parallel computation Pig, Hive - avoid writing M/R NoSQL - Cassandra and Mongo Sunday, 21 October 12
  • 21. Map Reduce Sunday, 21 October 12
  • 23. Big Data Scale I want to ask a 2.5TB question Process 2.5TB, 8 hours, 4 Compute units. Process 2.5TB, 4 hours , 8 Compute units. Process 2.5TB, 2 hours, 16 Compute units. Process 2.5TB, 1 hour, 32 Compute units. Process 2.5 TB, 30 minutes, 64 Compute units. Process 2.5 TB , 15 minutes, 128 Compute units. Scale my compute to answer my question. Sunday, 21 October 12
  • 24. Big Data Scale Complex Job (Approx 2.5TB) 500 480 Minutes 375 250 240 125 120 60 0 30 15 4 8 16 32 64 128 Sunday, 21 October 12
  • 25. History Google Map Reduce Whitepaper (2004) Google File System Whitepaper (2003) Hadoop is an Apache Project for M/R (2007) Hadoop File System is a distributed file system for Hadoop nodes (2007) Pig is a data analysis language to ease the creation of Map / Reduce jobs that run on Hadoop Clusters (2008) Sunday, 21 October 12
  • 26. @packetpig @packetpig = Packets (FPC) + Pig Pig uses a data flow language called Pig Latin. Executes Map/Reduce Jobs over Hadoop Clusters. Works identically on-premise or in the cloud (Amazon’s EMR) Sunday, 21 October 12
  • 27. Features Full access to IP packets at scale. Threat Analysis (Snort) Traffic Analysis. Flow-based deep packet inspection. Geo-Location Passive OS Detection (p0f) File Dissection Sunday, 21 October 12
  • 29. Finding Zero Days Sunday, 21 October 12
  • 30. Worth a coffee JD? Motivation Time window Attacker Attack type Target Obfuscated Anonymised Sunday, 21 October 12
  • 33. Big Data Security Analytics Sunday, 21 October 12
  • 34. Anscombe’s Quartet I II III IV x y x y x y x y 0.0 8.04 10.0 9.14 10.0 7.46 8.0 6.58 8.0 6.95 8.0 8.14 8.0 6.77 8.0 5.76 13.0 7.58 13.0 8.74 13.0 12.74 8.0 7.71 9.0 8.81 9.0 8.77 9.0 7.11 8.0 8.84 11.0 8.33 11.0 9.26 11.0 7.81 8.0 8.47 14.0 9.96 14.0 8.10 14.0 8.84 8.0 7.04 6.0 7.24 6.0 6.13 6.0 6.08 8.0 5.25 4.0 4.26 4.0 3.10 4.0 5.39 19.0 12.50 12.0 10.84 12.0 9.13 12.0 8.15 8.0 5.56 7.0 4.82 7.0 7.26 7.0 6.42 8.0 7.91 5.0 5.68 5.0 4.74 5.0 5.73 8.0 6.89 Source: http://en.wikipedia.org/wiki/Anscombe%27s_quartet Sunday, 21 October 12
  • 35. Anscombe’s Quartet Source: http://visual.ly/anscombes-quartet Sunday, 21 October 12
  • 36. Big Data Security Analytics Visualization Prediction and Probability Fidelity Intelligence sharing Interaction Statistical Analysis Outlier Detection Feature Extraction Attacker Profiling Machine Learning Enrichment Transform Sunday, 21 October 12
  • 37. This is not SIEM. Sunday, 21 October 12
  • 38. Not SIEM Full Fidelity Explore and explain the data (evidence). Play, Pause and Rewind. Blink and you miss it technology. No aggregation. No parsers or complex integration. Clear intelligence. Sunday, 21 October 12
  • 40. Full HD Play, Pause, Rewind Sunday, 21 October 12
  • 46. Entropy and Covert Channels Sunday, 21 October 12
  • 52. Network Graphs and Relationships Sunday, 21 October 12
  • 53. Intelligence and Metric Sharing Sunday, 21 October 12
  • 54. Indicators of Compromise OpenIOC and CyBOX Open Indicators of Compromise (XML) Host and Network Indicators of Compromise Fork a github repository Execute Packetpig scripts that find bad things and visualise them Sunday, 21 October 12
  • 55. DNS and Malware Sunday, 21 October 12
  • 56. www.weatherzone.com.au www.watoday.com.au www.tweednews.com.au www.triplem.com.au www.tradingroom.com.au www.tradingpost.com.au www.themorningbulletin.com.au www.theherald.com.au www.thechronicle.com.au www.theage.com.au www.sunshinecoastdaily.com.au www.stayz.com.au www.smhshop.com.au www.smartedition.smh.com.au www.rsvp.com.au www.qt.com.au www.portnews.com.au www.northerndailyleader.com.au www.magic1278.com.au www.investsmart.com.au www.goodguides.com.au www.fox.com.au dns$V1 www.fairfaxsyndication.com www.fairfaxevents.com.au www.facebook.com www.adcentre.com.au www.3aw.com.au twitter.com tributes.smh.com.au tradingroom.com.au subscriptions.fairfax.com.au smhshop.com.au magic1278.com.au m.smh.com.au investsmart.com.au goodguides.com.au fairfaxsyndication.com fairfaxevents.com.au dsa.f2.com.au.edgesuite.net classifieds.fairfax.com.au apndigital.com.au apm.com.au adcentre.com.au a1040.g.akamai.net 3aw.com.au 2000 4000 6000 8000 10000 dns$V2 Sunday, 21 October 12
  • 57. Analytics or Surveillance Sunday, 21 October 12
  • 59. Questions? @packetpig @packetloop Sunday, 21 October 12
  • 60. Thank you! http://blog.packetloop.com Sunday, 21 October 12