20. NSM - “Analysis and escalation of
indications and warnings to detect and
respond to intrusions”
- Richard Bejtlich
20
21. NSM - “focused on providing an
intrusion analyst with the best possible
information in the shortest amount of
time” - NSMWiki
21
22. Network Security Monitoring
• Advocates focus on detection and that prevention
will fail.
• Believes in inventoried and defensible networks.
• Build entropy from alert (attack) information.
• Provide analysts with the accurate information
and context as fast as possible.
• Products provide collection, People the analysis
22
25. Context
• Going back to the well.
• Providing as much context as possible in
relation to attacks and attackers.
• Security analysis is detective work.
• Able to ask What if? Branch our analysis. React
to new information.
• Providing full fidelity and full context quickly.
25
27. That’s no moon.
• Pretty WebGL globe by Google.
• Each line represents a source IP address of an
attacker.
• Height is frequency and Colour is Severity
• The destination (victim) is located in Sydney
Australia.
• Approximately 420K Snort alerts in a 12 day
period.
27
28. Full Packet Capture
• Complete record of all network transactions.
• If the attack takes place across a network it is
in the packet captures.
• Provides the highest fidelity to analysts.
• Only way to really understand subtle and
targeted attacks.
• Played, Paused, Stop, Rewind using NSM tools.
• No need to have specific logging setup.
28
30. “The difficulty shifts from traffic
collection to traffic analysis. If you can
store hundreds of gigabytes of traffic
per day, how do you make sense of
it?”
- Richard Bejtlich
30
34. Distributed Processing
• Google Map Reduce Whitepaper (2004)
• Google File System Whitepaper (2003)
• Hadoop is an Apache Project for M/R (2007)
• Hadoop File System is a distributed file system
for Hadoop nodes (2007)
• Pig is a data analysis language to ease the
creation of Map / Reduce jobs that run on
Hadoop Clusters (2008)
34
36. Pig
• A acyclic data flow (analysis) language.
• Transforms data rather than queries data.
– Think key/value.
• Builds map reduce jobs that are distributed
across a Hadoop Cluster.
• Write and debug on your laptop and run on the
cluster.
• Relatively easy to learn and brute force.
36
37. Pig
• Loaders
• Types
• UDFs
• Java
• Scripts (Python) and
• Binaries (p0f and Snort)
37
38. Piglatin
• The data analysis language for Pig scripts
• Group on keys and iterate values
• Iterate using FOREACH
• Filter, Distinct, Sort, Count, Avg, Sum
• Join (LEFT, OUTER)
• Piggybank community tools.
• Illustrate, Explain, Dump or Store.
38
40. @packetpig
• Packetloop + Pig = Packetpig
• https://github.com/packetloop/packetpig
• Open Source, Big Data, Security Analytics
• One stop shop for Network Security
Monitoring for large data sets.
• Capable of integrating other NSM tools and
LOGS!
• Visualisations
40
41. Packetpig - Use Cases
• Threat analysis
• Incident Response
• Research
41
42. @packetpig - Features
• Wireshark the Internet!!
• Bin Time
• Threat Analysis
• Traffic Analysis
• Conversations and Flows
• Geo-Location
• Operating System Fingerprinting
• File Dissection
42
44. Wireshark the Internet!
• Access to raw packet captures
• Query, filter, group, sort, count or average
anything in the IP, TCP and UDP Headers.
• PacketLoader() acts as a base class for most
Loaders.
• Used for bandwidth, packets per second,
breakdown by protocol and global queries.
• Search for strange combination of TCP flags
44
46. Binning Time
• Packetpig can process billions and billions of
data points.
• Most other software can’t handle loading
datasets that big.
• Bin on 5m, 30m, 1h, 4h, 8h, 12h, 24h etc.
• Convert your bin time to seconds and pass as
parameter to Packetpig scripts.
• Anything with a timestamp can be binned.
46
48. Threat Analysis
• SnortLoader() is a wrapper for Snort.
• Runs Snort distributed across Hadoop nodes.
• Pass different snort.conf at run time.
• Snort output is returned to the script.
• Loader returns the following schema.
– Timestamp, Sig ID, Priority, Message, Protocol,
– Source IP, SPort, Destination IP, DPort
48
49. Geo Location
• GeoIP User Defined Function (UDF)
• Wraps the Maxmind Geoip Java library.
• Returns
– Country
– ASNum
– Lat/Long
• Can be used in any script with any loader.
49
54. Traffic Analysis
• Detecting covert and encrypted channels
• Packet Size
– “Traffic analysis of SSL Encrypted Web Browsing” - Heyning
Cheng and Ron Avnur
• Packet Size / Inter-packet delay
– “Datamining for Hackers” - Stefan Burschka at the Chaos
Communications Conference.
• Packet Size and Ngram analysis
– “Anomalous Payload-based Network Intrusion Detection” -
Ke Wang and Salvatore J Stolfo.
54
55. Ngram Analysis
• Analyse packet payloads for what ASCII codes
are used between 0-255.
• Perform unigram, bigram and trigram analysis.
• Analyse the number of characters uses in a
frequency of byte ordered plot.
• Packetpig supports N number of grams.
55
59. Deep Packet Inspection
• DNSConversationLoader()
– Access DNS queries and responses
– Timestamp, Query ID, Mode, Name, IP Address and
TTL
• HTTPConversationLoader()
– Access to HTTP fields (e.g. user-agent, set-cookie,
etag)
– Access to requests and responses.
59
61. Malware Domain Analysis
• Track increases and decreases in the number of
queries per domain.
• Track the TTL for domains and see how they
change over time.
• Track the number of IP’s returned for each
domain and how many distinct countries those
IP’s reside in.
61
64. Conversations / Flows
• Track conversation establishment and
termination
• Return all packets related to a conversation
• Return inter-packet delay
• Return the size of packets in the conversation.
• Return the end state of the conversation
64
65. OS Fingerprinting
• FingerprintLoader() is a wrapper for
@lcamtuf’s p0f
• FingerprintLoader() returns the information
from p0f to the Pig script.
• Perform passive operating system detection
across terabytes of data.
65
67. File Extraction
• Analyse every conversation on the network.
• Extraction or just output information
– Choose whether to extract files or not.
– Extract based on mime type or file extension.
– Extract or search for particular hashes.
• Additional file information through libmagic.
• Output file name, file type, name, extension,
MD5, SHA1, SHA256 hashes.
67
69. Is Big Data - Big Surveillance?
• Packet capture is analogous to wire
tapping.
• Distributed processing of full network
data starts to worry you.
• Potential for mis-use, surveillance, data
warehouses.
• Reputation services that sell dossiers on
hashed IP addresses.
• Data mining ‘networks’ for long periods.
69
73. Torrents
• Connect to the top torrent trackers like the
PirateBay 100, dump Seeders and Leechers.
• Record the BitTorrent Client type (entropy).
• Look at all attacks on a particular dataset
• Join Torrent Data and Snort data on Source IP
• What files are downloaded by the people that
trigger IDS alerts?
• Does Torrent data allow me to triangulate?
73
75. A Decent Join
• 420,000 security events over a 12.49 day
period
• 3 Billion Packets analysed.
• Total size of the data was 2.5TB.
• 1,890 sources of attack (distinct IP’s)
• 168,490 torrent peers (seeders and leechers)
• Around 242 torrents from tracking the PB top
100 for Movies, Music, Books
75
76. Attackers and Torrents
• 17 IP addresses matched attacks and torrents.
• 5 Countries - Australia, China, South Africa,
Phillipines, Singapore.
• Mainly protocol anomalies and Spyware
upload detection.
• Two cases are worth looking into further.
• Use Packetpig to gather Snort, Torrent, p0f and
User-Agent data.
76
77. Torrent Files
• Justice.League.Doom.2012.BRRip.XviD.Ac3.Feel-Free
• Tower Heist (2011) DVDRip XviD-MAXSPEED
• Friends with Benefits 2011 R5 LiNE READNFO XViD-IMAGiNE
• The.Adventures.of.Tintin.2011.1080p.BluRay.x264-MaxHD
• 7 Weeks to 100 Push-Ups: Strengthen and Sculpt Your Arms, Abs, C
• The Walking Dead S02E04 HDTV XviD-ASAP[ettv]
• Footloose.2011.DVDRip.XviD- PADDO
• Thor (2011) DVDRip XviD-MAX
• Daredevil - Soundtrack:-:Thar
• Rise of the Planet of the Apes (2011) DVDRip XviD-MAX
• Revenge S01E15 HDTV XviD-LOL [VTV]
• 1000 Photoshop Tips and Tricks (Dec 2010)-Mantesh
• CSI.S12E14.HDTV.XviD-LOL.avi
77
78. The Suspects
• Two attackers analysed due to Snort Alert
severity.
• The South African.
– Snort triggering on Spyware ‘PUT’ to a web site.
• The Australian.
– Protocol anomalies that are worth investigating.
78
79. The South African
• 9 IP addresses from AS5713 SAIX-NET
• Reverse DNS links them to SAIX Proxies.
– e.g. wblv-ip-pcache-4-vif0.telkom-ipnet.co.za.
• Attacks
– (http_inspect) LONG HEADER
– SPYWARE-PUT Trackware funwebproducts
mywebsearchtoolbar-funtools runtime detection
79
81. The South African
• Packetpig UDF provided two lat/longs but they
were worthless.
• Linked to 7 Torrent files.
– “7 Weeks to 100 Push-Ups: Strengthen and Sculpt Your Arms,
Abs, C” linked two IP addresses to the one user.
– “The.Adventures.of.Tintin.2011.DVDRip.XviD-TARGET” linked
another two IP addresses to another user.
– User-agents for these individual users confirmed this.
• Using torrents as a way to triangulate works.
81
82. The South African
• Google the IP addresses.
• Linked to a botnet traffic
– Spreading malware and forum spam.
• Likely some home machine that’s torrents and
is also infected with Malware.
82
86. Packetpig Query
• Packetpig
– SnortLoader()
– FingerPrintLoader()
– HTTPConversationLoader()
– GeoIP UDF to determine Country, ASNum, Lat/Long
– Torrent output as a CSV text file.
86
87. Packetpig Query
• All 9 IP addresses are OpenBSD 3.X
• A number of distinct user-agents making
requests of different web sites.
• User-agents triggering alerts include the
Trident/4.0 or Trident/5.0 and
FunWebProducts user-agent strings.
• Host,Connection=[keep-alive],Accept=[*/*],?Referer,Accept-Language=[en-ZA],User-
Agent,Accept-Encoding=[gzip, deflate],?X-Forwarded-For,?Via:Accept-Charset,Keep-
Alive:Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1; WOW64; Trident/4.0;
FunWebProducts; GTB7.1; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR
3.0.30729; Media Center PC 6.0; HPNTDF; InfoPath.)
87
88. Conclusion
• Just some infected Windows machines sitting
behind OpenBSD proxies that were browsing a
website.
• When analysing the HTTP queries there were
no PUTs.
• Snort signature is matching on the user-agent.
• No reason to run more detailed jobs to dump
all conversations, protocols or files.
88
89. The Australian
• Single IP address from AS18291 Vodafone
Australia.
• Attacks (read: Crimes!)
– TCP Timestamp is outside of PAWS window
– Bad segment, adjusted size <= 0
• Packetpig geoip provides a Lat/Long.
89
90. Packetpig Query
• Packetpig
– SnortLoader()
– FingerPrintLoader()
– HTTPConversationLoader()
– GeoIP UDF to determine Country, ASNum, Lat/Long
– Torrent output as a CSV text file.
90
105. Moral of the Story
• In both cases IDS alerts can be investigated
quickly through a joined query.
• Joining Snort Source IP with p0f, http and
torrent data provides additional context.
• Torrent data provides triangulation when
matched to user-agent information.
• Didn’t justify a query of all connections,
sessions, protocols and files for these sources.
105
107. TOR
• Track TOR Exit Gateways every hour.
• Record every IP address with Geo information
• Correlate TOR Gateway addresses with sources
of attack on the network.
107
112. Information
• Packetpig
– https://github.com/packetloop/packetpig
– @packetpig on twitter.
• Packetloop
– www.packetloop.com
– @packetloop on twitter.
– blog.packetloop.com
112
Hinweis der Redaktion
Juxtaposing the HOLLYWOOD DETECTIVE with what we know to be true in real life. It ’ s a great METAPHOR FOR SECURITY Enriching -> Loss of Resolution Play, Pause, Rewind etc -> One chance and then gone. Exploring through vector space -> Looking at isolated events, little context. Focused on detection -> Focused on Prevention. Event has happened -> Events are yet to occur. Operating on a full copy of the incident -> I think we have logs? a SIEM? Running Algorithms -> Huh? Algo-what? Both end with the matrix.
With important people standing around someone that is expected to see through the matrix and find the offender
On a long enough timeline prevention fails. Exhibited by 2011 Exhibited by half the audience here at Blackhat. It ’ s inherent in our language ‘ Zero Day ’ equates to “ Prevention Failed ”
By that logic detection is the key. This was highlighted in the RSA case. A product bearing the characteristics of Network Security Monitoring tools was used to quantify the breach and aid incident response. A tool bearing all the hallmarks of NSM Good enough for RSA to acquire Netwitness Good enough for the CSO of Netwitness to be promoted to the CSO of RSA.
So what is NSM? Analysis and Escalation Indicators and Warnings Detect and Respond.
http://nsmwiki.org/Main_Page Provide as much information and context to analysts to make a key decision. There are some criticisms of NSM that it relies to much on people. That ’ s like having CCTV and Red Light cameras but no police? And how is having no people working for you? I believe the analyst is an important role for the defending against complex and motivated attacks.
One criticism of NSM is it ’ s reliance on people?!? It ’ s analgous to having CCTV or Red light cameras without Police! People are key to performing analysis.
Any tool can provide collection. Humans do the analysis. Anyone using these kinds of NSM tools?
It ’ s about providing as much linked information about a security event to an analysis as fast as possible. It ’ s what SIEM ’ s do with logs. They correlate on a 4 tuple and provide context to an event. But what happens when the event has passed? - How do you explore the event? - Examine it from different angles? - How do you test and re-test your hyptothesis? And I know some of you are thinking, I don ’ t even have a SIEM ;)
Going back to the well -> Pause, Play, Rewind, Fast Forward. I don ’ t want a loss of fidelity. I don ’ t want minutes, rolled into hours, hours, rolled into days etc.
Just by showing this your mind is already trying to make sense of it. You ’ re searching for context, trying to understand the encoded data. Well over 420K attacks in 12 days, 2.5TB of traffic. Zoom in on South Africa.
BTW Packetpig includes this visualisation.
FPC gives you the highest quality data and greatest possible context related to a security event. It can be played, paused, rewound and fast worded. I can help me find Zero Days in past data.
Somewhere a math teacher just died ;) But through my bad maths I want to make the point -> “ You can ’ t unmask sophisticated attacks without starting to use FPC. NSM + FPC gives you the greatest % chance, the most options the best possible return. But it ’ s not a capture problem. Cards that can capture at 10Gbs and appliances you can buy. It ’ s a processing problem - how do I query, persist and visualise the data. What I want is something like this.
Tao of Network Security Monitoring. “ The difficulty shifts from traffic collection to traffic analysis. If you can store hundreds of gigabytes of traffic per day, how do you make sense of it? ” ... ” How do you pick out the phone call or e-mail....within a sea of billions of conversations? ” “ by keeping a record of the maximum amount of network activity allowed by policy and collection hardware, analysts buy themselves the greatest likelihood of understanding the extent of an intrusion ”
So let ’ s devise a system Store my pcaps on storage nodes. Ask a question of compute nodes. Compute nodes process storage nodes Scale horizontally depending on A> How fast I need my answer? B> How much data I need to process.
Put it another way. I want to ask a 2.5TB question and I am prepared to wait only so long for it. Distribute my question over N nodes. Then I just need to know how many questions I can ask
4 nodes take 8 hours for 2.5 4 nodes take 32 hours for 10TB 128 nodes take 1 hour for 10TB. 512 nodes process 10TB in 15minutes. This is a complex job that we run in-house with Dell 1950 ’ s purchased off an auction site. You can run complex jobs nightly that break down the data you use during the day to speed up data analysis.
GFS - http://research.google.com/archive/gfs.html M/R - http://research.google.com/archive/mapreduc e.html Hadoop - https://hadoop.apache.org/common/ releases.html First Hadoop release 4 September, 2007: rele ase 0.14.1 available Pig First 11 September, 2 008: release 0.1.0 available
Explain HDFS Replication. Explain each box is compute and storage running map reduce jobs. Pig provides the M/R plan.
Loaders load and present the data via a schema. e.g. PigStorage and TextLoader Types are typeful so you can some error indication prior to running a job. - Scalar -> int, long, float, double, bytearray, chararray - Complex -> Map, Bag, Tuples. Grouping of Scalar types. UDF ’ s allow you to pass a value to the UDF and return a value. dns_parser.py, tcp.py with libnids/pynids lib/scripts/
Think of the data as falling through a series of gates where you can pick out keys and return values. It ’ s not a select * from table language.
One of the basic demos. describe, illustrate, explain, dump, store
Need to sell this slide a bit more.
Finish.
May sound ambitious but it operates directly on packet captures at Terabyte scale.
Demo pig/examples/basic_packets.pig pig -x local -f pig/examples/basic_packets.pig -param pcap=data/ web.pcap
rm -rf output/binning/ pig -x local -f pig/examples/binning.pig -param pcap=~/Downloads/SlowchopCaptures/capture.cap mvim output/binning/part Change to 3600 pig -x local -f pig/examples/binning.pig -param pcap=~/Downloads/SlowchopCaptures/capture.cap mvim output/binning/part
Use different signature sets. Process the same data multiple times. Great on their own but joining is more fun ;)
Maxmind provide a free database you can use which isn ’ t as accurate as the paid version but still good for basic analysis.
mvim pig/globe/globe_snort.pig Talk about running the web server. Drag Demo 1 snort onto the globe. Drag Demo 2 Drag Demo 3
Time series. Comparison of each country. Height is fixed to the largest 1 min bin period. Data looks like mvim the data.
Drag the data onto the vis. Show the data. mvim pig/choropleth/choropleth-snort.pig
Link these to conversations.
Link to the paper.
Byte ordered by frequency. Histogram. Think about the total characters used from 0 - 255. We aren ’ t marking which is which.
0-255 on three axis grouped in 16 x,y,z x axis to the left y axis to the right z up 0,0,0 is the bottom of the globe 255,0,0 is the left bottom edge 0,255,0 is the right bottom edge
mvim pig/examples/dns_response_ttl.pig mvim r/examples/dns_response_ttl.r R CMD BATCH r/examples/dns_response_ttl.r
Run Data/Ubigraph/Ubigraph/bin/ubigraph_server python vis/ubigraph/dns.py output/ubigraph-dns/part-m-00000 All DNS queries. Would generally only do this for particular sources or per source. Nodes are the domains queried. Domain length is what I am looking for here.
pig/examples/conversations.pig pig/examples/conversation_info.pig vi output/conversation_info/ They take to long to run so maybe just vim and show the code.
Basic Examples pig -x local -f pig/examples/p0f_http.pig -param pcap=data/ web.pcap pig/examples/p0f_fingerprint.pig pig/examples/p0f_http.pig mvim pig/output/p0f_http Takes a long time to run so just show the output
pig/examples/extract_files.pig Takes way to long to run on capture.cap. So vi the output. Show on capture.cap for just files etc. then add tmp and do it on capture.cap.
Creepy factor ‘ Networks ’ in this case could be social, IP based networks, cellular and phone. Anything that produces a log.
I figured most people here would be fans of Piratebay. Piratebay logo. Take all the attacks we saw in ~3TB and get the sources of the attack. See if I can infer more about the attackers using publicly available Torrent data.
Inspired by the youhavedownloaded idea of tracking torrents over long periods of time. Would we see any of our attackers pop up in the PB top 100? Or would we see them in the full magnet dump file.
Inspired by the http://youhavedownloaded.com idea of tracking torrents over long periods of time. Would we see any of our attackers pop up in the PB top 100? Or would we see them in the full magnet dump file.
Multiple passes across the Piratebay Top 100 and found the same source connecting to the torent multiple times. 17 snort attacker IP addresses were linked to these torrents.
Begin a 6 month stakeout of their premises. RED CAR!!!
Credit @nearmap for these images.
Credit @nearmap for these images.
Credit @nearmap for these images.
Credit @nearmap for these images.
I don ’ t know what all the fuss is about. Load up google maps Hit street view and boom.
That ’ s my guy!
The Geo Location in the FREE Maxmind database was wrong. It ’ s really a University 1000Kms away ;)
- p0f told us that the South African IP ’ s were OpenBSD - The user-agents told us that there were distinct clients behind the proxies. E.g. User-agent didn ’ t match OS detection. - Torrent data acts as a triangulation if there is enough entropy. - Both false alarms in this case. However depending on what was found we would have been able to initiate a job specific to these source IP ’ s.
I figured most people here would be fans of Piratebay. Piratebay logo. Take all the attacks we saw in ~3TB and get the sources of the attack. See if I can infer more about the attackers using publicly available Torrent data.