Ever feel like you spend more time converting security information from one format to another, than actually connecting the dots hidden within it? The Collective Intelligence Framework (CIF) is a data processor for pulling in and normalizing out all these threat intel sources into a single combined dataset. Watch it on-demand http://ow.ly/li8Lf #TTTSec
Biography Of Angeliki Cooney | Senior Vice President Life Sciences | Albany, ...
Â
How to Normalize Threat Intelligence Data from Multiple Sources - Tech Talk Tuesday
1. How to Normalize Threat
Intelligence Data from
Multiple Sources
#TTTsec @AlienVault
Your Host
Conrad Constantine
Community Manager,
AlienVault
@cpconstantineTodd Leetham
Cyber Threat Intelligence Lead, EMC
@rudehimself
2. Covered In This Talk
⢠Getting the Collective Intelligence
Framework installed, collecting intelligence
from external sources and generating a
custom feed to use with your security
controls.
⢠Making sense of the Threat Intelligence
Resources out there as part of your
security monitoring program.
3. What Youâll Need to Build and
Use CIF
⢠A Linux system, (a Debian-based distro is
preferred) with plenty of resources allocated â
4GB and 20GB of storage recommended for
experimentation, 16GB and 500GB recommended
for production.
⢠Experience installing Linux software from source.
⢠Basic DBA skills with the PostGreSQL Database.
⢠Admin experience with the BIND DNS resolver
⢠Admin experience with the Apache Webserver
⢠Know how to locate and install Perl Modules.
⢠Familiarity with essential internet topology
mechanisms (BGP ASâs, registrars, etc)
4. Collective Intelligence Framework:
Redux
⢠Just as a SIEM allows the consumption of log data,
normalizing to allow queries, transforms and correlations to be
run against them, CIF does the same for Threat Intel Data.
⢠IP addresses, Domains, URI substrings â threat intel comes in
many formats and we donât have time to spend our days
converting datasets by hand â automate once, use it forever.
⢠Store Data from multiple sources, combine, process and
produce customized output in formats suitable for
consumption by the security controls you have in place
already.
⢠Query the intelligence data via programming API or human-
readable web interface.
⢠Customize output for different audiences, maintain access
through a key-based API system, share tokenized, sanitized
intelligence amongst multiple organizations without disclosing
sensitive information in the process.
http://code.google.com/p/collective-intelligence-framework/
6. Threat Intelligence For Mere
Mortals⢠Security Controls (for the most part) detect technical
threats â they canât determine intent.
⢠Malicious activity can be indistinguishable from
legitimate, to a software control.
⢠Nothing identifies a False Positive like a second (or
third, or fourth) opinion.
⢠Attackers have agility that defenders do not â
keeping them on the move and unable to launch an
attack from the same place twice raises their costs
of âdoing businessâ
⢠Information about where they are launching attacks
from, what tools they are using â any piece of
information that can make the difference between
responding to an Alert, and responding to a Threat.
7. Putting Threat Intel to Work
⢠Security Controls generate hundreds of alerts per day
(on a slow day).
⢠Threat Intelligence allows you to prioritize response
efforts around caused by external parties known to be
conducting malicious activities.
⢠Threat Intel allows you to group individual alerts together
into a larger picture of coordinated activity against your
assets, and enable you to strike at the roots of an attack
campaign instead of chasing each compromise
individually.
⢠50 compromised machines? Or one Command And
Control system to identify and block communications to?
8. The Threat Intelligence
Marketplace
⢠Public internet threat intelligence began with Anti-
Spam Blacklists.
⢠Now covers a multitude of open repositories of
host/network reputation, malware and exploit
signatures and other more specialized information.
⢠Several Public and Private organizations maintain
private (or commercial subscription) feeds of Threat
Intelligence, ranging from IP Reputation to
specialized research about the individuals carrying
out attacks.
⢠Many emerging standards for defining and
exchanging threat information â and security
controls often have only limited support for
consuming this information.
9. Building your first CIF Server
⢠You either:
â Want to start incorporating some public Threat
Data into your security controls
⢠Or
â Youâre currently consuming several threat
data feeds and want a better way to combine,
aggregate and query them, and process them
with your security controls and analysis tools
11. Prerequisites and
Environment
⢠A working BIND installation on the CIF server, configured to
use trusted public DNS servers for upstream forwarding:
https://code.google.com/p/collective-intelligence-framework/wiki/BindSetup_v1
⢠A working PostgreSQL installation on the CIF server,
configured for user/pass based auth
https://code.google.com/p/collective-intelligence-framework/wiki/PostgresInstall_v1
⢠An Apache Webserver installation, with Mod_Perl loaded.
⢠A fairly extensive collection of Perl modules
http://code.google.com/p/collective-intelligence-framework/wiki/ServerInstall_DebianSqueeze_v1
https://code.google.com/p/collective-intelligence-framework/wiki/ServerInstall_DebianWheezy_v1
https://code.google.com/p/collective-intelligence-framework/wiki/ServerInstall_Ubuntu12_v1
https://code.google.com/p/collective-intelligence-framework/wiki/ServerInstall_CentOS6_v1
https://code.google.com/p/collective-intelligence-framework/wiki/ServerInstall_CentOS5_v1
12. CIF Server Installation
⢠Download the CIF archive, extract it, run the configure scripts.
⢠Build and âmake installâ
⢠Run âmake initdbâ this will fail if postgresql was not configured.
⢠Create a new service account â âcifâ, generate the base CIF
configuration file for it. ~/.cif
⢠Configure Apache to load the CIF http API perl modules via mod_perl.
⢠Install the Cron entries for CIF to update its threat sources periodically
⢠CIF installs to /opt/cif by default.
https://code.google.com/p/collective-intelligence-
framework/wiki/ServerInstall_v1
13. Creating API Keys
⢠Access to the CIF datastore is done via client apps using an API key.
⢠Youâll need to generate an access key for each client that will have
access to the CIF datastore.
⢠The initial key creation is going to look something like this:
$ cif_apikeys -u myuser@mydomain.com -a -g everyone -G everyone
userid key description guid default_guid access write
revoked expires created
myuser@mydomain.com 249cd5fd-04e3-46ad-bf0f-c02030cc864a 8c864306-d21a-37b1-
8705-746a786719bf true all 2012-08-01 11:50:15.969724+00
⢠Youâre going to need this API key to configure a CIF client
14. Installing a Client
⢠The Client is contained in the âlibcifâ source package â install the perl
dependencies and configure && make && make install, as usual.
⢠This contains the âcifâ binary used for commandline interaction with
the CIF server.
⢠Configuration is just the URI for the CIF server API, and the clientâs
API key (generated previous)
https://code.google.com/p/collective-intelligence-framework/wiki/ClientInstall_v1
15. Threat Intel Sources
⢠The default threat intel sources are defined in individual configs in
{installdir}/etc/
⢠They are updated periodically with the {installdir}/bin/cif_crontool
executable.
⢠They define a source of information, and some basic transforms to
begin the normalization process.
⢠Sources are defined with global access rights, confidence levels,
that control how their information is used within CIF client queries.
detection = daily
feed = http://reputation.alienvault.com/reputation.data
source = 'reputation.alienvault.com'
guid = everyone
confidence = 65
severity = medium
restriction = 'need-to-know'
alternativeid = "http://reputation.alienvault.com/reputation.generic"
alternativeid_restriction = 'public'
mirror = /tmp
16. CIF and AlienVault Open Threat Exchange.
⢠CIF comes with a few public Threat Intel
sources by default.
⢠CleanMX, Zeustracker, MalwareDomainList..
⢠âŚ.and AlienVault Open Threat Exchange.
⢠The same IP reputation and Threat Data we
use in the AlienVault product.
⢠With CIF you can consume it..
⢠..With AlienVault OSSIM you can contribute
to it automatically and help take the fight to
the Threat Actors.
17. Querying Feeds
⢠Commandline client allows querying the
normalized feed data by confidence level, type
of activity seen, network location, domain, etc
⢠Query if a URI exists in the Threat Feeds:
$ cif -q 'http://www.yahoo.com/example.htmlâ
⢠Query for all information about hosts on a given network:
$ cif -q 130.201.0.0/16
⢠Has anyone seen this file before? Try a SHA-1 Hash
query:
$ cif -q a5135ec6f2322cc12f3d9daa38dfb358
⢠Some simple Web Interfaces created for the HTTP API,
or query from your own tools if they are capable of
making API queries.
18. Consuming Feeds
⢠CIF comes with a selection of output feed plugins, available via the
commandline tool , using the âp (plugin) argument, using the perl
IODEF module or the HTTP API.
https://github.com/collectiveintel/iodef-pb-simple-perl/tree/master/lib/Iodef/Pb/Format
⢠Some included formats:
ď§ snort rules
ď§ csv
ď§ json
ď§ bindzone
ď§ html table
ď§ ascii table
ď§ bro (network monitor)
ď§ pcap filter
ď§ iptables
19. Putting it to Work
⢠Define feeds that query information according to your
conditions
ď§ Type of Threats observed
ď§ Confidence Levels
ď§ Network Locations, etc etc
⢠Export in a format consumable by your security controls.
⢠Automatically block connections, or just raise priority on
alerts that show up in aggregate threat data.
⢠Create your own data source from your own Security
Analysis work, create limited views on the information
and share with Security Partners.
20. Taking it from Here
⢠Get a basic system up
⢠Start Experimenting with the CIF query tools
⢠Generate a feed to automatically pass on to
one of your security controls or analysis tools.
⢠SIEM WatchLists are excellent things to
populate with Threat Intel, to alert and
prioritize on.
⢠Start responding to attacks made by people,
not signatures triggered by systems.
21. ⢠Collective Intelligence Framework (CIF)
Websitehttps://code.google.com/p/collective-intelligence-framework/
â Server Installation Instructionshttps://code.google.com/p/collective-intelligence-
framework/wiki/ServerInstall_v1
(Donât forget to check the dependencies page for your Linux Distro!)
â Client Installation Instructionshttps://code.google.com/p/collective-intelligence-
framework/wiki/ClientInstall_v1
â API Documentationhttps://code.google.com/p/collective-intelligence-
framework/wiki/API_v1
⢠AlienVault Open Threat Exchange
(OTX)http://www.alienvault.com/alienvault-labs/open-threat-exchange
REFERENCE
24. Thank You.
#TTTsec @AlienVault
Your Host
Conrad Constantine
Community Manager,
AlienVault
@cpconstantine
To learn more about AlienVault please visit:
www.alienvault.com