1. Copyright
©
2014
Splunk
Inc.
Splunk
App
for
Stream
August
12,
2014

2. Agenda
• Capture
101
• Stream
Forwarder
Architecture
• App
for
Stream
Architecture
• Deployment
Architectures
2

3. What
is
Wire
Data?
" Machine
Data
" Poly-­‐Structured
" Record
of
the
CommunicaIon
between
Hosts
3
tcpdump
-­‐qns
0
-­‐A
-­‐r
blah.pcap
20:57:47.368107
IP
205.188.159.57.25
>
67.23.28.65.42385:
tcp
480
0x0000:
4500
0214
834c
4000
3306
f649
cdbc
9f39
E....L@.3..I...9
0x0010:
4317
1c41
0019
a591
50fe
18ca
9da0
4681
C..A....P.....F.
0x0020:
8018
05a8
848f
0000
0101
080a
ffd4
9bb0
................
0x0030:
2e43
6bb9
3232
302d
726c
792d
6461
3033
.Ck.220-­‐rly-­‐da03
0x0040:
2e6d
782e
616f
6c2e
636f
6d20
4553
4d54
.mx.aol.com.ESMT
0x0050:
5020
6d61
696c
5f72
656c
6179
5f69
6e2d
P.mail_relay_in-­‐
0x0060:
6461
3033
2e34
3b20
5468
752c
2030
3920
da03.4;.Thu,.09.
0x0070:
4a75
6c20
3230
3039
2031
363a
3537
3a34
Jul.2009.16:57:4
0x0080:
3720
2d30
3430
300d
0a32
3230
2d41
6d65
7.-­‐0400..220-­‐Ame
0x0090:
7269
6361
204f
6e6c
696e
6520
2841
4f4c
rica.Online.(AOL
0x00a0:
2920
616e
6420
6974
7320
6166
6669
6c69
).and.its.affili
0x00b0:
6174
6564
2063
6f6d
7061
6e69
6573
2064
ated.companies.d

4. OSI
(conceptual)
Model
App
for
Stream:
collects
4
-­‐
7
layers
4

5. 5
Supported
Protocols
In
Splunk
App
for
Stream
v6.0
• UDP
• TCP
• HTTP
• IMAP
• MySQL
(login/
cmd/query)
• Oracle(TNS)
• PostgresSQL
• Sybase/SQL
Server
(TDS)
• FTP
• SMB
• NFS
• POP3
• SMTP
• LDAP/AD
• SIP
• DNS
• DNCP
• Radius
Linux
32-­‐bit/64-­‐bit
and
Mac
OSX
64-­‐Bit
Linux
only

6. Why
Wire
Data?
6
• Wire
data
compliments
Log
data
• Wire
Data
can
contain
IT
and
business
informaIon
not
found
in
Log
data
and
vice
versa
• Wire
Data
can
be
passively
gathered
without
any
impact
to
producIon
workloads
without
tagging,
embedded
code,
or
addiIonal
agents
• Wire
Data
does
not
require
semanIc
logging
by
customer
or
byte-­‐code
instrumentaIon
• Wire
Data
can
be
gathered
across
many
protocols
(SSH,
FTP,
SMTP,
IMAP/MAPI,
TDS,
MQTT,
etc.)
• Can
be
A
LOT
of
data!
A"ribute
Log
Data
Network
Data
WIRE
DATA
/
LOG
DATA
FOR
HTTP
WEB
TRAFFIC

7. What
is
available
from
the
Wire?
7
Performance
Metrics
Round
Trip
Time
Client
Request
Time
Server
Reply
Time
Server
Send
Time
Total
Time
Taken
Base
HTML
Load
Time
Page
Content
Load
Time
Total
Page
Load
Time
ApplicaGon
Data
POST
Content
AJAX
Data
SecIon
Sub-­‐SecIon
Page
Title
Session
Cookie
Proxied
IP
Address
Error
Message
Business
Data
Product
ID
Customer
ID
Shopping
Cart
ID
Cart
Items
Cart
Values
Discounts
Order
ID
Abandoned?

8. Example
Customer
Use
Case
" Customer
FrustraIon
– DBA’s
refuse
to
provide
visibility
to
Log
events
(i.e.
SQL
Queries)
or
DB
performance
! No
Splunk
Forwarder
on
SQL
hosts
– Need
bemer
visibility
into
HTTP
traffic
for
security
purposes
! Logs
from
Web
Servers
contains
some
but
not
all
data
" SoluIon
– Use
App
for
Stream
to
grab
data
off
the
wire
!
Out
of
Band
collecIon
to
get
SQL
performance
using
Stream
from
the
ApplicaIon
side
! “Use
Splunk
as
an
IDS
to
see
strange
things
on
the
wire”
8

9. TURN
WIRE
DATA
INTO
OPERATIONAL
INTELLIGENCE
CLOUD ON-PREMISES
Splunk App for Stream (FREE)

10. Copyright
©
2014
Splunk
Inc.
Stream
Forwarder

11. What
is
“Splunk
Stream
Add-­‐on?”
" Technology
Add-­‐on
or
TA
(Splunk_TA_stream)
" Provides
a
new
Data
Input
called
“Wire
Data”
– passively
captures
traffic
using
a
modular
input
– C++
executable
called
“Stream
Forwarder”
(streamfwd)
" Captures
applicaIon
layer
(level
7)
amributes
for:
– UDP,
TCP,
DHCP,
DNS,
FTP,
HTTP,
IMAP,
LDAP,
MySQL,
NFS,
POP3,
PostgreSQL,
SIP,
SMB,
SMTP,
TDS,
TNS,
and
more
" AutomaIcally
decrypts
SSL/TLS
traffic
using
RSA
keys
11

12. Stream
Forwarder
Architecture
12
Protocol
Decoder
Events
DecrypGon
Request/
Response
Network
Interface
(eth1)
Standard
Out
(To
Splunk
Forwarder)
Packets
Flows
Request/
Response
Request/
Response
Protocol
Decoder
Events
DecrypGon
Standard
Out
(To
Splunk
Forwarder)
Protocol
Decoder
Events
DecrypGon
Standard
Out
(To
Splunk
Forwarder)
Network
Interface
(ethN)
Packets
…
Threads

13. Relevant
Moving
Parts
" All
Plasorm
(Linux
x86,
Linux
x64,
Darwin)
binaries
shipped
with
TA
" inputs.conf
– [streamfwd://streamfwd]
splunk_stream_app_locaIon
=
hmp://localhost:8000/en-­‐us/
custom/splunk_app_stream/
disabled
=
0
" Config
held
in
memory
on
Streamfwd
" Splunk_TA_stream/default/streamfwd.xml
– Which
interfaces
to
listen
on
13

14. Copyright
©
2014
Splunk
Inc.
Splunk
App
for
Stream

15. What
is
“Splunk
App
for
Stream?”
" Includes
a
new
Splunk
Stream
Add-­‐on
(TA)
– AutomaIcally
installs
the
TA
locally
(disabled)
– Makes
it
easy
to
deploy
TA
using
Deployment
Server
" Manages
configuraIon
for
all
Stream
TA’s
" Provides
REST
API
for
configuraIon
" Includes
New
Dashboards
" Supported
Plasorms:
Linux
32/64bit
&
Mac
OSX
64bit
15

16. Stream
Amributes
are
configurable
16

17. AggregaIon
“Many
to
One”
17
Key
amributes
make
aggregaIon
buckets
unique
Sum
amributes
summarize
numeric
metrics
I
want
one
event
every
60
secs
Capture
Data
for
Specific
Events
• Buckets
always
include
a
“count”
amribute
for
#
of
events
it
represents
• Buckets
are
flushed
on
a
configurable
interval
of
Ime

18. Stream
Filters
" Filters
allow
you
to
only
capture
data
for
specific
events
" Example:
HTTP
with
status=404
(File
Not
Found)
18

19. Simple
Deployment
Supports
Fast
Time
to
Value
19
Respond
quickly
to
incidents
by
rapidly
deploying
data
collecIon
directly
from
the
interface
Scale-­‐out
deployment
across
enterprise
networks
with
centralized
configuraIon
and
management

20. Performance
and
Deployment
RecommendaIons

21. Architecture
Deployment
OpIon
1
Dedicated
Server
21
End
Users
SPAN
or
TAP
Firewall
Splunk
Indexers
Search
head
Linux
Forwarder
Splunk_TA_Stream
Servers
Internet

22. Architecture
Deployment
OpIon
2
Run
on
Servers
22
End
Users
Firewall
Splunk
Indexers
Search
head
Physical
or
Virtual
Servers
Universal
Forwarder
Splunk_TA_stream
Internet
Physical
Datacenter,
Public
or
Private
Cloud

23. Summary
23
Enhanced
OperaGonal
Intelligence
Efficient,
Cloud-­‐ready
Wire
Data
CollecGon
Simple
Deployment
Supports
Fast
Time
to
Value
Explore,
analyze
and
visualize
real-­‐Ime
wire
data
for
OperaIonal
Intelligence
Instantly
access
wire
data
across
infrastructures
with
a
simple
so}ware
soluIon;
manage
wire
data
volumes
with
fine-­‐grained
filtering
Enable
rapid
deployment
and
reduced
complexity
with
interface-­‐driven
install
and
configuraIon
Splunk
Stream
Delivers
Wire
Data
AnalyIcs