The document discusses Nordstrom's use of Splunk for log aggregation and analytics across their IT systems. Some key points: - Nordstrom uses Splunk to consolidate machine data from various systems like point-of-sale devices, web servers, and monitoring tools for unified visibility. - Splunk has been adopted organically by over 300 users across Nordstrom who use it for tasks like performance monitoring, troubleshooting, and building custom reports. - Nordstrom applies DevOps principles and tools to manage their large and distributed Splunk deployment, with components like search heads, indexers, and deployment servers. Configuration is managed through version control.

1. Copyright
©
2014
Splunk
Inc.
Bernie
Macias
Applied
Architect,
Nordstrom
Mobile
POS,
DevOps
and
the
Role
of
Splunk

2. Disclaimer
2
During
the
course
of
this
presentaIon,
we
may
make
forward-­‐looking
statements
regarding
future
events
or
the
expected
performance
of
the
company.
We
cauIon
you
that
such
statements
reflect
our
current
expectaIons
and
esImates
based
on
factors
currently
known
to
us
and
that
actual
events
or
results
could
differ
materially.
For
important
factors
that
may
cause
actual
results
to
differ
from
those
contained
in
our
forward-­‐looking
statements,
please
review
our
filings
with
the
SEC.
The
forward-­‐looking
statements
made
in
the
this
presentaIon
are
being
made
as
of
the
Ime
and
date
of
its
live
presentaIon.
If
reviewed
aTer
its
live
presentaIon,
this
presentaIon
may
not
contain
current
or
accurate
informaIon.
We
do
not
assume
any
obligaIon
to
update
any
forward-­‐looking
statements
we
may
make.
In
addiIon,
any
informaIon
about
our
roadmap
outlines
our
general
product
direcIon
and
is
subject
to
change
at
any
Ime
without
noIce.
It
is
for
informaIonal
purposes
only,
and
shall
not
be
incorporated
into
any
contract
or
other
commitment.
Splunk
undertakes
no
obligaIon
either
to
develop
the
features
or
funcIonality
described
or
to
include
any
such
feature
or
funcIonality
in
a
future
release.

3. Who
Am
I?
! Current
PosiIon:
1+
years
– Applied
Architect
! Past
Experience:
Since
2005
– Tech
support,
Windows
system
admin,
windows
system
engineer,
system
architect,
Unix
analyst
! Splunk
Experience:
3+
years
– Deployed
approved
Splunk
architecture
at
Capital
One
– Started
the
Splunk>
Sea^le
user
group
! Other
Tech
Interests:
Ongoing
– Python,
Django,
Data
visualizaIon
(d3.js),
Open
Stack
• 3

4. Agenda
! About
Nordstrom
! What s
Nordstrom
Doing
with
Splunk?
! Managing
and
Scaling
Splunk
! DevOps
on
Splunk
! What
About
the
Users?
(Gecng
the
Users
Involved)
4

5. About
Nordstrom
! Founded
by
John
W.
Nordstrom
in
1901
! Over
260
stores
in
US
and
Canada
! eCommerce:
Nordstrom.com,
NordstromRack.com,
HauteLook.com
! 65,000
employees;
over
1,000
in
IT
! MulIple
data
centers
with
an
eye
on
the
cloud
or
mulI-­‐cloud
! DevOps
mission:
scale,
manage,
and
quickly
deliver
on
Nordstrom
iniIaIves
• 5

6. Omni-­‐channel
at
Nordstrom
Deliver
a
Seamless
Customer
Experience
6
Nordstrom
Store
Nordstrom
Online
Nordstrom
Rack
Nordstrom
Rack
Online
/
Haute
Look
Nordstrom
Omni-­‐Channel

7. What’s
Nordstrom
Doing
with
Splunk?

8. Web
apps
OperaIonal
logs,
system
logs,
Web
logs,
Crash
logs
Win/Unix
metrics
(2000+
servers)
Chef
data
POS
and
Wi-­‐Fi
data
NNMi,
SCOM,
Gomez
Splunk
@
Nordstrom
ConsolidaIon
of
all
our
machine
data
for
unified
visibility
8

9. Example:
Point
of
Sales
(POS)
9
• New
customer
experience
• Faster
checkout
anywhere
in
store
• Small
real
estate
footprint
v.
tradiIonal
POS

10. Performance
Monitoring
of
POS
Devices
10

11. 11
POS
Inventory
Mapping
11

12. Splunk
AdopIon
Has
Been
Organic
and
Viral
• AgnosIc
to
technology
• Index
any
type
of
data
from
any
data
source
• Scalable
soluIon
SINGLE
SOURCE
OF
TRUTH
• Accelerated
adopIon
with
300+
users
across
the
organizaIon
• Dev,
App,
NW,
Ops
and
global
offshore
teams
create
their
own
data
inputs,
reports
and
dashboards
EASE
OF
USE
• Role-­‐based
access
controls
to
provide
dev
teams
access
to
producIon
logs
and
metrics
• Flexible
reporIng
across
a
variety
of
use-­‐cases
SECURE
&
FLEXBILE
PLATFORM
12

13. AcceleraIng
Value
by
Using
Splunk
Apps
! Technology
Add-­‐ons:
– *nix
– Windows
– Cisco
IOS
– MicrosoT
Exchange
! Homegrown
App
– Splunk
for
Gomez
– Few
Django
Apps
! Splunk
for
simple
XML
13

14. Secng
the
Stage
for
Managing
and
Scaling
Splunk
14

15. How
Many
of
You
Would
Consider
Yourself…?
15
SYS
ADMIN
SPLUNK
ADMIN
SPLUNK
USER

16. Distributed
Splunk
gets
complex
16

17. 17
All-in-One
Search
Indexers
FW
HFWSYSLOG
TCP
HTTP
FW
HFW
Pool
Storage
Search
Pool #1
Search
Pool #2
GA
Indexers
Pool
Storage
Secure
Indexers
Search
Pool #3
Pool
Storage
Indexers Indexers
FW
HFW
Managed Per
Available Zone
Forwarder/Agents
only send data to
Zone Specific
indexers
TOPOLOGIES
OF
SPLUNK

18. InstallaIon
and
ConfiguraIon
18

19. Using
DevOps
Principles
to
Manage
Splunk
19

20. Why
Not
DevOps
for
Splunk?
20

21. DevOps
Tools
! Config
Management
Plauorm
! Source
Control
! Dev
Environments
! ConInuous
IntegraIon
Tools
for
building
tools
spanning:
21

22. How
Many
Splunk
Components?
! NFS
Server
–
shared
search
head
storage
! Search
Head
–
searches
indexed
data
! Indexer
–
parsing
and
indexing
data
! Deployment
Server
–
App
Deployment
! Intermediate
forwarder
–
receiving
or
collect
data
where
forwarder
cannot
be
install
directly
! Master
–
Cluster
and
ReplicaIon
Master
! Universal
Forwarder
–
local
collecIon
agent
22

23. nord_chef-­‐splunk:
a
CHEF
cookbook
! Ruby
code
that
models
distributed
Splunk
(search,
index,
etc)
! Reuse
able
code
defined
by
a^ributes
CONSISTENT,
SCALABLE,
REPEATABLE
23

24. What's
Automagically
Configured?
• Splunk
SSL
• TCP
and
UDP
listen
Ports
• Set
system
local
configs
• Distributed
Search
• Mounted
Bundles
• Indexers
aware
of
all
search
pools
• And
More
• Move
default
DB
locaIons
• Add
user
and
change
Admin
• Splunk
servers
share
Secret
• Search
pooling
• Set
Deployment
Server
• Search
Heads
aware
of
Indexers
• Drives
configured
• Web
server
24

25. What
About
the
Users?
25

26. What
Can
Users
Do
In
Splunk?
• Create
private
objects
• Cannot
create/edit
global
objects
directly
in
Splunk
• Dashboards?
Extracts?
Saved
searches?
26
USE
GIT!

27. CreaIng
a
Custom
GIT
CLI
• Downloaded
Web
Terminal
for
Splunk
App
• Installed
on
limited
capability
search
head
• Customized
for
GIT
CLI
27

28. Sample
Deployment
Workflow
28
Insert
Image(s)

29. 29
Splunk
is
a
Journey
What’s
Next?

30. Special
Offer:
Try
Splunk
MINT
Express
for
Free!
Splunk
MINT
offers
a
fast
path
to
mobile
intelligence.
How
fast?
Find
out
with
a
6-­‐month
trial*
• Register
for
your
free
trial:
h^p://mint.splunk.com/conf2014offer
• Download
the
Splunk
MINT
SDKs
• Add
the
Splunk
MINT
line
of
SDK
code
and
publish**
• Start
gecng
digital
intelligence
at
your
fingerIps!
*Offer
valid
for
.conf2014
a5endees
and
coworkers
of
a5endees
only.
**Trial
allows
monitoring
of
up
to
750,000
monthly
acDve
users
(MAUs).
30

31. THANK
YOU
31