Splunk Application Monitoring

1. Copyright*©*2012*Splunk*Inc.*
Thinking*Beyond*
Applica=on*Monitoring*
Jus=n*Hardeman,*Unix*Administrator*
Availity*Health*Informa=on*Networks*

2. Legal*No=ces*
During*the*course*of*this*presenta=on,*we*may*make*forwardLlooking*statements*regarding*future*events*or*the*
expected*performance*of*the*company.*We*cau=on*you*that*such*statements*reflect*our*current*
expecta=ons*and*es=mates*based*on*factors*currently*known*to*us*and*that*actual*events*or*results*could*differ*
materially.*For*important*factors*that*may*cause*actual*results*to*differ*from*those*contained*in*our*forwardLlooking*
statements,*please*review*our*filings*with*the*SEC.**The*forwardLlooking*statements*made*in*this*presenta=on*are*
being*made*as*of*the*=me*and*date*of*its*live*presenta=on.**If*reviewed*aTer*its*live*presenta=on,*this*presenta=on*
may*not*contain*current*or*accurate*informa=on.***We*do*not*assume*any*obliga=on*to*update*any*forwardLlooking*
statements*we*may*make.**In*addi=on,*any*informa=on*about*our*roadmap*outlines*our*general*product*direc=on*
and*is*subject*to*change*at*any*=me*without*no=ce.**It*is*for*informa=onal*purposes*only*and*shall*not,*be*
incorporated*into*any*contract*or*other*commitment.**Splunk*undertakes*no*obliga=on*either*to*develop*the*
features*or*func=onality*described*or*to*include*any*such*feature*or*func=onality*in*a*future*release.*
*
Splunk,(the(engine(for(machine(data(is(a(registered(trademark(of(Splunk(Inc.(and/or(its(subsidiaries(and/or(affiliates(in(the(United(States(
and/or(other(jurisdic=ons.(*All(other(brand(names,(product(names(or(trademarks(belong(to(their(respec=ve(holders.((
©2012(Splunk(Inc.(All(rights(reserved.*
2*

3. About*Availity*
3*
Pa=ents.*Not*Paperwork.®"
• A*na=onal*leading*health*informa=on*network*(HIN)*
• Op=mizing*informa=on*exchange*through*a*single,*secure*network*
• 200,000+*physicians*and*1,000+*hospitals*
Mul=ple*payers.*One*website.*One*login.*
• 1300+*health*plans*and*450+*industry*partners*in*one*place*
Aetna,*20+*Blue*Cross*and*Blue*Shield*plans,*CIGNA,*Humana*and*more*
• RealL=me*info*exchange*means*highest*accuracy*of*health*plan*info**
HIPAA*transac=ons*and*proprietary*products.*
• File*claims,*check*eligibility,*review*clinical*history,*process*payments*
• Products*include:**Availity*CareCollect®,CareProfile®,*CareRead®,*CareCost*Es=mator®,*
Remit*Reader®,"and*more*

4. About*Me*
4*
Jus=n*Hardeman*
Unix*Administrator*
Availity*Health*Informa=on*Networks**
Core*Func=ons*
• Produc*on"Opera*ons"(4"years)"
L Monitoring*
L Incident*response*
L 24/7*Support*on*Unix*systems,*authen=ca=on*
systems*and*webLportal*
• Resident"Splunk"user"(3"years)"
L BeLall*and*endLall*of*Splunk*

5. Agenda*
• Going*beyond*system*monitoring*
• Challenges*we’ve*faced*at*Availity*
• Senior*leadership*views*–*how’s*the*business*doing*overall?*
• System*opera=ons*–providing*insights*into*the*nuts*and*bolts,*
and*maybe*even*the*broken*gears*
• Marke=ng*and*training*–*are*people*using*our*demos?*
• Wins!*
• Takeaways?*
*

6. Ask*Yourself*
I*already*have*monitoring,*so*that*should*be*
good*enough,*right?*
6*

7. What*Do*I*Mean*by*Monitoring?*
" The*easy*stuff*
– Is*a*process*up?**Ac=ve*or*stale?*
– What’s*my*CPU/disk/RAM*usage?*
– I*need*to*monitor*a*log*file*for*a*par=cular*error*message*
" What*about*the*more*difficult*measurements*
– How*fast*is*my*average*user*login*=mes?**What*was*it*last*week?*
– What*is*the*top*error*code*for*a*par=cular*payer*and/or*transac=on*
type?*
– How*many*hits*did*we*receive*to*a*training*demo*during*the*
=meframe*of*last*week?**Last*month?*
*
*
7*

8. Before*Splunk*
8*
Reac=ve*approach*to*problems*
Lack*of*visibility*into*root*cause*of*
failures*/*issues*
Long*resolu=on*=mes*–*always*needed*the*
availability*of*the*right*person*at*the*right*
=me*
• RealL=me*visibility*into*issues*with*proac=ve*
aler=ng*
• Reports*and*dashboards*with*realL=me*
transac=ons*and*B2B*workflows*
• Combining*mul=ple*sources*of*informa=on*in*
varying*formats*into*a*single*view*for*capacity*
planning*
• Secure,*roleLbased*views*for*IT*teams*
OPERATIONAL*CHALLENGES* MONITORING*REQUIREMENTS*

9. Transforming*Machine*Data*with*Splunk*
9*
Proac*ve"monitoring"
Real?*me"visibility"
Opera*onal"insight"
Enterprise"adop*on"
Availity’s*data* …*for*applica=on*intelligence*
Red"Hat"Linux"
Servers" Oracle"Iden*ty"
Manager"
Custom"metrics"
from"NFS"Servers"
PlaLorm"independent"SAN"
systems"
Homegrown"
Applica*ons""
Oracle"Databases"

10. Scheduled*Searches*Become*Alerts*
Recent*Request:*
I(need(to(be(alerted(if(the(batch(
aggrega=on(queue(backs(up*
! I*should*have*no*more*then*500*items*
for*longer*than*30*minutes*
! I*need*to*receive*an*email*
! I*need*this*yesterday*
! You*should*be*able*to*find*what*you*
need*in*aries.log*
*
10*

11. Scheduled*Search*Demo/Example*
" Let’s*build*an*alert!*
*
11*

12. Use*Your*Alerts*to*Drive*Dashboard*Usage*
FollowLup*Request:*
Can(you(add(some(visualiza=ons?(
! Can*you*provide*me*with*some*
history?*
! Can*you*show*me*what*my*counts*are*
in*rela=on*to*the*same*=me*last*
week?*
! Can*you*build*it*so*that*I*can*easily*
edit*and*add*to*it*later?*
*
12*

13. Dashboard*Demo/Example*
" Let’s*build*a*dashboard!*
*
13*

14. Team*Dashboards*
• Senior*Leadership**
– Overall*health*of*the*business*
• Account*Management**
– Payer*stats*
– Portal*stats*
• Opera=ons*
– Transac=on*processing*measurements*
– Job*tracking*
• Marke=ng*and*Training*
– Page*hits*
– File*downloads*
14*

15. Senior*Leadership*
" What’s*really*going*on?*
– I*don’t*need*to*see*the*dayLtoLday*issues,*I*need*the*big*picture.*
" Don’t*kill*me*with*data!*
– I*should*receive*what*I*need*in*a*quick*glance.*
" “X”*number*of*events*isn’t*good*enough!*
– There*are*5,000*events.**Is*that*good*or*bad?**Is*that*what*we*
normally*run?*
15*

16. 16*

17. Account*Management*
" How*is*my*account*doing?*
– Are*transac=ons*comple=ng*successfully?*
– Have*response*=mes*increased/decreased?*
" What*do*I*need*to*report*back*to*my*account*contacts?*
– Trouble*transac=on*types*
– Regional*outages*
" What*might*cause*an*increase*in*call*volume*to*the*help*desk?*
17*

18. 18*

19. Opera=ons*
• How*do*the*transac=on*volumes*look?*
• RealL=me,*B2B,*and*Batch*
• Are*automated*connec=on*jobs*occurring?*
• Job*Scheduler*–*Tidal*
• Example*log*entry*L*07/31 15:05:29:740[16:MD-5]: (mem=1609407968/2142568448)
ServerNode: Event = SystemActivityMessage: Job FTP MEDICAREB277IN[189] completed
status [Completed Abnormally].*
• Any*issues*with*our*batch*processing*engine?*
• Homegrown*Applica=on*–*Aries*
• Example*log*entry*L*08/10/2012 13:41:47 [IR-0] INFO c.a.aries.gateway.GatewayScanner
- [splunk] tpt="66",qn="ARIES.INTERNAL.REQUESTS",ts="08-10-2012
13:41:47.723",di="9d531343-1d56-417c-
abac-8c367e87ef99",psz="957",nn="10.xx.xx.xx",qt="76871",ct="16",si="Availity",evi="16
36814868"!
19*

20. 20*

21. Marke=ng*and*Training*
" Who*uses*our*product?*
– Page*hits*
– Browser*usage*
– Loca=on*
– Market*penetra=on*
" ATer*hos=ng*a*webinar,*do*we*see*an*increase*in*demo*hits?*
" Which*part*of*the*FAQ*receives*the*most*views?*
" Who*is*reading*the*links*in*the*“Availity*News”*sec=on?*
*
*
21*

22. 22*

23. 23*

24. Availity’s*Top*Wins*with*Splunk*
24*
Ease*of*data*
consolida=on*
From*IT*to*the*
enterprise*
From*reac=ve*to*
proac=ve*
" Find*clues*easily*with*
simple*searches*
" Transform*the*data*
to*eliminate*IT*
problems*proac=vely*
" Trace*transac=ons*
quickly*across*the*
applica=on*and**
infrastructure*stack*
" Incrementally*grow**
based*on*need*
" Mine*exis=ng*
structured*and*
unstructured*data*
" Consolidate*and*
integrate*mul=ple*
sources*easily*
without*any*prior*
knowledge*on*the*
data*
" Customized*roleL
based*views*in*less*
than*10*minutes*
" RealL=me*views*for*
our*execu=ve*
management*(CTO)**
" Viral*adop=on*across*
the*enterprise*–*over*
126*ac=ve*users*

25. What*Should*I*Do*with*This?*
" Don’t*stop*at*simple*shell*scripts.**Paint*a*picture*with*your*data.*
" Encourage*your*users*to*build*their*own*dashboards.*
" Try*to*replace*a*piece*of*monitoring*with*Splunk.**Improve*on*the**
process,*and*offer*more*then*the*consumer*originally*asked*for.*
25*

26. Ques=ons?*
26*