At Cricket Communications, Splunk started as a way to correlate all of our data into one view to help our operations team keep processes humming. Then we gave secured access to our developers, now they’re addicted. In fact, Splunk is critical in helping us speedup deployment of new systems (like our recent multi-million dollar billing system implementation). Learn how we use Splunk to display key metrics for the business, track overall system health, track transactions, optimize license usage, and support capacity planning.

1. Supporting Enterprise System Rollouts with Splunk Clint Sharp, Cricket Communications

3. Who Am I? Clint Sharp Director of Application Operations Cricket Communications Application Operations IT Operations Center (ITOC) Configuration Management ETL and BI Operations 15 Employees/50 Contractors

5. Cricket’s Architecture Service Gateway EMS Business Works Tibco Enterprise Services Bus (ESP) CID RSR CID ISR CID CSR MyCricket SOAP API Web Logic Biller Oracle Database SOAP API App Server Point of Sale Oracle DB SOAP API Data-Guard Biller Replica & Back-Office Systems

6. Trial and Rollout

8. First Use Case, In-house Middleware Performance 2011-07-28 09:21:47 server=sandapcspapl1 adaptor=APL call=ValidateActivationPayment type=Requests val=1 newval=109083 oldval=109082 2011-07-28 09:21:47 server=sandapcspapl1 adaptor=APL call=GetCustomerInformation type=ResponseTime val=1142 newval=1142 oldval=1318 2011-07-28 09:21:47 server=sandapcspapl1 adaptor=APL call=UpdateActivationPayment type=Successful val=3 newval=103334 oldval=103331 2011-07-28 09:21:47 server=sandapcspapl1 adaptor=APL call=ValidateActivationPayment type=RequestsOneMinuteCount val=1 newval=1 oldval=0 2011-07-28 09:21:47 server=sandapcspapl1 adaptor=APL call=PostPaygoPayment type=Successful val=6 newval=178006 oldval=178000

10. Expansion of Forwarders, cont’d Round 1 Splunk Server Test Middleware Server Log Server

11. Expansion of Forwarders, cont’d Round 2 Initial Deployment QA Environment Trial

12. Expansion of Forwarders, cont’d Full Deployment

13. New Use Cases

14. What Data? HTTP Logs Exception Logs CRM Logs Business Events Logs POS Logs F5 Logs Operational Statistics IT Data Business Data New Data!

16. Finding Errors

18. Web Services Operational Stats

19. Activations and Sales

20. Work Orders By Type

23. F5 VIP by Source IP

24. The Big Project (“Billing”)

25. Billing Nine-figure Cost 600+ People 2 year implementation

28. F5 VIP Tap Point Service Gateway Tibco Enterprise Services Bus (ESP) Web Logic Biller Oracle Database App Server Point of Sale Oracle DB Data-Guard Biller Replica & Back-Office Systems Tap F5 VIP EMS Business Works CID RSR CID ISR CID CSR MyCricket SOAP API SOAP API SOAP API

30. F5 Log Data Jul 29 08:28:55 10.3.0.61 DEVICE=SANPRODF5 ENV=PROD APP=IRB COMMENT=XML SRC=10.12.18.199 DST=10.3.0.22 URI=CricketAccountPort CALL=manageAccountArrangement_1 POOL=cvg-prod-irb-80 NODE=155.90.153.62 STATUS=200 RESPONSETIME=232 Jul 29 08:28:55 10.3.0.61 DEVICE=SANPRODF5 ENV=PROD APP=IRB COMMENT=XML SRC=10.12.18.197 DST=10.3.0.22 URI=CricketPaymentPort CALL=queryBucketBalances_1 POOL=cvg-prod-irb-80 NODE=155.90.153.62 STATUS=200 RESPONSETIME=87 Jul 29 08:28:55 10.3.0.61 DEVICE=SANPRODF5 ENV=PROD APP=IRB COMMENT=XML SRC=10.12.18.199 DST=10.3.0.22 URI=CricketAccountPort CALL=getBillCycleDates_1 POOL=cvg-prod-irb-80 NODE=155.90.153.62 STATUS=200 RESPONSETIME=70 Jul 29 08:28:55 10.3.0.61 DEVICE=SANPRODF5 ENV=PROD APP=IRB COMMENT=XML SRC=10.12.18.200 DST=10.3.0.22 URI=OMOrderPort CALL=searchOrder_2 POOL=cvg-prod-irb-80 NODE=155.90.153.62 STATUS=200 RESPONSETIME=75 Jul 29 08:28:55 10.3.0.61 DEVICE=SANPRODF5 ENV=PROD APP=IRB COMMENT=XML SRC=10.12.18.200 DST=10.3.0.22 URI=CricketPaymentPort CALL=queryBucketBalances_1 POOL=cvg-prod-irb-80 NODE=155.90.153.62 STATUS=200 RESPONSETIME=84 Jul 29 08:28:55 10.3.0.61 DEVICE=SANPRODF5 ENV=PROD APP=IRB COMMENT=XML SRC=10.12.18.197 DST=10.3.0.22 URI=CricketSystemPort CALL=getTimestamp_1 POOL=cvg-prod-irb-80 NODE=155.90.153.62 STATUS=200 RESPONSETIME=69 Jul 29 08:28:55 10.3.0.61 DEVICE=SANPRODF5 ENV=PROD APP=CSP3 COMMENT=TCP SRC=10.72.0.133 SRCPORT=56759 DST=10.0.4.36 DSTPORT=80 OPEN=1195 Jul 29 08:28:55 10.3.0.61 DEVICE=SANPRODF5 ENV=PROD APP=CSP3 COMMENT=XML SRC=10.72.0.133 DST=10.0.4.36 URI=/csp-contentportal/CPortal CALL=debitAccount POOL=sandapcsp-80 NODE=10.0.24.106 STATUS=500 RESPONSETIME=1094 Jul 29 08:28:55 10.3.0.61 DEVICE=SANPRODF5 ENV=PROD APP=IRB COMMENT=XML SRC=10.12.18.198 DST=10.3.0.22 URI=CricketPaymentPort CALL=queryBucketBalances_1 POOL=cvg-prod-irb-80 NODE=155.90.153.62 STATUS=200 RESPONSETIME=82 Jul 29 08:28:55 10.3.0.61 DEVICE=SANPRODF5 ENV=PROD APP=CIDISR COMMENT=TCP SRC=96.17.70.167 SRCPORT=42971 DST=10.12.0.78 DSTPORT=443 OPEN=2233 Jul 29 08:28:55 10.3.0.61 DEVICE=SANPRODF5 ENV=PROD APP=CIDISR COMMENT=TCP SRC=96.17.70.155 SRCPORT=38332 DST=10.12.0.78 DSTPORT=443 OPEN=654 Jul 29 08:28:55 10.3.0.61 DEVICE=SANPRODF5 ENV=PROD APP=IRB COMMENT=XML SRC=10.12.18.195 DST=10.3.0.22 URI=CricketSystemPort CALL=getTimestamp_1 POOL=cvg-prod-irb-80 NODE=155.90.153.62 STATUS=200 RESPONSETIME=69 Jul 29 08:28:55 10.3.0.61 DEVICE=SANPRODF5 ENV=PROD APP=IRB COMMENT=XML SRC=10.12.18.199 DST=10.3.0.22 URI=CricketPaymentPort CALL=createAccountAdjustment_1 POOL=cvg-prod-irb-80 NODE=155.90.153.62 STATUS=200 RESPONSETIME=102 Jul 29 08:28:55 10.3.0.61 DEVICE=SANPRODF5 ENV=PROD APP=IRB COMMENT=XML SRC=10.12.18.196 DST=10.3.0.22 URI=CricketAccountPort CALL=queryDetailedInfo_1 POOL=cvg-prod-irb-80 NODE=155.90.153.62 STATUS=200 RESPONSETIME=124 Jul 29 08:28:55 10.3.0.61 DEVICE=SANPRODF5 ENV=PROD APP=CSP3 COMMENT=TCP SRC=10.72.0.133 SRCPORT=56761 DST=10.0.4.36 DSTPORT=80 OPEN=1080 Jul 29 08:28:55 10.3.0.61 DEVICE=SANPRODF5 ENV=PROD APP=CSP3 COMMENT=XML SRC=10.72.0.133 DST=10.0.4.36 URI=/csp-contentportal/CPortal CALL=getAccountInfo POOL=sandapcsp-80 NODE=10.0.24.149 STATUS=200 RESPONSETIME=977 Jul 29 08:28:55 10.3.0.61 DEVICE=SANPRODF5 ENV=PROD APP=CIDISR COMMENT=TCP SRC=96.17.70.155 SRCPORT=38326 DST=10.12.0.78 DSTPORT=443 OPEN=1307 Jul 29 08:28:55 10.3.0.61 DEVICE=SANPRODF5 ENV=PROD APP=CIDISR COMMENT=TCP SRC=96.17.70.155 SRCPORT=38325 DST=10.12.0.78 DSTPORT=443 OPEN=1396 Jul 29 08:28:55 10.3.0.61 DEVICE=SANPRODF5 ENV=PROD APP=CIDISR COMMENT=TCP SRC=64.211.145.189 SRCPORT=44340 DST=10.12.0.78 DSTPORT=443 OPEN=67

31. Biller Dashboard

33. Questions? Clint Sharp, Cricket Communications