Empfohlen
Weitere ähnliche Inhalte
Ähnlich wie Getting started with splunk it service intelligence
Ähnlich wie Getting started with splunk it service intelligence (20)
Kürzlich hochgeladen
Kürzlich hochgeladen (20)
Getting started with splunk it service intelligence
- 1. Copyright © 2015 Splunk Inc. The Hands-On Version Burch Systems Engineer, IT Operations Analytics SME
- 2. Setup Before You Can Play Download this presentation slide deck: https://splunk.box.com/splunkliveitsi16 Follow the instructions on your paper hand-out to log in to your VM. Please log in as either • user1@buttercupgms.com OR • user2@buttercupgms.com • Password is “Changeme1” or “Changeme2” After logging in, select IT Service Intelligence from the list of apps at the left 2
- 4. Experience with ITSI? Splunk? 4
- 5. ITSI – The Burch Version 5
- 7. ITSI – The Burch Version 7
- 8. ITSI – The Texbook Version 8
- 9. What is a Service? Service Requests Responses In ITSI, a Service is a logical group of technology components that a user deems need to be monitored together. It can often be generalized as a “black box” which we send requests, and expect responses 9
- 10. What is a Service? DNS Requests Responses Technical Services Auth Requests Responses Web Requests Responses Services can be lower level (technical) … 10
- 11. What is a Service? DNS Requests Responses Technical Services Customer Transactions Requests Responses Business Services Auth Requests Responses Web Requests Responses Support Desk Requests Responses Services can also be higher level (business) … 11
- 12. What is a Service? Packet Network Hypervisor and Hosts RBMDBs Storage Tier API Services Web Services CustomerTransactions Mobile API/Middleware PartnerPortal DNS Services can encompass multiple tiers of the IT domain. Services may also depend upon other services 12
- 13. What is a KPI? DNS Requests Responses KPI: Number of requests KPI: Error rate KPI: Average response time KPI: Server CPU load KPI: Server network I/F errors Customer Transactions Requests Responses KPI: Number of transactions KPI: Error rate KPI: Average response time KPI: Count of Incident Tickets KPI: Synthetic Transx Health KPIs and Health scores constitute the means by which Services are monitored. 13
- 14. Key Performance Indicators (KPIs) 14 A Key Performance Indicator (KPI) is a Splunk saved search created within the ITSI UI that helps monitor a specific field like CPU, Memory, Number of Errors and so on. KPIs are contained within Services.
- 15. Service Health Scores 15 A Health score is a score form 0-100 (0 being critical and 100 being normal) that helps determine the health of a Service. It is calculated based on all KPIs importance and its status (e.g. green, orange, red), once every minute.
- 16. Now in ITSI 16
- 18. New Requirements! 28 ● Create a new KPI for the DB Service: ● Network Utilization ● Modify the Executive Glass Table in order to show off the services you slave over “WE only have about 15min TO DO WHAT ???!!???” Think about how long this would take you today?
- 19. 29 Configuration of DB Service Click Configure > Click Services
- 20. Let’s Talk Entities 30 ● Select DB Service ● Entities are the relevant things which support this service (usually hosts) ● Select the right entries with filters, ANDs, ORs ● Original Entity list can come from CMDB, spreadsheet, Splunk search, others
- 21. A KPI in 5 minutes? Absolutely! 31 Click New – Generic KPI Select Data Model ● Host Operating System ● Network ● # bytes ● Next
- 22. KPIs Continued…. 32 Splunk Builds Searches for you – Oh Yeah, that’s happening ● Select Yes for Split by & Filter options ● Select host for Entity Lookup & Alias options ● Click Next
- 23. Almost There… 33 Select ● KPI Search Schedule: Every Minute ● Entity Calculation: Average ● Service/Agg Calculation: Average ● Calculation Window: Last Minute ● Click Next ● Unit: Bps ● Click Next
- 24. Final Steps … 34 Set your thresholds: ● Aggregate (All) ● Per Entity ● Click “Add Threshold” TWICE ● Make the Neapolitan ice cream colors Yellow, Green, Yellow ● Drag the sliders around in order to get the current data graph entirely inside the Green (normal) band ● Click Finish ● Other options are also available, including adaptive thresholds and anomaly detection
- 25. Name that KPI! 35 From the list of KPIs, select your new one (at the bottom) ● Click on the little pencil next to the name ● Call it “Network Utilization”, with your username up front ● Click on Save at bottom right when finished!
- 26. Adaptive Thresholds 36 What if your KPI data looks like this?
- 27. 37 Adaptive Thresholds Static thresholds will not work…
- 28. 38 Adaptive Thresholds Adaptive Thresholding works beautifully with cyclical (and other dynamic) data
- 29. Anomaly Detection 39 ● Machine Learning ● Works well for data with patterns ● Requires some “training” (trial & error) to zero in on best sensitivity ● More sophisticated capabilities coming! (multivariate, more algorithms, etc)
- 30. Let’s Fix that Glass Table 40
- 31. Clone the Glass Table 41 Return to Saved Glass Tables page (click on Glass Tables in the upper menu bar) CLICK Edit for “Buttercup Games Business Process (IN PROGRESS)” • Select Clone • Title: Add your username to the front • Permissions: Shared in App • Click Clone Page • Click on your new Glass Table from the list, to view it
- 32. Edit & Have Fun! 42 Click on Edit in the upper right corner of your Glass Table Use the “Services” panel on the left to select Individual KPIs, or Aggregate Service Health Scores • Choose 2 KPIs from Online Store that would be useful in the “Order Process” section • Drag the selected widgets onto the canvas, positioning in the gray oval • What’s the difference between the and tools at the top left?
- 33. More Fun with the Glass Table Editor… 43 Use the Configurations panel on the right to edit a selected widget • Can change the visualization type, drilldown behavior, and other settings • You should hit Save frequently • I wonder what Auto Layout does? • (YIKES!) Revert All Changes might be helpful
- 34. Finishing up … 44 • Add a ServiceHealthScore widget for Online Store under Buttercup • Choose a Viz Type with a sparkline graph, then resize to make it look pretty • Modify the Custom Drilldown action to go to the saved glass table, Buttercup Games Online Store • Bonus Points: Make the label bigger, more readable • Click Save • View when done
- 35. A Troubleshooting Exercise 45 Let’s use ITSI to troubleshoot an outage ● Start at your Glass Table, “<UserName> Buttercup Business Process” ● Customer Care reports that unhappy customers are complaining of failures and long delays when trying to purchase ● The calls began coming in at around ten minutes after the hour. ● In the upper right corner of the Glass Table, change the time picker from Now to XX:10:00.0, where XX is the appropriate hour. For example, if it is currently 14:05, set the time picker to 13:10:00.0, then Apply ● This is how we can “time travel” back to see conditions at a particular outage– oh yeah!
- 36. A Troubleshooting Exercise, cont’d 46 ● The Online Store seems to be degraded, just as Customer Care reported. Click on the widget under Buttercup to drill down further
- 37. A Troubleshooting Exercise, cont’d. 47 ● The Online Store Glass Table shows a much more detailed view, including the impacted customer-facing KPIs at the far left (Revenue, etc) ● Based on this view of all the relevant services, where do you think the root cause lies? ● Which service should we troubleshoot first? ● Click on Health widget for that service, to drill down to a Deep Dive
- 38. Deep Dive 48 ● Deep Dive shows multiple KPIs and Health Scores in parallel “swim lanes”. ● The Health Score for this Service is the top swim lane. Can you see when it begins to degrade from 100%? ● Mousing over this point in time, can you spot the KPI with the leading fault indication, i.e., what failed first?
- 39. Review 51 ● High-value services can be decomposed and modeled in ITSI, using machine data from the relevant systems ● Services and KPIs can be created in minutes, with sophisticated thresholding techniques to distinguish “normal” from “not normal” ● Glass Tables allow service health and KPI metrics to be displayed in a way that makes sense to specific groups, such as Executive Leadership, Business Service Owners, the NOC, DevOps & Others ● Deep Dives allow KPIs to be compared side-by-side across any time range, accelerating root cause analysis and significantly reducing MTTR ● Multi-KPI Alerts and Notable Events reduce alert noise, producing actionable events and a means to manage them ● … and it’s fun to build!
- 40. PLAY TIME IS OVER! Everyone out of the sandbox! 52 NOT! You can have your very own 7-day free eval sandbox, to continue playing: ● http://splunk.com/ITSI Then select: And a Guidebook to help you explore ITSI’s capabilities: ● https://splunk.box.com/ITSI-Sandbox-Guidebook
- 41. 53 SEPT 26-29, 2016 WALT DISNEY WORLD, ORLANDO SWAN AND DOLPHIN RESORTS • 5000+ IT & Business Professionals • 3 days of technical content • 165+ sessions • 80+ Customer Speakers • 35+ Apps in Splunk Apps Showcase • 75+ Technology Partners • 1:1 networking: Ask The Experts and Security Experts, Birds of a Feather and Chalk Talks • NEW hands-on labs! • Expanded show floor, Dashboards Control Room & Clinic, and MORE! The 7th Annual Splunk Worldwide Users’ Conference PLUS Splunk University • Three days: Sept 24-26, 2016 • Get Splunk Certified for FREE! • Get CPE credits for CISSP, CAP, SSCP • Save thousands on Splunk education!
- 42. Aflying start to Service Intelligence Start With A problem worth solving Collaborate with Subject Matter Experts Design Before Configuring
- 43. Sign Up Here - We’re Here To Help! Harness the creativity and domain knowledge of your organization to unlock the value of data and solve an important service problem through a joint service intelligence workshop with key stakeholders Define methods for: • Proactive service monitoring • Reduced risk and failures • Faster issue resolution • Increased business performance What is it? • 1 Day Onsite Workshop • Tightly linked with value • Collaborative approach • Build your own Splunk ITSI Glass Table……
- 44. Copyright © 2015 Splunk Inc. Thank You Burch Systems Engineer, IT Operations Analytics SME
Hinweis der Redaktion
- Check in with Stuart Fill in IP sheets Practice screen switching and maybe use the PDF Open These: Service Analyzer Buttercup Games Business Process (IN PROGRESS) Buttercup Games Online Store DB Deep Dive Multi KPI Alert Notable Event Review
- “Service Decomposition” = Show my ITSI workflow
- For a given sub-service, such as “Database”, what are some useful KPIs which would describe its health, status and performance? These KPI metrics are based on Splunk searches, so they can be almost anything. Be creative!
- For a particular KPI, what is the Splunk search to generate the KPI metrics? The example here could be used for the Database KPI, “DB errors”.
- In the “real world”, it will probably be necessary to iterate up & down these steps a few times. For example, what if a KPI requires data which is not being collected by Splunk?
- TO STUDENTS: You have this glass table on your own system. This Glass Table shows the high-level business process for Buttercup Games. Does anyone notice anything missing? (no info in Order Entry) We need better visibility into our Online Store, which is part of the Order Entry process.
- TO STUDENTS: You have this glass table on your own system. This Glass Table shows a more detailed process flow for the Online Store service. Notice the sub-services which make up our Online Store service, and how the process flows.
- Show the slides, then do it together
- Generated Search
- Keep moving…
- FOR THE PRESENTER: This might take a while for “waiting for data” to produce an actual graph for the students (1-2 minutes, typically). Instruct the students that if will take a couple of minutes for the data to appear, and to not click on anything in the meantime. Then skip to the Adaptive Thresholds and Anomaly Detection slides and discussion, while the students wait. Afterwards, can be helpful to gauge progress by asking for a show of hands to see how many students are still waiting. If necessary, simply show the students how to set thresholds (on your own browser), then move forward.
- How long did it take to make this KPI?
- FOR THE PRESENTER: Talk through-- NOT HANDS ON
- FOR THE PRESENTER: Talk through-- NOT HANDS ON
- FOR THE PRESENTER: Talk through-- NOT HANDS ON
- Talk through NOT WORK
- We’ve already discussed the high-level business process for Buttercup Games. We need better visibility into our Online Store, which is part of the Order Entry process.
- FOR THE PRESENTER: As before, switch to your own browser and demonstrate these steps “live”. Have fun with the concept of saving a copy before editing– so that you don’t muck it up.
- FOR THE PRESENTER: Have fun with this GT editor section. The GT editor is a bit twitchy, so exploit the humor and have fun with the students. GOALS (for the next 3 slides): Identify 2 “interesting/useful” KPIs from the Online Store service, to position in the gray “Order Entry” oval; let the students choose details and viz types Put a ServiceHealthScore widget (from Online Store) under the pony, to show overall health of the service. Modify “custom drilldown” to land on the “Buttercup Games Online Store” GT Encourage the students to use text boxes and other techniques to make the widget more readable, prettier to look at Remind the students that “the boss’ boss” will be looking at this GT, and we want to make sure that they’ve got good visibility into “our” service (Online Store).
- FOR THE PRESENTER: If you use the “Auto Layout” gag (i.e., hinting that the students should click on this, resulting in total destruction of their GT), MAKE SURE that everyone has SAVED before doing so. This gag can be fun, especially pointing out how deceitful/evil the instructor is.
- FOR THE PRESENTER: When finished (after everyone have hit ‘Save’ and ‘View’, and are looking at their own beautiful GTs): How long did it take to create a new KPI and make major changes to a Glass Table? Pretty cool! Ask the students if this (ITSI) could be useful in their own environments If you have more than 15 min of remaining time, speak through some actual (referenceable) customer ITSI use cases.
- FOR THE PRESENTER: This hands-on section can be very powerful for the students. This allows them to “put it all together”, driving ITSI with their own fingers. NOTE: PRESENTER will need to determine which hour had the DB failure scenario in advance and tell students to change to the appropriate hour. If you’re using your local laptop, your datagen may not be in sync with the students’ instances, so you may want to check before the presentation. If the students are seeing the wrong failure scenario, have them change to the previous hour and they should be fine. As before, switch to your own browser and demonstrate these troubleshooting steps “live”. The corresponding slides are intended as reference for the students. If pressed for time (i.e., less than about 10 min), talk through and show this process– but don’t have the students attempt to “click along” in real time.
- If pressed for time, talk through and show this process– but don’t have the students attempt to “click along” in real time
- Note that this “drill down” has inherited the same time selection (i.e., an earlier outage)– pretty cool! FOR THE PRESENTER: The major points here: During the heat of battle, when troubleshooting an outage, being able to visualize the entire service flow is extremely valuable By being able to see health status of all the underlying services, we can quickly choose where and how best to proceed. Potentially huge time savings– customers report major reductions in MTTR
- FOR THE PRESENTER: This is a good “variable time” section. You can spend as little or as much time as you choose, depending on how much time you have remaining in the session. Remind the students that they will have more time to play with DD later (yes, they might be confused by this, since only a few minutes remain in the session) If there’s issues with students not able to see the outage, have them change the time range to the last 2 hours – there will be at least one issue in there.
- FOR THE PRESENTER: This is another good “variable time” section. You can spend as little or as much time as you choose, depending on how much time you have remaining in the session. Remind the students that they will have more time to play with this stuff later (yes, they might be confused by this, since only a few minutes remain in the session)
- Look! Students have more time to play in their own sandbox environment, after all.
- We’re headed to the East Coast! 2 inspired Keynotes – General Session and Security Keynote + Super Sessions with Splunk Leadership in Cloud, IT Ops, Security and Business Analytics! 165+ Breakout sessions addressing all areas and levels of Operational Intelligence – IT, Business Analytics, Mobile, Cloud, IoT, Security…and MORE! 30+ hours of invaluable networking time with industry thought leaders, technologists, and other Splunk Ninjas and Champions waiting to share their business wins with you! Join the 50%+ of Fortune 100 companies who attended .conf2015 to get hands on with Splunk. You’ll be surrounded by thousands of other like-minded individuals who are ready to share exciting and cutting edge use cases and best practices. You can also deep dive on all things Splunk products together with your favorite Splunkers. Head back to your company with both practical and inspired new uses for Splunk, ready to unlock the unimaginable power of your data! Arrive in Orlando a Splunk user, leave Orlando a Splunk Ninja! REGISTRATION OPENS IN MARCH 2016 – STAY TUNED FOR NEWS ON OUR BEST REGISTRATION RATES – COMING SOON!
- This workshop requires that the presenter be able to deftly toggle quickly between slides and browser– often. This is even trickier when using “full screen” mode for the browser and slides. For this reason, I recommend using a PDF version of the slides (rather than PowerPoint), since Acrobat is simpler to operate, especially when connected to an external projector. However you choose to display the slides and browser stuff, you should probably practice toggling quickly between the two. You should have a timer visible (to you only), counting down from 60 min, to help with pacing.