Splunk Webinar: Maschinendaten anreichern mit Informationen
•
0 gefällt mir•1,113 views
Maschinendaten anreichern mit Informationen mit Splunk Enterprise
Empfohlen
Empfohlen
Weitere ähnliche Inhalte
Andere mochten auch
Andere mochten auch (16)
Ähnlich wie Splunk Webinar: Maschinendaten anreichern mit Informationen
Ähnlich wie Splunk Webinar: Maschinendaten anreichern mit Informationen (20)
Mehr von Georg Knon
Mehr von Georg Knon (20)
Splunk Webinar: Maschinendaten anreichern mit Informationen
- 1. 1 Copyright © 2014 Splunk Inc. Maschinendaten anreichern mit Informationen Philipp Drieger Sales Engineer
- 2. 2 Splunk Webinar Maschinendaten anreichern mit Information Ihr Ansprechpartner: Philipp Drieger Sales Engineer philipp@splunk.com
- 3. 3 Disclaimer During the course of this presentation, we may make forward looking statements regarding future events or the expected performance of the company. We caution you that such statements reflect our current expectations and estimates based on factors currently known to us and that actual events or results could differ materially. For important factors that may cause actual results to differ from those contained in our forward-looking statements, please review our filings with the SEC. The forward-looking statements made in the this presentation are being made as of the time and date of its live presentation. If reviewed after its live presentation, this presentation may not contain current or accurate information. We do not assume any obligation to update any forward looking statements we may make. In addition, any information about our roadmap outlines our general product direction and is subject to change at any time without notice. It is for informational purposes only and shall not, be incorporated into any contract or other commitment. Splunk undertakes no obligation either to develop the features or functionality described or to include any such feature or functionality in a future release.
- 4. 4 Agenda Splunk für Maschinendaten Methoden zur Anreicherung Live: CSV | KVStore | Script | DBX | ODBC Q&A
- 5. 5 Make machine data accessible, usable and valuable to everyone.
- 6. 6 Das beschleunigte Wachstum von Daten Volume | Velocity | Variety | Variability GPS, RFID, Hypervisor, Web Servers, Email, Messaging, Clickstreams, Mobile, Telephony, IVR, Databases, Sensors, Telematics, Storage, Servers, Security Devices, Desktops Machinendaten umfassen den am schellsten wachsenden, komplexesten und wertvollsten Bereich von Big Data 6
- 7. 7 Machinendaten enthalten wertvolle Informationen Order ID Customer’s Tweet Time Waiting On Hold Product ID Company’s Twitter ID Order ID Customer ID Twitter ID Customer ID Customer ID Sources Order Processing Twitter Care IVR Middleware Error
- 8. 8 Industry Leading Platform For Machine Data Machine Data: Any Location, Type, Volume Online Services Web Services Servers Security GPS Location Storage Desktops Networks Packaged Applications Custom ApplicationsMessaging Telecoms Online Shopping Cart Web Clickstreams Databases Energy Meters Call Detail Records Smartphones and Devices RFID On- Premises Private Cloud Public Cloud Platform Support (Apps / API / SDKs) Enterprise Scalability Universal Indexing Answer Any Question Developer Platform Report and analyze Custom dashboards Monitor and alert Ad hoc search Universal Machine Data Platform
- 9. 9 Industry Leading Platform For Machine Data Machine Data: Any Location, Type, Volume Online Services Web Services Servers Security GPS Location Storage Desktops Networks Packaged Applications Custom ApplicationsMessaging Telecoms Online Shopping Cart Web Clickstreams Databases Energy Meters Call Detail Records Smartphones and Devices RFID On- Premises Private Cloud Public Cloud Platform Support (Apps / API / SDKs) Enterprise Scalability Universal Indexing Answer Any Question Developer Platform Report and analyze Custom dashboards Monitor and alert Ad hoc search Any amount, any location, any source Schema- on-the-fly Universal indexing No back-end RDBMS No need to filter data Schema on the Fly
- 11. 11 Anreicherung von Events in Splunk Erweiterung der raw events um zusätzliche Felder, welche aus externen Datenquellen stammen. LDAP, AD Watch Lists CRM/ERP CMDB Externe Datenquellen Insight OUT Data IN …
- 12. 12 Anreicherung mit Lookups Durch Lookups können in Splunk Maschinendaten mit zusätzlichen Informationen angereichert werden. Es wird dabei ein Mapping von Feldwerten in Events auf Feldwerte einer externen Datenquelle realisiert und neue Werte den Eventdaten zugefügt. Beispiel: Lookup von HTTP Status Codes in einem CSV File mit der entsprechenden Beschreibung des Codes.
- 13. 13 Mehrwerte durch Lookups Darstellung von Maschinendaten in der Sprache der Fachabteilungen Differenziertere Analysen und Aufteilungen von Auswertungen – z.B. Monitoring von Manager User Accounts, HR, Finance, IT Verlinkung von Maschinendaten zu geschäftsrelevanten Prozessen. Z.B. Anreicherung von Bestelldaten mit Artikellisten inklusive Beschreibung, Verfügbarkeit, Preis etc. Integration von SAP Bestandsdaten CRM Daten Produktinfos Preislisten WHOIS Geolocation Zip codes
- 14. 14 Übersicht: Methoden für Lookups in Splunk ODBC driver (MS Excel, Tableau, …) CSV File Lookup Script (Python, Perl, shell, …) DB Connect (DB2, Oracle, MySQL, …) KVStore (Key Value Store)
- 15. 15 Übersicht: Methoden für Lookups in Splunk CSV File KVStore Script DB Connect ODBC Indexer / Search Head Search Head ONLY Indexer / Search Head Indexer / Search Head Indexer / Search Head Statisch Dynamisch Dynamisch Dynamisch Dynamisch DEMO DEMO DEMO DEMO OVERVIEW
- 17. 17 CSV Lookup Setup Lookup Table Files
- 18. 18 CSV Lookup Setup Lookup Definitions
- 19. 19 CSV Lookup And: Look IT up
- 20. 20 CSV Lookup Ressourcen Dokumentation: http://docs.splunk.com/Documentation/Splunk/latest/Knowledge/Addfi eldsfromexternaldatasources Einführung zur Einrichtung von Lookups mit Beispiel HTTP codes: http://docs.splunk.com/Documentation/Splunk/latest/Knowledge/Usefi eldlookupstoaddinformationtoyourevents Blog Artikel: Enriching Data with Lookups (Part 1-2): http://blogs.splunk.com/2009/07/27/enriching-data-with-lookups-part- 1/
- 22. 22 KV Store Lookup Setup /splunk/etc/apps/<appname>/local
- 23. 23 KV Store Lookup Bring your data in:
- 24. 24 KV Store Lookup Ressourcen Dokumentation: http://docs.splunk.com/Documentation/Splunk/latest/Knowledge/Confi gureKVstorelookups Migration von Lookups mit CSV Files zu KV Store: http://dev.splunk.com/view/webframework-features/SP-CAAAEZQ Beispiele KV Store: http://dev.splunk.com/view/webframework- features/SP-CAAAEZH Konfiguration KV Store über REST API: http://dev.splunk.com/view/webframework-features/SP-CAAAEZG
- 26. 26 Scripted Lookup Example DNSLOOKUP (external_lookup.py)
- 27. 27 Scripted Lookup Example DNSLOOKUP (external_lookup.py) Usage lookup hostname: … | lookup dnslookup clientip as ip Usage lookup ip: … | lookup dnslookup clienthost as domain
- 28. 28 Demo #4 DB Connect (MySQL)
- 29. 29 Lookup mit DB Connect: MySQL
- 30. 30 Lookup mit DB Connect: Identity
- 31. 31 Lookup mit DB Connect: Connection
- 32. 32 Lookup mit DB Connect: Test Query
- 33. 33 Lookup mit DB Connect Ressourcen Splunk App for DB Connect (V 2.0): https://splunkbase.splunk.com/app/2686 Dokumentation DB Connect: http://docs.splunk.com/Documentation/DBX/latest/DeployDBX/AboutS plunkDBConnect
- 34. 34 Overview ODBC
- 35. 35 Übersicht Splunk ODBC (Windows) Analyst Splunk admin Saved Searches ODBC driver (SQL to SPL translation layer) STEP 1 Business user and Admin work together to define and build the saved search in Splunk. 1 STEP 1 Business user uses tool to access saved search and retrieve data from Splunk. 2
- 36. 36 ODBC Treiber für Splunk Enterprise Download Splunk App: https://splunkbase.splunk.com/app/1606 Documentation ODBC: http://docs.splunk.com/Documentation/ODBC On Windows OS: Splunk ODBC Driver + Tool of choice Splunk Enterprise auf jeder unterstützten Platform: - Windows - Linux - OS X - Solaris
- 38. 38 Zusammenfassung Maschinendaten enthalten wertvolle Informationen Durch Anreicherung mit externen Datenquellen können diese explizit gemacht werden Splunk bietet viele Methoden für Lookups: – CSV Files – KV Store – Script – DB Connect – ODBC Machen Sie Ihre Maschinendaten durch Lookups verständlicher für Ihre Nutzer!
- 39. 39 Vielen Dank! Q&A: Bitte den Chat benutzen Philipp Drieger Sales Engineer
- 40. 40 Thank You
Hinweis der Redaktion
- The Splunk Enterprise Technical Overview
- This presentation covers 4 key areas about our technology and how it is typically used.
- As a company our mission is to make machine data accessible, usable and valuable to everyone. This overarching mission is what drives our company and product priorities.
- Data is growing and embodies new characteristics not found in traditional structured data: Volume, Velocity, Variety, Variability. Machine data is one of the fastest, growing, most complex and most valuable segments of big data. "Big data" is a term applied to these expanding data sets whose size is beyond the ability of commonly used software tools to capture, manage, and process the data within a tolerable elapsed time. All the webservers, applications, network devices – all of the technology infrastructure running an enterprise or organization – generates massive streams of data, in an array of unpredictable formats that are difficult to process and analyze by traditional methods or in a timely manner. Why is this “machine data” valuable? Because it contains a trace - a categorical record - of user behavior, cyber-security risks, application behavior, service levels, fraudulent activity and customer experience.
- To frame our discussion, let’s use this example of purchasing a product from your tablet or smartphone: the purchase transaction fails, you call the call center and then tweet about your experience. All these events are captured - as they occur - in the machine data. Each of the underlying systems hast the potential to generate millions of machine data events daily. Here we see small excerpts from just some of them. When we look more closely at the data we see that it contains valuable information – right down to what was tweeted. What’s important, is first of all, the ability to actually see across all these data sources, but then also to correlate related events and provide meaningful insight. If you can correlate and visualize the data, you can build a picture of activity, behavior and experience. And what if you can do all of this in real-time? You can respond more quickly to events that matter. This example ties into your scenario but you can also extrapolate this example to a wide range of use cases – security and fraud, transaction monitoring and analysis, web analytics, IT operations and so on.
- One of of the key differentiators of Splunk is the ability to digest all machine data and allow users to quickly analyze it for insight. We call this the universal machine data platform. We’ll look at this in more detail in a bit, but for now, understand that the platform was designed around the premise of being able to consume any machine data even if the format changes; something a relational database cannot do. (Splunk Cloud is only available in the U.S. and Canada.)
- Splunk is able to do this because there’s no requirement to “understand” the data upfront – this is one of our key differentiators that we call “schema on the fly”. Simply point Splunk at the data or deploy Splunk forwarders to stream data from remote systems. Splunk immediately starts collecting and indexing, so users can start searching and analyzing. No more armies of consultants, backend database or DBA to make it work. Once you’ve Splunked your data, it is time-stamped and easily searchable. Because we don’t have to do all the up front work to be able to look at the data we can load it all and make it all relevant. There’s no need to limit what you load and what you don’t.
- Now that we understand the high level question of What is Splunk Enterprise, let’s talk about how the technology can be deployed and integrated into your existing environment.
- The data for example may have a userid but you want to search on a name. Splunk’s lookup capability can enrich the raw data by adding additional fields at search time. Some common use cases including event and error code description fields. Think about how much easier it would be if you could see“Page not Found” instead of the error code “404” in the search results. Enriching your data can lead to entirely new insight. In the example shown, Splunk took the userid and looked up the name and role of the user from an HR database. Similarly, it determined the location of the failed log in attempt by correlating the IP address. Even though these fields don’t exist in the raw data, Splunk allows you to search or pivot on them at any time. You can also mask data. For example, you may want social security numbers to be replaced with all X’s for regular users but not masked for others. Removing data can also be useful, such as filtering PII, before writing it to an index in Splunk.
- Now, let’s talk about how Splunk is used for real-time searching, alerting, and reporting.
- Now, let’s talk about how Splunk is used for real-time searching, alerting, and reporting.
- Now, let’s talk about how Splunk is used for real-time searching, alerting, and reporting.
- Now, let’s talk about how Splunk is used for real-time searching, alerting, and reporting.
- Now, let’s talk about how Splunk is used for real-time searching, alerting, and reporting.
- For customers who like to run their business analytics in tools such as Microsoft Excel and Tableau Desktop and want to retrieve data from Splunk, the Splunk ODBC Driver lets you interact with, manipulate and visualize machine data stored in Splunk Enterprise using existing business software tools. This flexibility gives you the features available in Excel or Tableau Desktop as well as the advanced analytics capabilities of Splunk Enterprise. Splunk Administrators need to create saved searches once. Business users then use a tool they are already familiar with to access those saved searches. Time savings and increased productivity are benefits everyone experiences.
- Now, let’s talk about how Splunk is used for real-time searching, alerting, and reporting.
- I want to conclude by reinforcing what makes Splunk unique. (1) Splunk is a universal machine data platform. It can ingest any data from any source. It’s open, extensible platform delivers integrated, end-to-end data collection, management and analysis. (2) Splunk’s Real Time Architecture provides real-time data collection from thousands of heterogeneous sources – physical, virtual and cloud (3) We also allow you to use search-time schemas, which deliver flexibility to interact with the data and change perspective on the fly at search time. (4) Splunk offers agile reporting and analytics - interactive search and reporting enables rapid, iterative analysis and visualization of data that IT AND business users can use. (5) Our flexible data engine scales from desktop to enterprise deployments. Customers can index terabytes of data per day and permits thousands of users to concurrently search petabytes of data. (6) Splunk has a Fast Time to Value. You can get productive quickly. Deployments take hours or days, not weeks or months. And Splunk is easy to use and learn. (7) Perhaps in the future, you too might join in the discussions and development pursuits of Splunk’s passionate and vibrant community.
- Now, let’s talk about how Splunk is used for real-time searching, alerting, and reporting.