SplunkLive! New York April 2013 - Enrich Machine Data with Structured Data
•
5 gefällt mir•3,113 views
Empfohlen
Empfohlen
Weitere ähnliche Inhalte
Was ist angesagt?
Was ist angesagt? (20)
Ähnlich wie SplunkLive! New York April 2013 - Enrich Machine Data with Structured Data
Ähnlich wie SplunkLive! New York April 2013 - Enrich Machine Data with Structured Data (20)
Mehr von Splunk
Mehr von Splunk (20)
Kürzlich hochgeladen
Kürzlich hochgeladen (20)
SplunkLive! New York April 2013 - Enrich Machine Data with Structured Data
- 1. Copyright © 2013 Splunk Inc. Tapan Bhatt / Raanan Dagan May 2013 Splunk DB Connect: Enrich Machine Data with Structured Data
- 2. Agenda Background and Overview DB Connect Demo Technical Overview Customer Examples & Summary Questions
- 3. Splunk: the Platform for Machine Data 3 Real-time Business Insights Operational Visibility Proactive Monitoring Search and Investigation Operational IntelligenceMachine Data
- 4. What about Structured Data? 4 Customer Profile Product Attributes Employee Details Pricing & Rate Plans Asset Info
- 5. Machine Data – Delivers Real-time Insights 5 Media Server Logs (Machine Data) Mar 01 19:18:50:000 aaa2 radiusd[12548]:[ID 959576 local1.info] INFO RADOP(13) acct start for 2172618992@splunktel.com 10.164.232.181 from 12.130.60.5 recorded OK. 2013-03-01 19:18:50:150 10.2.1.34 GET /sync/addtolibrary/01011207201000005652000000000053 - 80 - 10.164.232.181 "Mozilla/5.0 (iPhone; CPU iPhone OS 5_0_1 like Mac OS X) AppleWebKit/534.46 (KHTML, like Gecko) Version/5.1 Mobile/9A405 Safari/7534.48.3" 503 0 0 825 1680 Mar 01 19:18:50:163 aaa2 radiusd[12548]:[ID 959576 local1.info] INFO RADOP(13) acct stop for 2172618992@splunktel.com 10.164.232.181 from 12.130.60.5 recorded OK. Phone Number IP Address Track ID
- 6. Structured Data – Contains Business Context 6 Media Server Logs (Machine Data) Mar 01 19:18:50:000 aaa2 radiusd[12548]:[ID 959576 local1.info] INFO RADOP(13) acct start for 2172618992@splunktel.com 10.164.232.181 from 12.130.60.5 recorded OK. 2013-03-01 19:18:50:150 10.2.1.34 GET /sync/addtolibrary/01011207201000005652000000000053 - 80 - 10.164.232.181 "Mozilla/5.0 (iPhone; CPU iPhone OS 5_0_1 like Mac OS X) AppleWebKit/534.46 (KHTML, like Gecko) Version/5.1 Mobile/9A405 Safari/7534.48.3" 503 0 0 825 1680 Mar 01 19:18:50:163 aaa2 radiusd[12548]:[ID 959576 local1.info] INFO RADOP(13) acct stop for 2172618992@splunktel.com 10.164.232.181 from 12.130.60.5 recorded OK. Track ID Artist Title Format ID Run time 01011207201000005652000000000053 Maroon 5 Moves like Jagger MP3 4:30 Phone # Subscriber ID 2172618992 53546 Subscriber ID First Name Last Name Age State Customer Score 53546 Jim Morrison 25 CA 93 Customer, Product Databases Phone Number IP Address Track ID
- 7. Operational Dashboards with Business Context 7 Top TracksUser Activity Click to investigate Customer experience Download Errors by device
- 8. Enrich Machine Data with Structured Data 8 Structured Databases CSV Lookup DB Connect Launched March 2013
- 9. Introducing Splunk DB Connect Enrich search results with additional business context Easily import data into Splunk for deeper analysis Integrate multiple DBs concurrently Simple set-up, non-evasive and secure Reliable, scalable, real-time integration between Splunk and traditional relational databases Microsoft SQL Server JDBC Database Lookup Database Query Connection Pooling Other Databases Oracle Database Java Bridge Server 9
- 10. Delivering Operational Intelligence 10 IT Operations Analytics > Machine Data Application logs, monitoring data, disk utilization Operational Intelligence Security AnalyticsFirewall logs, Radius logs, Nessus vulnerability Critical assets, watch- lists, privileged user lists, black-lists, device data > CMDB, asset inventory, topology, user, c ost and department information Structured Data Business Analytics Device activation, Radius, application logs Rate plans, customer profile, geo location
- 12. Splunk DB Connect Technical Overview
- 13. Splunk DB Connect: Main Features Database Connection Management SQL Database Lookups Splunk Search Language extensions – Database Query – Database Info SQL Database Input 13
- 14. Installing Splunk DB Connect Simple app setup, no configuration files to touch Automatically checks for the required Java version 14
- 15. Database Connection Management Configure new database connection settings in minutes from the Splunk user interface 15
- 16. Microsoft SQL Server JDBC Database Lookup Database Query Connection Pooling Other Databases Oracle Database Java Bridge Server Works with Many Databases Supports mainstream databases – Oracle Database – Microsoft SQL Server – MySQL – PostgreSQL – Sybase – Generic JDBC support Database connection pooling limits load on Database 16
- 17. Database Lookups 17 Enrich machine data by adding structured data from traditional relational databases
- 18. Three Steps to Enriching Machine Data 18 1. Connect 2. Configure 3. Enrich
- 19. Splunk Search Language Extensions Execute database queries directly from the Splunk user interface with new Dbquery and Dbinfo Splunk search commands 19 *** DBoutput (BETA) - Create or Update database records on information Splunk searches
- 20. Explore Database Structure Wrapping dbinfo and dbquery 20 Browse and navigate database schemas and tables from the Splunk DB Connect user interface
- 21. Import and Index Database Data Combine machine data with structured data from relational databases 21 New dbmon-tail and dbmon-dump input types can be used to import rows from the database
- 22. Technical Summary Quick to set-up, scales to multiple concurrent databases Enrich machine data with database data in three easy steps Execute SQL queries to visualize database data directly in the Splunk user interface Import and index database data for historical analysis and correlation with machine data 22
- 23. Success Stories
- 24. Powering Search Analytics 24 Understanding customer usage Client Name, Country, Email Feed ID SQL SQL SQL = Client Databases contain Client Name, Country and Email information Database + Machine Data = Search Activity tracked by Feed ID
- 25. Enabling Exceptional Customer Service 25 Users to Customers mapping SQL SQL User Activity = Customer details, external/internal details Database + Machine Data = User activity data from SaaS application, websites SaaS Real-time visibility of customer experience Website
- 26. Powerful Connectivity Drives Better Insights 26 Developer Platform Report and analyze Custom dashboards Monitor and alert Ad hoc search SQL Splunk Hadoop Connect • Reliable bi-directional integration to Hadoop Splunk DB Connect • Real-time integration to relational DBs Splunk Dev Platform • API and SDKs to build Big Data apps
- 27. Summary Machine data contains a categorical record of activity and behavior Enrich with structured data to provide business context – for better IT, security and business insights Splunk DB Connect delivers reliable, scalable, real-time integration between Splunk and traditional relational databases 27
Hinweis der Redaktion
- Splunk Enterprise is the platform for machine data. It reliably collects and indexes all the streaming data from IT systems and technology devices in real-time - tens of thousands of sources in unpredictable formats and types. Splunk software is optimized for real-time, low latency and interactivity.Once in Splunk, your data is available for searching, monitoring, analysis and visualizations. With Splunk you can gain new levels of visibility and insight. This is called Operational Intelligence.Use Splunk to:1. Find and fix problems dramatically faster2. Automatically monitor to identify issues, problems and attacks3. Gain end-to-end visibility to track and deliver on IT KPIs and make better-informed IT decisions4. Gain real-time insight from operational data to make better-informed business decisions
- Splunk software has proven uses for IT, security and business users.These users can meaningfully improve their performance in a wide range of areas e.g. meet service levels, reduce costs, mitigate security risks, maintain compliance and gain insights.
- None of the customer's identifying information is in the logs (which I will add to the dashboard); we simply identified a phone that has had problems downloading music, and enriched that information with data from our customers database.
- None of the customer's identifying information is in the logs (which I will add to the dashboard); we simply identified a phone that has had problems downloading music, and enriched that information with data from our customers database.
- This dashboard provides an example of the value of combining machine data with structured data. Top left: unique user activity can be gained directly from the machine data logs.Right and bottom: combining with structured data makes our dashboard more meaningful:Top track titles in aggregate (and updated in real-time) – can also be used to reconcile royalty payments in this example. Download errors by a specific device (phone number) Enrich behavior - click to bring up the customer record details and all associated activity for further analysis.
- DB Connect available today – Steve Sommer and I met with CIO of Rakuten, a leading online retailer from Japan last year. Nice guy, very interested in Splunk but then he said, I want correlate all this data with my product master. Ok, how large is your product master? We have 3 millions SKUs….no way we can handle with CSV look up. But now, we can handle with DB Connect – in real-time, at scale with much easier set up & maintenance.Guys, this is a huge Game Changer….
- Splunk DB Connect delivers reliable, scalable, real-time integration between Splunk Enterprise and traditional relational databases. With Splunk DB Connect, structured data from relational databases can be easily integrated into Splunk Enterprise, driving deeper levels of operational intelligence and richer business analytics across the organization.Organizations can drive more meaningful insights for IT operations, security and business users. For example, IT operations teams can track performance, outage and usage by department, location and business entities. Security professionals can correlate machine data with critical assets and watch-lists for: incident investigations, real-time correlations and advanced threat detection using the award-winning Splunk Enterprise. Business users can analyze service levels and user experience by customer in real-time to make more informed decisions.
- To net it out, what DB Connect does is provide business context to machine data. Sales Analytics – activation data + rate plan + customer profileProduct Analytics – application logs + customer profile (what features are being used by what type of customers and when – in real-time)We talk about how with Splunk, all you need a browser and your imagination. With DB Connect, you need to take that imagination to the next level because so many amazing use cases are now at your finger tips!Learn more about this at the showcase…..
- Splunk DB Connect is simple to install and set up. Simply go to Splunkbase and download the App. It automatically checks for the required Java version.
- UI-driven, users can configure new databases in minutes.
- Splunk DB Connect is compatible with most relational databases including Oracle® Database, Microsoft® SQL Server, Sybase®, PostgreSQL, MySQL™, SQLite, H2, HyperSQL and support for Generic ODBC driver.
- With Database Lookup, users can enrich machine-generated data by adding structured data from relational databases. For example, by using Splunk Enterprise and Splunk DB Connect, key values contained in machine data can be used to reference related business data in relational databases, such as device addresses, product codes, media identifiers, etc. For example, telecom providers have the ability to combine real-time service activation data with profile data from a customer master database to understand what types of customers are purchasing what types of plans – enabling in-depth real-time sales and customer analytics not possible before.
- To summarize, we’ve made it easy to connect to new relational databases and start enriching machine data.With Splunk DB Connect, the addition of business context from structured sources delivers new value and actionable metrics for any organization.
- Splunk software lets users search and navigate their data from one place. Splunk DB Connect includes search language extensions that can be executed directly from the Splunk user interface. Dbquery and Dbinfo are Splunk search commands that enable you to execute database queries directly from the Splunk Enterprise user interface. Dbinfo fetches schema information from the database.
- Splunk software lets users search and navigate their data from one place. Splunk DB Connect includes search language extensions that can be executed directly from the Splunk user interface. Dbquery performs SQL queries and presents the results as Splunk visualizations. For example, dbquery database=ASSETDB “SELECT hostname, owner, department from host_information WHERE location LIKE ‘%NY%.
- Import and Index Data from Relational Databases into Splunk. Combine business/structured data from relational databases with machine data to drive end-to-end operational insights. The Splunk Tail command can be use to detect updated or new rows in the database by referencing time stamp values. Splunk DB Connect also enables you to import data via periodic snapshots of the database—where database tables are recorded from a single point in time.
- Quick to set-up, scales to multiple concurrent databasesEnrich machine data with structured data from relational databasesExecute database queries directly from the Splunk user interface Browse and navigate database schemas and tablesCombine machine data with structured data from relational databases
- Community search engine Boardreader uses Splunk software to monitor its web crawlers and other essential business systems. In addition to generating large amounts of machine data on a daily basis, Boardreader maintains SQL databases to store business data and lookup tables such as country names, languages, production metrics and status, event and error codes. With Splunk DB Connect, Boardreader can run internal reports on usage trends and enhance the queries and reports with data from their SQL databases. Boardreader says Splunk DB Connect enabled it to replace its data warehouse.Boardreader offer two main services:Website – can be used anonymously or using a login for access and use certain features or functions, such as receiving email alerts, or certain services, usersneed to register and create an account.APIs – used by business customers to to run queries and load their analytical systems with critical social media data. 100s of requests per second.Before Splunk:Their existing reporting was broken. Too much data (and growing).Visibility was at best “a day behind”.“Online services, require real-time analytics”.Finding Splunk:In minutes: Downloaded Splunk and blasted it with packets. And it didn’t even break a sweat.In hours: Showed our CEO a dashboard – when he saw the graphics building in real-time and knew the data volumes behind it, he wanted it.In days: Entire Splunk deployment up and running and generating reports and dashboards.Now have processed and analyzed billions of events in Splunk.In terms of their environment:As well as data from 1000s of servers, they are bringing together business and IT data from 3 Microsoft SQL Server & MySQL databases (15-20 eventually possible) for real-time monitoring and some real-time business analytics.Standout factor of Splunk: Real-time. They are basically replacing a data warehouse.
- Corporation Service Company (CSC) is a leader for business, legal and financial services worldwide, offering a solution for every phase of the business life cycle. CSC uses Splunk Enterprise to monitor and correlate page load, traffic and user data to help deliver exceptional customer service (something they pride themselves on). Splunk DB Connect makes it easier for CSC customer service representatives to see top usage, overall volume and any problems customers are facing, by ensuring that they can map the customers’ usage and experience from the machine logs with their credentials contained in the database.CSC use the User ID from the machine data to correlate and analyze usage with customer details from their customer master.Before DB Connect, CSC used Splunk normal Lookup with CSV files. The main problem was that these CSV files where not refreshed for 90 days. So although the customer added new users to the database, these new users where not reflected in the CSV file. Therefore, the monitoring and reports included inaccurate Customer data.
- Most organizations maintain a diverse set of data stores – machine data, relational data and other unstructured data. Splunk DB Connect delivers real-time connectivity to relational databases and Splunk Hadoop Connect delivers bi-directional connectivity to Hadoop. Both Splunk Apps enable you to drive more meaningful insights from all of your data.