This document discusses building a data-driven log analysis application using LucidWorks SILK. It begins with an introduction to LucidWorks and discusses the continuum of search capabilities from enterprise search to big data search. It then describes how SILK can enable big data search across structured and unstructured data at massive scale. The solution components involve collecting log data from various sources using connectors, ingesting it into Solr, and building visualizations for analysis. It concludes with a demo and contact information.
McKinsey estimates that search and big data analysis can increase profits in the retail sector by 60%. Increasingly, innovation in this sector means simulation, experimentation and iteration. Access to data and understanding the user patters in order to run different modes is what drives this growth. These are over course techniques that’s search practioners have been perfecting for over a decade
McKinsey estimates that search and big data analysis can increase profits in the retail sector by 60%. Increasingly, innovation in this sector means simulation, experimentation and iteration. Access to data and understanding the user patters in order to run different modes is what drives this growth. These are over course techniques that’s search practioners have been perfecting for over a decade
Rather than speak solely in the abstract, I shall illustrate how we internally use LucidWorks SILK to get insight from search logs
For the Search Analytics case, I am fortunate that my users are sitting next to me
I chose LogStash for data transformation and import for two reasons: It provides a powerful framework for extracting, grokking and transforming log data into a structured format that Solr can consume and that SILK can use for dashboards.LucidWorks’ Hadoop Connectors have a GrokIngestMapper that allows me to reuse the same LogStash Filters to work with larger volumes of files on HDFS (more details on this in a future article).