Extending Solr: Building a Cloud-like Knowledge Discovery Platform

Search engines are known for "relevancy", but the relevancy models that ship out of the box (BM25, classic tf-idf, etc.) are just scratching the surface of what's needed for a truly insightful application. What if your search engine could automatically tune its own domain-specific relevancy model based on user interactions? What if it could learn the important phrases and topics within your domain, learn the conceptual relationships embedded within your documents, and even use machine-learned ranking to discover the relative importance of different features and then automatically optimize its own ranking algorithms for your domain? What if you could further use SQL queries to explore these relationships within your own BI tools and return results in ranked order to deliver relevance-driven analytics visualizations? In this presentation, we'll walk through how you can leverage the myriad of capabilities in the Apache Solr ecosystem (such as the Solr Text Tagger, Semantic Knowledge Graph, Spark-Solr, Solr SQL, learning to rank, probabilistic query parsing, and Lucidworks Fusion) to build self-learning, relevance-first search, recommendations, and data analytics applications.

Presentation from Lucene Revolution 2011

Technologie Bildung

Extending Solr: Building a Cloud-like Knowledge Discovery Platform Trey Grainger , ,CareerBuilder

Overview ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

My background ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

CareerBuilder’s Search Scale ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Recommendations ,[object Object],[object Object],[object Object],[object Object]

Multi-lingual Analysis ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Multi-lingual Analysis ,[object Object],[object Object],Take-away : Lemmatization allows you to greatly increase recall while preserving the precision you lose with stemming i.e. English shows 92% increase in recall using Lemmatization with minimal impact on precision Measuring Recall Overlap Between Options

Custom Scoring ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Stopping Keyword Spamming ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

CareerBuilder’s Search Cloud ,[object Object],[object Object],[object Object],[object Object]

Creating a virtual search engine ,[object Object]

Creating a virtual search engine ,[object Object],[object Object],[object Object],[object Object],[object Object]

Take Aways ,[object Object],[object Object],[object Object]

Contact ,[object Object],[object Object],[object Object]

Weitere ähnliche Inhalte

Was ist angesagt?

Self-learned Relevancy with Apache Solr

Implementing Conceptual Search in Solr using LSA and Word2Vec: Presented by S...

What if your search engine could automatically tune its own domain-specific relevancy model? What if it could learn the important phrases and topics within your domain, automatically identify alternate spellings (synonyms, acronyms, and related phrases) and disambiguate multiple meanings of those phrases, learn the conceptual relationships embedded within your documents, and even use machine-learned ranking to discover the relative importance of different features and then automatically optimize its own ranking algorithms for your domain? In this presentation, you’ll learn you how to do just that - to evolving Lucene/Solr implementations into self-learning data systems which are able to accept user queries, deliver relevance-ranked results, and automatically learn from your users’ subsequent interactions to continually deliver a more relevant experience for each keyword, category, and group of users. Such a self-learning system leverages reflected intelligence to consistently improve its understanding of the content (documents and queries), the context of specific users, and the relevance signals present in the collective feedback from every prior user interaction with the system. Come learn how to move beyond manual relevancy tuning and toward a closed-loop system leveraging both the embedded meaning within your content and the wisdom of the crowds to automatically generate search relevancy algorithms optimized for your domain.

Reflected Intelligence: Lucene/Solr as a self-learning data system

When searching on text, choosing the right CharFilters, Tokenizer, stemmers, and other TokenFilters for each supported language is critical. Additional tools of the trade include language detection through UpdateRequestProcessors, parts of speech analysis, entity extraction, stopword and synonym lists, relevancy differentiation for exact vs. stemmed vs. conceptual matches, and identification of statistically interesting phrases per language. For multilingual search, you also need to choose between several strategies such as: searching across multiple fields, using a separate collection per language combination, or combining multiple languages in a single field (custom code is required for this and will be open sourced). These all have their own strengths and weaknesses depending upon your use case. This talk will provide a tutorial (with code examples) on how to pull off each of these strategies as well as compare and contrast the different kinds of stemmers, review the precision/recall impact of stemming vs. lemmatization, and describe some techniques for extracting meaningful relationships between terms to power a semantic search experience per-language. Come learn how to build an excellent semantic and multilingual search system using the best tools and techniques Lucene/Solr has to offer!

Semantic & Multilingual Strategies in Lucene/Solr

Leveraging Lucene/Solr as a Knowledge Graph and Intent Engine: Presented by T...

Search engines frequently miss the mark when it comes to understanding user intent. This talk will describe how to overcome this by leveraging Lucene/Solr to power a knowledge graph that can extract phrases, understand and weight the semantic relationships between those phrases and known entities, and expand the query to include those additional conceptual relationships. For example, if a user types in (Senior Java Developer Portland, OR Hadoop), you or I know that the term “senior” designates an experience level, that “java developer” is a job title related to “software engineering”, that “portland, or” is a city with a specific geographical boundary, and that “hadoop” is a technology related to terms like “hbase”, “hive”, and “map/reduce”. Out of the box, however, most search engines just parse this query as text:((senior AND java AND developer AND portland) OR (hadoop)), which is not at all what the user intended. We will discuss how to train the search engine to parse the query into this intended understanding, and how to reflect this understanding to the end user to provide an insightful, augmented search experience. Topics: Semantic Search, Finite State Transducers, Probabilistic Parsing, Bayes Theorem, Augmented Search, Recommendations, NLP, Knowledge Graphs

Leveraging Lucene/Solr as a Knowledge Graph and Intent Engine

Anyone Can Build A Recommendation Engine With Solr: Presented by Doug Turnbul...

Presented by Trey Grainger | CareerBuilder - See conference video - http://www.lucidimagination.com/devzone/events/conferences/lucene-revolution-2012 Searching text is what Solr is known for, but did you know that many companies receive an equal or greater business impact through implementing a recommendation engine in addition to their text search capabilities? With a few tweaks, Solr (or Lucene) can also serve as a full featured recommendation engine. Machine learning libraries like Apache Mahout provide excellent behavior-based, off-line recommendation algorithms, but what if you want more control? This talk will demonstrate how to effectively utilize Solr to perform collaborative filtering (users who liked this also liked…), categorical classification and subsequent hierarchical-based recommendations, as well as related-concept extraction and concept based recommendations. Sound difficult? It’s not. Come learn step-by-step how to create a powerful real-time recommendation engine using Apache Solr and see some real-world examples of some of these strategies in action.

Vespa, A Tour

MatthewOverstreet2

Building a Real-time Solr-powered Recommendation Engine

lucenerevolution

his talk will feature some of my recent research into the alternative uses for Solr facets and facet metadata. I will develop the idea that facets can be used to discover similarities between items and attributes in a search index, and show some interesting applications of this idea. A common takeaway is that using facets and facet metadata in non-conventional ways enables the semantic context of a query to be automatically tuned. This has important implications for user-centric and semantically focused relevance.

Haystacks slides

Ted Sullivan

While traditional keyword search is still useful, pure text-based keyword matching is quickly becoming obsolete; today, it is a necessary but not sufficient tool for delivering relevant results and intelligent search experiences. In this talk, we'll cover some of the emerging trends in AI-powered search, including the use of thought vectors (multi-level vector embeddings) and semantic knowledge graphs to contextually interpret and conceptualize queries. We'll walk through some live query interpretation demos to demonstrate the power that can be delivered through these semantic search techniques leveraging auto-generated knowledge graphs learned from your content and user interactions.

Thought Vectors and Knowledge Graphs in AI-powered Search

Building a semantic search system - one that can correctly parse and interpret end-user intent and return the ideal results for users’ queries - is not an easy task. It requires semantically parsing the terms, phrases, and structure within queries, disambiguating polysemous terms, correcting misspellings, expanding to conceptually synonymous or related concepts, and rewriting queries in a way that maps the correct interpretation of each end user’s query into the ideal representation of features and weights that will return the best results for that user. Not only that, but the above must often be done within the confines of a very specific domain - ripe with its own jargon and linguistic and conceptual nuances. This talk will walk through the anatomy of a semantic search system and how each of the pieces described above fit together to deliver a final solution. We'll leverage several recently-released capabilities in Apache Solr (the Semantic Knowledge Graph, Solr Text Tagger, Statistical Phrase Identifier) and Lucidworks Fusion (query log mining, misspelling job, word2vec job, query pipelines, relevancy experiment backtesting) to show you an end-to-end working Semantic Search system that can automatically learn the nuances of any domain and deliver a substantially more relevant search experience.

How to Build a Semantic Search System

Searching text is what Solr is known for, but did you know that many companies receive an equal or greater business impact through implementing a recommendation engine in addition to their text search capabilities? With a few tweaks, Solr (or Lucene) can also serve as a full featured recommendation engine. Machine learning libraries like Apache Mahout provide excellent behavior-based, off-line recommendation algorithms, but what if you want more control? This talk will demonstrate how to effectively utilize Solr to perform collaborative filtering (users who liked this also liked…), categorical classification and subsequent hierarchical-based recommendations, as well as related-concept extraction and concept based recommendations. Sound difficult? It’s not. Come learn step-by-step how to create a powerful real-time recommendation engine using Apache Solr and see some real-world examples of some of these strategies in action.

Building a real time, solr-powered recommendation engine

Solr 6.0 Graph Query Overview

Kevin Watters

Haystack 2018 - Algorithmic Extraction of Keywords Concepts and Vocabularies

Max Irwin

Search Accuracy Metrics and Predictive Analytics - A Big Data Use Case: Prese...

Webinar: Simpler Semantic Search with Solr

Evolving the Optimal Relevancy Ranking Model at Dice.com

Simon Hughes

Solr Graph Query: Presented by Kevin Watters, KMW Technology

This presentation is from the inaugural Atlanta Solr Meetup held on 2014/10/21 at Atlanta Tech Village. Description: CareerBuilder uses Solr to power their recommendation engine, semantic search, and data analytics products. They maintain an infrastructure of hundreds of Solr servers, holding over a billion documents and serving over a million queries an hour across thousands of unique search indexes. Come learn how CareerBuilder has integrated Solr into their technology platform (with assistance from Hadoop, Cassandra, and RabbitMQ) and walk through api and code examples to see how you can use Solr to implement your own real-time recommendation engine, semantic search, and data analytics solutions. Speaker: Trey Grainger is the Director of Engineering for Search & Analytics at CareerBuilder.com and is the co-author of Solr in Action (2014, Manning Publications), the comprehensive example-driven guide to Apache Solr. His search experience includes handling multi-lingual content across dozens of markets/languages, machine learning, semantic search, big data analytics, customized Lucene/Solr scoring models, data mining and recommendation systems. Trey is also the Founder of Celiaccess.com, a gluten-free search engine, and is a frequent speaker at Lucene and Solr-related conferences.

Scaling Recommendations, Semantic Search, & Data Analytics with solr

See conference video - http://www.lucidimagination.com/devzone/events/conferences/revolution/2011 For CareerBuilder, a 1% deviance in search relevancy can mean millions of missed job opportunities for our users. When CareerBuilder moved to Solr from an expensive, proprietary search vendor, our top priorities were maintaining the quality of our search results and drastically improving our agility. This talk will describe how we addressed both needs. For search quality, we’ll cover some of our internal studies and resulting methods for dealing with multi-lingual content across dozens of languages, as well as customizing and experimenting with relevancy calculations. For platform agility, we’ll discuss CareerBuilder’s cloud-like search API framework which seamlessly handles millions of searches an hour, processes hundreds of millions of documents, and is powered by hundreds of globally-distributed servers. Come hear the results of our studies and some best practices for quality and performance. Learn how our framework has lead to staggering improvements in both maintainability and technology innovation, allowing us to learn from our content, not just find it.

Was ist angesagt? (20)

Self-learned Relevancy with Apache Solr

Implementing Conceptual Search in Solr using LSA and Word2Vec: Presented by S...

Reflected Intelligence: Lucene/Solr as a self-learning data system

Semantic & Multilingual Strategies in Lucene/Solr

Leveraging Lucene/Solr as a Knowledge Graph and Intent Engine: Presented by T...

Leveraging Lucene/Solr as a Knowledge Graph and Intent Engine

Anyone Can Build A Recommendation Engine With Solr: Presented by Doug Turnbul...

Vespa, A Tour

Building a Real-time Solr-powered Recommendation Engine

Haystacks slides

Thought Vectors and Knowledge Graphs in AI-powered Search

How to Build a Semantic Search System

Building a real time, solr-powered recommendation engine

Solr 6.0 Graph Query Overview

Haystack 2018 - Algorithmic Extraction of Keywords Concepts and Vocabularies

Search Accuracy Metrics and Predictive Analytics - A Big Data Use Case: Prese...

Webinar: Simpler Semantic Search with Solr

Evolving the Optimal Relevancy Ranking Model at Dice.com

Solr Graph Query: Presented by Kevin Watters, KMW Technology

Scaling Recommendations, Semantic Search, & Data Analytics with solr

Ähnlich wie Extending Solr: Building a Cloud-like Knowledge Discovery Platform

Extending Solr: Behind CareerBuilder’s Cloud-like Knowledge Discovery Platfor...

lucenerevolution

Extending Solr: Building a Cloud-like Knowledge Discovery Platform

Lucidworks (Archived)

COMMitMDE'18: Eclipse Hawk: model repository querying as a service

Antonio García-Domínguez

Domain Driven Design

Ryan Riley

Vital AI MetaQL: Queries Across NoSQL, SQL, Sparql, and Spark

Vital.AI

Automated product categorization

Andreas Loupasakis

Automated product categorization

Warply

Unit 1 - TypeScript & Introduction to Angular CLI.pptx

Malla Reddy University

Sumo Logic QuickStart Webinar - Jan 2016

Sumo Logic QuickStart

Achieve big data analytic platform with lambda architecture on cloud

Scott Miao

Sumo Logic QuickStart - May 2016

Sumo Logic Quick Start - Feb 2016

Apache Solr for begginers

Alexander Tokarev

Sumo Logic Quickstart Training 10/14/2015

Software systems are growing in size and complexity when the business is growing, and sometimes it is hard to figure out what is going on. Various teams make different changes for different business capabilities. Distributed Tracing is a useful way to look under the hood and see for yourself what operations are being performed, what services are used in a certain use case, and how performant are they. In this talk, I will present what Distributed Tracing is and how we introduced it into our software system with some tips and tricks on what you should focus on if you want to do the same.

"Introducing Distributed Tracing in a Large Software System", Kostiantyn Sha...

Fwdays

SURE Research Report

Alex Sumner

Azure CosmosDb - Where we are

Marco Parenzan

Has your app taken off? Are you thinking about scaling? MongoDB makes it easy to horizontally scale out with built-in automatic sharding, but did you know that sharding isn't the only way to achieve scale with MongoDB? In this webinar, we'll review three different ways to achieve scale with MongoDB. We'll cover how you can optimize your application design and configure your storage to achieve scale, as well as the basics of horizontal scaling. You'll walk away with a thorough understanding of options to scale your MongoDB application. Topics covered include: - Scaling Vertically - Hardware Considerations - Index Optimization - Schema Design - Sharding

Webinar: Scaling MongoDB

MongoDB

Specialized tools for machine learning development and model governance are becoming essential. MlFlow is an open source platform for managing the machine learning lifecycle. Just by adding a few lines of code in the function or script that trains their model, data scientists can log parameters, metrics, artifacts (plots, miscellaneous files, etc.) and a deployable packaging of the ML model. Every time that function or script is run, the results will be logged automatically as a byproduct of those lines of code being added, even if the party doing the training run makes no special effort to record the results. MLflow application programming interfaces (APIs) are available for the Python, R and Java programming languages, and MLflow sports a language-agnostic REST API as well. Over a relatively short time period, MLflow has garnered more than 3,300 stars on GitHub , almost 500,000 monthly downloads and 80 contributors from more than 40 companies. Most significantly, more than 200 companies are now using MLflow. We will demo MlFlow Tracking , Project and Model components with Azure Machine Learning (AML) Services and show you how easy it is to get started with MlFlow on-prem or in the cloud.

Introducing MlFlow: An Open Source Platform for the Machine Learning Lifecycl...

DataWorks Summit

Ähnlich wie Extending Solr: Building a Cloud-like Knowledge Discovery Platform (20)

Extending Solr: Behind CareerBuilder’s Cloud-like Knowledge Discovery Platfor...

Extending Solr: Building a Cloud-like Knowledge Discovery Platform

COMMitMDE'18: Eclipse Hawk: model repository querying as a service

Domain Driven Design

Vital AI MetaQL: Queries Across NoSQL, SQL, Sparql, and Spark

Automated product categorization

Unit 1 - TypeScript & Introduction to Angular CLI.pptx

Sumo Logic QuickStart Webinar - Jan 2016

Sumo Logic QuickStart

Achieve big data analytic platform with lambda architecture on cloud

Sumo Logic QuickStart - May 2016

Sumo Logic Quick Start - Feb 2016

Apache Solr for begginers

Sumo Logic Quickstart Training 10/14/2015

"Introducing Distributed Tracing in a Large Software System", Kostiantyn Sha...

SURE Research Report

Azure CosmosDb - Where we are

Webinar: Scaling MongoDB

Introducing MlFlow: An Open Source Platform for the Machine Learning Lifecycl...

Mehr von Trey Grainger

The first step in returning relevant search results is successfully interpreting the user’s intent. This requires combining a holistic understanding of your content, your users, and your domain. Traditional keyword search focuses on the content understanding dimension. Knowledge graphs are then typically built and leveraged to represent an understanding of your domain. Finally, Collaborative recommendations and user profile learning are typically the tools of choice for generating and modeling an understanding of the preferences of each user. While these systems (search, recommendations, and knowledge graphs) are often built and used in isolation, combining them together is the key to truly understanding a user’s query intent. For example, combining traditional keyword search with your knowledge graph leads to semantic search capabilities, and combining traditional keyword search with recommendations leads to personalized search experiences. Combining all of these dimensions together in an appropriately balanced way will ultimately lead to the most accurate interpretation of a user’s query, resulting in a better query to the core search engine and ultimately a better, more relevant search experience. In this talk, we’ll demonstrate strategies for delivering and combining each of these dimensions of user intent, and we’ll walk through concrete examples of how to balance the nuances of each so that you also don’t over-personalize, over-contextualize, or under appreciate the nuances of your user’s intent.

Balancing the Dimensions of User Intent

The goal of most digital transformations is to create competitive advantage by enhancing customer experience and employee success, so giving these stakeholders the ability to find the right information at their moment of need is paramount. Employees and customers increasingly expect an intuitive, interactive experience where they can simply type or speak their questions or keywords into a search box, their intent will be understood, and the best answers and content are then immediately presented. Providing this compelling experience, however, requires a deep understanding of your content, your unique business domain, and the collective and personalized needs of each of your users. Modern artificial intelligence (AI) approaches are able to continuously learn from both your content and the ongoing stream of user interactions with your applications, and to automatically reflect back that learned intelligence in order to instantly and scalably deliver contextually-relevant answers to employees and customers. In this talk, we'll discuss how AI is currently being deployed across the Fortune 1000 to accomplish these goals, both in the digital workplace (helping employees more efficiently get answers and make decisions) and in digital commerce (understanding customer intent and connecting them with the best information and products). We'll separate fact from fiction as we break down the hype around AI and show how it is being practically implemented today to power many real-world digital transformations for the next generation of employees and customers.

Reflected Intelligence: Real world AI in Digital Transformation

To optimally interpret most natural language queries, its important to understand a highly-nuanced, contextual interpretation of the domain-specific phrases, entities, commands, and relationships represented or implied within the search and within your domain. In this talk, we'll walk through such a search system powered by Solr's Text Tagger and Semantic Knowledge graph. We'll have fun with some of the more search-centric use cases of knowledge graphs, such as entity extraction, query expansion, disambiguation, and pattern identification within our queries: for example, transforming the query "best bbq near activate" into: {!func}mul(min(popularity,1),100) bbq^0.91032 ribs^0.65674 brisket^0.63386 doc_type:"restaurant" {!geofilt d=50 sfield="coordinates_pt" pt="38.916120,-77.045220"} We'll see a live demo with real world data demonstrating how you can build and apply your own knowledge graphs to power much more relevant query understanding like this within your search engine.

Natural Language Search with Knowledge Graphs (Chicago Meetup)

What does it really mean to deliver an "AI-powered Search" solution? In this talk, we’ll bring clarity to this topic, showing you how to marry the art of the possible with the real-world challenges involved in understanding your content, your users, and your domain. We'll dive into emerging trends in AI-powered Search, as well as many of the stumbling blocks found in even the most advanced AI and Search applications, showing how to proactively plan for and avoid them. We'll walk through the various uses of reflected intelligence and feedback loops for continuous learning from user behavioral signals and content updates, also covering the increasing importance of virtual assistants and personalized search use cases found within the intersection of traditional search and recommendation engines. Our goal will be to provide a baseline of mainstream AI-powered Search capabilities available today, and to paint a picture of what we can all expect just on the horizon.

The Next Generation of AI-powered Search

Natural Language Search with Knowledge Graphs (Activate 2019)

AI, Search, and the Disruption of Knowledge Management

Measuring Relevance in the Negative Space

To optimally interpret most natural language queries, it is necessary to understand the phrases, entities, commands, and relationships represented or implied within the search. Knowledge graphs serve as useful instantiations of ontologies which can help represent this kind of knowledge within a domain. In this talk, we'll walk through techniques to build knowledge graphs automatically from your own domain-specific content, how you can update and edit the nodes and relationships, and how you can seamlessly integrate them into your search solution for enhanced query interpretation and semantic search. We'll have some fun with some of the more search-centric use cased of knowledge graphs, such as entity extraction, query expansion, disambiguation, and pattern identification within our queries: for example, transforming the query "bbq near haystack" into { filter:["doc_type":"restaurant"], "query": { "boost": { "b": "recip(geodist(38.034780,-78.486790),1,1000,1000)", "query": "bbq OR barbeque OR barbecue" } } } We'll also specifically cover use of the Semantic Knowledge Graph, a particularly interesting knowledge graph implementation available within Apache Solr that can be auto-generated from your own domain-specific content and which provides highly-nuanced, contextual interpretation of all of the terms, phrases and entities within your domain. We'll see a live demo with real world data demonstrating how you can build and apply your own knowledge graphs to power much more relevant query understanding within your search engine.

Natural Language Search with Knowledge Graphs (Haystack 2019)

The Future of Search and AI

The Semantic Knowledge Graph is an Apache Solr plugin that can be used to discover and rank the relationships between any arbitrary queries or terms within the search index. It is a relevancy swiss army knife, able to discover related terms and concepts, disambiguate different meanings of terms given their context, cleanup noise in datasets, discover previously unknown relationships between entities across documents and fields, rank lists of keywords based upon conceptual cohesion to reduce noise, summarize documents by extracting their most significant terms, generate recommendations and personalized search, and power numerous other applications involving anomaly detection, significance/relationship discovery, and semantic search. This talk will walk you through how to setup and use this plugin in concert with other open source tools (probabilistic query parser, SolrTextTagger for entity extraction) to parse, interpret, and much more correctly model the true intent of user searches than traditional keyword-based search approaches.

The Relevance of the Apache Solr Semantic Knowledge Graph

Searching for Meaning

What if instead of a query returning documents, you could alternatively return other keywords most related to the query: i.e. given a search for "data science", return me back results like "machine learning", "predictive modeling", "artificial neural networks", etc.? Solr’s Semantic Knowledge Graph does just that. It leverages the inverted index to automatically model the significance of relationships between every term in the inverted index (even across multiple fields) allowing real-time traversal and ranking of any relationship within your documents. Use cases for the Semantic Knowledge Graph include disambiguation of multiple meanings of terms (does "driver" mean truck driver, printer driver, a type of golf club, etc.), searching on vectors of related keywords to form a conceptual search (versus just a text match), powering recommendation algorithms, ranking lists of keywords based upon conceptual cohesion to reduce noise, summarizing documents by extracting their most significant terms, and numerous other applications involving anomaly detection, significance/relationship discovery, and semantic search. In this talk, we'll do a deep dive into the internals of how the Semantic Knowledge Graph works and will walk you through how to get up and running with an example dataset to explore the meaningful relationships hidden within your data.

The Apache Solr Semantic Knowledge Graph

South Big Data Hub: Text Data Analysis Panel

Presentation of the Semantic Knowledge Graph research paper at the 2016 IEEE 3rd International Conference on Data Science and Advanced Analytics (Montreal, Canada - October 18th, 2016) Abstract—This paper describes a new kind of knowledge representation and mining system which we are calling the Semantic Knowledge Graph. At its heart, the Semantic Knowledge Graph leverages an inverted index, along with a complementary uninverted index, to represent nodes (terms) and edges (the documents within intersecting postings lists for multiple terms/nodes). This provides a layer of indirection between each pair of nodes and their corresponding edge, enabling edges to materialize dynamically from underlying corpus statistics. As a result, any combination of nodes can have edges to any other nodes materialize and be scored to reveal latent relationships between the nodes. This provides numerous benefits: the knowledge graph can be built automatically from a real-world corpus of data, new nodes - along with their combined edges - can be instantly materialized from any arbitrary combination of preexisting nodes (using set operations), and a full model of the semantic relationships between all entities within a domain can be represented and dynamically traversed using a highly compact representation of the graph. Such a system has widespread applications in areas as diverse as knowledge modeling and reasoning, natural language processing, anomaly detection, data cleansing, semantic search, analytics, data classification, root cause analysis, and recommendations systems. The main contribution of this paper is the introduction of a novel system - the Semantic Knowledge Graph - which is able to dynamically discover and score interesting relationships between any arbitrary combination of entities (words, phrases, or extracted concepts) through dynamically materializing nodes and edges from a compact graphical representation built automatically from a corpus of data representative of a knowledge domain.

The Semantic Knowledge Graph

Having “big data” is great, but turning that data into actionable intelligence is where the real value lies. This talk will demonstrate how you can use Solr to build a highly scalable data analytics engine to enable customers to engage in lightning fast, real-time knowledge discovery. At CareerBuilder, we utilize these techniques to report the supply and demand of the labor force, compensation trends, customer performance metrics, and many live internal platform analytics. You will walk away from this talk with an advanced understanding of faceting, including pivot-faceting, geo/radius faceting, time-series faceting, function faceting, and multi-select faceting. You’ll also get a sneak peak at some new faceting capabilities just wrapping up development including distributed pivot facets and percentile/stats faceting, which will be open-sourced. The presentation will be a technical tutorial, along with real-world use-cases and data visualizations. After this talk, you'll never see Solr as just a text search engine again.

Building a real time big data analytics platform with solr