How is the Government Spending Your Money? How GCE is Using Lucene and the GCE Big Data Cloud

•

0 gefällt mir•472 views

Presented by Seshu Simhadri | Global Computer Enterprises - See conference video - http://www.lucidimagination.com/devzone/events/conferences/lucene-revolution-2012 A leader in bringing innovative technologies to the Federal Government, GCE looks to open source tools to drive down cost and provide the foundation for building value-added services for its customers. This talk will discus GCE’s innovative use of Lucene/Solr combined with the GCE Big Data Cloud to open up access to Federal spending data. This data is in wide use across the Federal government, Federal contracting community, media and press, as well as Capitol Hill. GCE has utilized this toolset to deliver the type of capability that users typically only find in web consumer applications. This session will highlight the technical side of the challenge in implementing these tools across a large user community and data set in a Cloud environment.

Technologie

Lucene in the Cloud:
Leveraging the
Power of Search and
Big Data to Shed
Light on Government
Spending

Seshubabu Simhadri
Chief Technology Officer, GCE

Confidential, Do Not Disclose. Property of Global
Computer Enterprises, Inc..

Background

What is USASpending.gov?

Moving to Our Big Data cloud

Some of the design decisions
Tool Selection
Cluster Design
Hardware Design

Limitations and enhancements

Overview
Confidential, Do Not Disclose. Property of Global Computer Enterprises, Inc..

What is USASpending.gov?
Confidential, Do Not Disclose. Property of Global Computer Enterprises, Inc..

U.S. Government Spending vs. Other Entities
Confidential, Do Not Disclose. Property of Global Computer Enterprises, Inc..

Distribution of U.S. Government Spending
Confidential, Do Not Disclose. Property of Global Computer Enterprises, Inc..

• Analytics
•  Stats
•  Top-K

• Free Text
Search
(With auto
Suggestions)

• Large
Data
Feeds

• APIs
What can users do on the site?
Confidential, Do Not Disclose. Property of Global Computer Enterprises, Inc..

• Public

• Media

• Congress

• Value Added
Resellers

Who are the users of the site?
Confidential, Do Not Disclose. Property of Global Computer Enterprises, Inc..

Leveraging the
industry leading
open source
platform to
deliver cost
savings and
scalability within
a Cloud
computing
model

GCE Big Data and Analytics Cloud
Confidential, Do Not Disclose. Property of Global Computer Enterprises, Inc..

•  Hadoop
− For indexing and downloads
Start by
•  Distributed Solr Looking at
− Analytics the Usual
− Free text search Suspects

•  Drupal static content

•  Visualization

What’s Inside the GCE Cloud?
Confidential, Do Not Disclose. Property of Global Computer Enterprises, Inc..

The greatest
challenge is how
to optimally
design a node –
which
combination of
CPUs, memory,
and shard size
delivers the
desired
performance?

Solr Node Sizing
Confidential, Do Not Disclose. Property of Global Computer Enterprises, Inc..

Multiple index types
Different types of spending
Varying sizes

Break complete dataset into shards as small as required to
meet the response times
Choose shard size based on response times

Single Core with multiple cores or Multiple Solr instances each
with single core?

Solr Node Sizing
Confidential, Do Not Disclose. Property of Global Computer Enterprises, Inc..

How do you
design the
cluster –
which ones
are individual
nodes and
which ones
are
aggregators?

Solr Cluster Design
Confidential, Do Not Disclose. Property of Global Computer Enterprises, Inc..

Should all shards be treated equal?

Userà Aggregator Nodes à Shards

Different requirements for nodes collecting the data
and nodes serving a specific dataset

Aggregator Node 1,2,3 ….m
Large Solr Instances, No local index

Shard Nodes 1,2,3,..100..n
Small Solr Instance with index

Solr Cluster Design
Confidential, Do Not Disclose. Property of Global Computer Enterprises, Inc..

Separate Solr
instances

Multiple hard
drives per
server

Solid state
disks

Infiniband

What configuration did we choose?
Confidential, Do Not Disclose. Property of Global Computer Enterprises, Inc..

Enhanced
Faceting:
Enabling
aggregation
by more than
one field

Will be
contributed to
Solr project

Solr Enhancements
Confidential, Do Not Disclose. Property of Global Computer Enterprises, Inc..

When the shards
increase,
management of
SQLs inside Solr
becomes a
challenge

External Data
Importer Using
Hadoop

Solr Data Importer: Why Not?
Confidential, Do Not Disclose. Property of Global Computer Enterprises, Inc..

Solr in the
Cloud required
building a cost
effective and
high
performance
infrastructure

Small vs. large
Commodity
servers

Utilizing Large Commodity Servers
Confidential, Do Not Disclose. Property of Global Computer Enterprises, Inc..

Failure of one
node results
in failure of
multiple
shards -
careful
design is
required

Disadvantages of higher capacity servers
Confidential, Do Not Disclose. Property of Global Computer Enterprises, Inc..

Sharded architecture

Multiple Solr instances per server each handling small
datasets

Aggregator nodes + shards

Hadoop for data indexing and data feeds

Large Commodity Servers
•  48-core
•  256GB RAM
•  SSD
•  Infiniband

Summary
Confidential, Do Not Disclose. Property of Global Computer Enterprises, Inc..

Come build
the future
of Big Data

GCECloud.com

We’re hiring!
Confidential, Do Not Disclose. Property of Global Computer Enterprises, Inc..

Questions?
ssimhadri at GCECloud.com

Visit us at www.GCECloud.com

Weitere ähnliche Inhalte

Was ist angesagt?

Hadoop 2.0 offers significant HDFS improvements: new append-pipeline, federation, wire compatibility, NameNode HA, performance improvements, etc. We describe these features and their benefits. We also discuss development that is underway for the next HDFS release. This includes much needed data management features such as Snapshots and Disaster Recovery. We add support for different classes of storage devices such as SSDs and open interfaces such as NFS; together these extend HDFS as a more general storage system. As with every release we will continue improvements to performance, diagnosability and manageability of HDFS.

HDFS - What's New and Future

DataWorks Summit

Larry Smarr - Making Sense of Information Through Planetary Scale Computing

Diamond Exchange

CH07-Types of Storage

Sukanya Ben

Getting more value out of your blade solution is a smart way to reduce data center and VDI costs. In our tests, the Dell PowerEdge M620 blade solution not only cost up to 19.4 percent less per user than blade solutions from both HP and Cisco, it also consumed as much as 63.1 percent less power per virtual desktop user. When extrapolated over an entire chassis full of servers, and potentially data centers full of these chassis, the potential savings both in hardware costs and power costs could be dramatic.

Dell PowerEdge M620 blade server solutions for virtual desktop infrastructures

Principled Technologies

Sun sparc enterprise t2 systems customer presentation

xKinAnx

Sun sparc enterprise t5140 and t5240 servers customer presentation

xKinAnx

As a C-level executive, you are always on the lookout for ways to reduce IT costs while increasing systems capability in order to grow sales and/or improve service. Leveraging applications with automated business processes that enable user centric interconnected applications embracing interfaces that are inherently enabled for smart devices with efficient systems represents a proven way to improve the business.

TCA/TCO Benefits of Consolidating Databases and x86 Servers on IBM Enterprise...

IBM India Smarter Computing

OMG DDS Tutorial - Part I

Angelo Corsaro

Prince Building Tech Talk 12102012

Andy Parsons

Webinar: eFolder Expert Series: BDR Pain Relief with Lloyd Wolf

Dropbox

The CIOs Guide to NoSQL 2012

DATAVERSITY

Interoperability demonstration between 5 different products that implement the OMG DDS Interoperability Wire Protocol (DDS-RTPS). The demonstration took place at the March 2011 OMG technical meeting in Washington DC. The following companies demonstrated interoperability between their products: RTI (Connext DDS). TwinOaks Computing (CoreDX), PrismTech (OpenSpliceDDS), Gallium Visual Systems/Kongsberg (Compass DDS), IBM.

Dds interop demo_washington_dds_2011_03_01

Gerardo Pardo-Castellote

ieee title

srikanthsailu

Netmagic Cloud Computing Services

Netmagic Solutions Pvt. Ltd.

Migrate

Herbert Balagtas

EvoApp Bermuda (patent pending) is a highly scalable, cloud-native, in-memory analytic engine capable of analyzing large amounts of data extremely fast. Bermuda provides cost-effective, real-time, Big Data analysis and insight for both unstructured and structured data, enabling a wide range of business applications. Bermuda is capable of performing sub-second queries over billions of items, leveraging virtual machines and a cloud-scale storage system providing transactional, persistent storage of data. In addition to world-leading performance on the data sets for which it is optimized, the other major benefit of Bermuda is that a user does not have to define specific queries ahead of time, as is required with traditional business intelligence systems or a platform like Hadoop. Bermuda was built to support real-time, ad-hoc queries over large datasets. With Bermuda, a user can change queries on the fly, adjusting charts and reports and seeing results immediately. This expands the options associated with analytics on big data--more closely resembling a web search than traditional business intelligence STET reports. Bermuda can achieve such exceptionally fast query response times because data is organized in a proprietary, patent-pending architecture that facilitates scan-intensive queries. These make up the bulk of business intelligence analytics computations (i.e. time series, computing averages or sums, grouping by day, hour, etc. over large datasets); by optimizing Bermuda for this type of query, the engine is able to allocate workload across hundreds or even thousands of servers, easily accommodating terabytes of information. Additionally, all queries are non-blocking to the writing of new information or updates to existing data. The Bermuda architecture is unique because it combines the scalability of NoSQL databases, the performance of pure in-memory processing, and the cost/benefit advantages of a cloud-native deployment. It creates value by allowing EvoApp customers to make decisions and gain insights from massive quantities of data in an iterative, real-time environment. This represents a huge advance in the state of the art of unstructured data analytics and delivers on the promise of real-time/ad-hoc queries at scale.

EvoApp - Bermuda Real-Time Analytics Platform

Sergei Dolukhanov

Juju

OpenCity Community

EFL Munich - February 2013 - "Conversational Big Data with Erlang"

darach

Power Optimization Through Manycore Multiprocessing

chiportal

Was ist angesagt? (19)

HDFS - What's New and Future

Larry Smarr - Making Sense of Information Through Planetary Scale Computing

CH07-Types of Storage

Dell PowerEdge M620 blade server solutions for virtual desktop infrastructures

Sun sparc enterprise t2 systems customer presentation

Sun sparc enterprise t5140 and t5240 servers customer presentation

TCA/TCO Benefits of Consolidating Databases and x86 Servers on IBM Enterprise...

OMG DDS Tutorial - Part I

Prince Building Tech Talk 12102012

Webinar: eFolder Expert Series: BDR Pain Relief with Lloyd Wolf

The CIOs Guide to NoSQL 2012

Dds interop demo_washington_dds_2011_03_01

ieee title

Netmagic Cloud Computing Services

Migrate

EvoApp - Bermuda Real-Time Analytics Platform

Juju

EFL Munich - February 2013 - "Conversational Big Data with Erlang"

Power Optimization Through Manycore Multiprocessing

Andere mochten auch

Configure your Solr cluster to handle hundreds of millions of documents without even noticing, handle queries in milliseconds, use Near Real Time indexing and searching with document versioning. Scale your cluster both horizontally and vertically by using shards and replicas. In this session you'll learn how to make your indexing process blazing fast and make your queries efficient even with large amounts of data in your collections. You'll also see how to optimize your queries to leverage caches as much as your deployment allows and how to observe your cluster with Solr administration panel, JMX, and third party tools. Finally, learn how to make changes to already deployed collections —split their shards and alter their schema by using Solr API.

Scaling Solr with SolrCloud

lucenerevolution

Presented by Daniel Beach, Search Application Developer, OpenSource Connections Solr is a powerful search engine, but creating a custom user interface can be daunting. In this fast paced session I will present an overview of how to implement a client-side search application using Solr. Using open-source frameworks like SpyGlass (to be released in September) can be a powerful way to jumpstart your development by giving you out-of-the box results views with support for faceting, autocomplete, and detail views. During this talk I will also demonstrate how we have built and deployed lightweight applications that are able to be performant under large user loads, with minimal server resources.

Building Client-side Search Applications with Solr

lucenerevolution

Presented by Trey Grainger | CareerBuilder - See conference video - http://www.lucidimagination.com/devzone/events/conferences/lucene-revolution-2012 Searching text is what Solr is known for, but did you know that many companies receive an equal or greater business impact through implementing a recommendation engine in addition to their text search capabilities? With a few tweaks, Solr (or Lucene) can also serve as a full featured recommendation engine. Machine learning libraries like Apache Mahout provide excellent behavior-based, off-line recommendation algorithms, but what if you want more control? This talk will demonstrate how to effectively utilize Solr to perform collaborative filtering (users who liked this also liked…), categorical classification and subsequent hierarchical-based recommendations, as well as related-concept extraction and concept based recommendations. Sound difficult? It’s not. Come learn step-by-step how to create a powerful real-time recommendation engine using Apache Solr and see some real-world examples of some of these strategies in action.

Building a Real-time Solr-powered Recommendation Engine

lucenerevolution

Where Search Meets Machine Learning: Presented by Diana Hu & Joaquin Delgado,...

Lucidworks

Search at Twitter

lucenerevolution

Presented by Timothy Potter, Founder, Text Centrix Storm is a real-time distributed computation system used to process massive streams of data. Many organizations are turning to technologies like Storm to complement batch-oriented big data technologies, such as Hadoop, to deliver time-sensitive analytics at scale. This talk introduces on an emerging architectural pattern of integrating Solr and Storm to process big data in real time. There are a number of natural integration points between Solr and Storm, such as populating a Solr index or supplying data to Storm using Solr’s real-time get support. In this session, Timothy will cover the basic concepts of Storm, such as spouts and bolts. He’ll then provide examples of how to integrate Solr into Storm to perform large-scale indexing in near real-time. In addition, we'll see how to embed Solr in a Storm bolt to match incoming tuples against pre-configured queries, commonly known as percolator. Attendees will come away from this presentation with a good introduction to stream processing technologies and several real-world use cases of how to integrate Solr with Storm.

Integrate Solr with real-time stream processing applications

lucenerevolution

Presented by Isabel Drost-Fromm, Software Developer, Apache Software Foundation/Nokia Gate 5 GmbH at Lucene/Solr Revolution 2013 Dublin Text classification automates the task of filing documents into pre-defined categories based on a set of example documents. The first step in automating classification is to transform the documents to feature vectors. Though this step is highly domain specific Apache Mahout provides you with a lot of easy to use tooling to help you get started, most of which relies heavily on Apache Lucene for analysis, tokenisation and filtering. This session shows how to use facetting to quickly get an understanding of the fields in your document. It will walk you through the steps necessary to convert your text documents into feature vectors that Mahout classifiers can use including a few anecdotes on drafting domain specific features. Configure

Text Classification Powered by Apache Mahout and Lucene

lucenerevolution

Presented by Markus Klose, Search + Big Data Consultant SHI Elektronische Medien GmbH at Lucene/Solr Revolution 2013 Dublin Kibana4Solr is search-driven, scalable, browser based and extremely user friendly (also for non-technical users). Logs are everywhere. Any device, system or human can potentially produce a huge amount of information saved in logs. The amount of available logs and their semi-structured nature make a meaningful processing in real-time quite a difficult task. Thus, valuable business insights stored in logs might be not found. Kibana4Solr is a search-driven approach to handle that challenge. It offers user-friendly and browser-based dashboard which can be easily customized to particular needs. In the session the Kibana4Solr will be introduced. Some light will be shed on the architectural features of Kibana4Solr. Some ideas will be given in terms of possible business uses cases. And finally a live demo of Kibana4Solr will be shown. Configure

State of the Art Logging. Kibana4Solr is Here!

lucenerevolution

Andere mochten auch (8)

Scaling Solr with SolrCloud

Building Client-side Search Applications with Solr

Building a Real-time Solr-powered Recommendation Engine

Where Search Meets Machine Learning: Presented by Diana Hu & Joaquin Delgado,...

Search at Twitter

Integrate Solr with real-time stream processing applications

Text Classification Powered by Apache Mahout and Lucene

State of the Art Logging. Kibana4Solr is Here!

Ähnlich wie How is the Government Spending Your Money? How GCE is Using Lucene and the GCE Big Data Cloud

The Cloud Changing the Game

Amazon Web Services

Cloud computing basics

Akshay Guleria

Cloudcomputing Nivo Consultancy 26 Mei 2009 Versie 1

Ruud Ramakers

Big Data on Cloud Native Platform

Sunil Govindan

Big Data on Cloud Native Platform

Sunil Govindan

Cloud computing offers a very important approach to achieving lasting strategic advantages by rapidly adapting to complex challenges in IT management and data analytics. This paper discusses the business impact and analytic transformation opportunities of cloud computing. Moreover, it highlights the differences among two cloud architectures—Utility Clouds and Data Clouds—with illustrative examples of how Data Clouds are shaping new advances in Intelligence Analysis.

Massive Data Analytics and the Cloud

Booz Allen Hamilton

Dr. Michael Valivullah, NASS/USDA - Cloud Computing

ikanow

The Sun Cloud

Megan Eskey

Cloud computing and libraries sndt

Vishwas Taralekar

ONS content extraction

KellyCheah

Zsl cloud-application migration-8_phased_approach

zslmarketing

The elephantintheroom bigdataanalyticsinthecloud

Khazret Sapenov

NCOIC Enterprise Cloud Computing - Kevin Jackson

GovCloud Network

Hive solutions cloudviews 2010 presentation

EuroCloud

colony framework & omni platform

Hive Solutions

The number of Ruby on Rails applications running in the cloud has been augmenting. And so has the number of public and private cloud solutions for Ruby on Rails. Choosing between them and—even before that—choosing between a public and a private cloud can be difficult. This white paper is intended to show you when you can benefit from running your Ruby on Rails application in the cloud, explain the differences between a public and a private cloud, and help you choose between the two.

Comparing Ruby on Rails Public vs. Private Cloud Options

Altoros

The who, what, why, when & how of cloud computing

Nari Kannan

Vr storm cips_03nov2010

National Research Council Canada

When where why cloud

reshmaroberts

When Where Why Cloud

reshmaroberts

Ähnlich wie How is the Government Spending Your Money? How GCE is Using Lucene and the GCE Big Data Cloud (20)

The Cloud Changing the Game

Cloud computing basics

Cloudcomputing Nivo Consultancy 26 Mei 2009 Versie 1

Big Data on Cloud Native Platform

Massive Data Analytics and the Cloud

Dr. Michael Valivullah, NASS/USDA - Cloud Computing

The Sun Cloud

Cloud computing and libraries sndt

ONS content extraction

Zsl cloud-application migration-8_phased_approach

The elephantintheroom bigdataanalyticsinthecloud

NCOIC Enterprise Cloud Computing - Kevin Jackson

Hive solutions cloudviews 2010 presentation

colony framework & omni platform

Comparing Ruby on Rails Public vs. Private Cloud Options

The who, what, why, when & how of cloud computing

Vr storm cips_03nov2010

When where why cloud

When Where Why Cloud

Mehr von lucenerevolution

Presented by Rafal Kuć, Consultant and Software engineer, , Sematext Group, Inc. Even though Solr can run without causing any troubles for long periods of time it is very important to monitor and understand what is happening in your cluster. In this session you will learn how to use various tools to monitor how Solr is behaving at a high level, but also on Lucene, JVM, and operating system level. You'll see how to react to what you see and how to make changes to configuration, index structure and shards layout using Solr API. We will also discuss different performance metrics to which you ought to pay extra attention. Finally, you'll learn what to do when things go awry - we will share a few examples of troubleshooting and then dissect what was wrong and what had to be done to make things work again.

Administering and Monitoring SolrCloud Clusters

lucenerevolution

In a recent project with the United States Patent and Trademark Office, Opensource Connections was asked to prototype the next generation of patent search - using Solr and Lucene. An important aspect of this project was the implementation of BRS, a specialized search syntax used by patent examiners during the examination process. In this fast paced session we will relate our experiences and describe how we used a combination of Parboiled (a Parser Expression Grammar [PEG] parser), Lucene Queries and SpanQueries, and an extension of Solr's QParserPlugin to build BRS search functionality in Solr. First we will characterize the patent search problem and then define the BRS syntax itself. We will then introduce the Parboiled parser and discuss various considerations that one must make when designing a syntax parser. Following this we will describe the methodology used to implement the search functionality in Lucene/Solr. Finally, we will include an overview our syntactic and semantic testing strategies. The audience will leave this session with an understanding of how Solr, Lucene, and Parboiled may be used to implement their own custom search parser.

Implementing a Custom Search Syntax using Solr, Lucene, and Parboiled

lucenerevolution

Many of us tend to hate or simply ignore logs, and rightfully so: they’re typically hard to find, difficult to handle, and are cryptic to the human eye. But can we make logs more valuable and more usable if we index them in Solr, so we can search and run real-time statistics on them? Indeed we can, and in this session you’ll learn how to make that happen. In the first part of the session we’ll explain why centralized logging is important, what valuable information one can extract from logs, and we’ll introduce the leading tools from the logging ecosystems everyone should be aware of - from syslog and log4j to LogStash and Flume. In the second part we’ll teach you how to use these tools in tandem with Solr. We’ll show how to use Solr in a SolrCloud setup to index large volumes of logs continuously and efficiently. Then, we'll look at how to scale the Solr cluster as your data volume grows. Finally, we'll see how you can parse your unstructured logs and convert them to nicely structured Solr documents suitable for analytical queries.

Using Solr to Search and Analyze Logs

lucenerevolution

Enhancing relevancy through personalization & semantic search

lucenerevolution

Building real-time notification systems is often limited to basic filtering and pattern matching against incoming records. Allowing users to query incoming documents using Solr's full range of capabilities is much more powerful. In our environment we needed a way to allow for tens of thousands of such query subscriptions, meaning we needed to find a way to distribute the query processing in the cloud. By creating in-memory Lucene indices from our Solr configuration, we were able to parallelize our queries across our cluster. To achieve this distribution, we wrapped the processing in a Storm topology to provide a flexible way to scale and manage our infrastructure. This presentation will describe our experiences creating this distributed, real-time inverted search notification framework.

Real-time Inverted Search in the Cloud Using Lucene and Storm

lucenerevolution

Like many Web-Applications in the past, the Solr Admin UI up until 4.0 was entirely server based. It used separate code on the server to generate their Dashboards, Overviews and Statistics. All that code had to be maintained and still ... you weren't really able to use that kind of data for the things you needed it for. It was wrapped into HTML, most of the time difficult to extract and changed the structure from time to time w/o announcement. After a short look back, we're going to look into the current state of the Solr Admin UI - a client-side application, running completely in your browser. We'll see how it works, where it gets its data from and how you can get the very same data and wire that into your own custom applications, dashboards and/oder monitoring systems.

Solr's Admin UI - Where does the data come from?

lucenerevolution

Schemaless Solr and the Solr Schema REST API

lucenerevolution

Presented by Renaud Delbru, Co-Founder, SindiceTech In this presentation, we will discuss how Lucene and Solr can be used for very efficient search of tree-shaped schemaless document, e.g. JSON or XML, and can be then made to address both graph and relational data search. We will discuss the capabilities of SIREn, a Lucene/Solr plugin we have developed to deal with huge collections of tree-shaped schemaless documents, and how SIREn is built using Lucene extensibility capabilities (Analysis, Codec, Flexible Query Parser). We will compare it with Lucene's BlockJoin Query API in nested schemaless data intensive scenarios. We will then go through use cases that show how relational or graph data can be turned into JSON documents using Hadoop and Pig, and how this can be used in conjunction with SIREn to create relational faceting systems with unprecedented performance. Take-away lessons from this session will be awareness about using Lucene/Solr and Hadoop for relational and graph data search, as well as the awareness that it is now possible to have relational faceted browsers with sub-second response time on commodity hardware.

High Performance JSON Search and Relational Faceted Browsing with Lucene

lucenerevolution

In this session we will show how to build a text classifier using the Apache Lucene/Solr with libSVM libraries. We classify our corpus of job offers into a number of predefined categories. Each indexed document (a job offer) then belongs to zero, one or more categories. Known machine learning techniques for text classification include naïve bayes model, logistic regression, neural network, support vector machine (SVM), etc. We use Lucene/Solr to construct the features vector. Then we use the libsvm library known as the reference implementation of the SVM model to classify the document. We construct as many one-vs-all svm classifiers as there are classes in our setting, then using the Hadoop MapReduce Framework we reconcile the result of our classifiers. The end result is a scalable multi-class classifier. Finally we outline how the classifier is used to enrich basic solr keyword search.

Text Classification with Lucene/Solr, Apache Hadoop and LibSVM

lucenerevolution

Faceted search is a powerful technique to let users easily navigate the search results. It can also be used to develop rich user interfaces, which give an analyst quick insights about the documents space. In this session I will introduce the Facets module, how to use it, under-the-hood details as well as optimizations and best practices. I will also describe advanced faceted search capabilities with Lucene Facets.

Faceted Search with Lucene

lucenerevolution

Presented by Shai Erera, Researcher, IBM Lucene's arsenal has recently expanded to include two new modules: Index Sorting and Replication. Index sorting lets you keep an index consistently sorted based on some criteria (e.g. modification date). This allows for efficient search early-termination as well as achieve better index compression. Index replication lets you replicate a search index to achieve high-availability, fault tolerance as well as take hot index backups. In this talk we will introduce these modules, discuss implementation and design details as well as best practices.

Recent Additions to Lucene Arsenal

lucenerevolution

As part of their work with large media monitoring companies, Flax has developed a technique for applying tens of thousands of stored Lucene queries to a document in under a second. We'll talk about how we built intelligent filters to reduce the number of actual queries applied and how we extended Lucene to extract the exact hit positions of matches, the challenges of implementation, and how it can be used, including applications that monitor hundreds of thousands of news stories every day.

Turning search upside down

lucenerevolution

Presented by Xavier Sanchez Loro, Ph.D, Trovit Search SL This session aims to explain the implementation and use case for spellchecking in Trovit search engine. Trovit is a classified ads search engine supporting several different sites, one for each on country and vertical. Our search engine supports multiple indexes in multiple languages, each with several millions of indexed ads. Those indexes are segmented in several different sites depending on the type of ads (homes, cars, rentals, products, jobs and deals). We have developed a multi-language spellchecking system using solr and lucene in order to help our users to better find the desired ads and avoid the dreaded 0 results as much as possible. As such our goal is not pure orthographic correction, but also suggestion of correct searches for a certain site.

Spellchecking in Trovit: Implementing a Contextual Multi-language Spellchecke...

lucenerevolution

Shrinking the haystack wes caldwell - final

lucenerevolution

Presented by Mark Miller, Software Developer, Cloudera Apache Lucene/Solr committer Mark Miller talks about how Solr has been integrated into the Hadoop ecosystem to provide full text search at "Big Data" scale. This talk will give an overview of how Cloudera has tackled integrating Solr into the Hadoop ecosystem and highlights some of the design decisions and future plans. Learn how Solr is getting 'cozy' with Hadoop, which contributions are going to what project, and how you can take advantage of these integrations to use Solr efficiently at "Big Data" scale. Learn how you can run Solr directly on HDFS, build indexes with Map/Reduce, load Solr via Flume in 'Near Realtime' and much more.

The First Class Integration of Solr with Hadoop

lucenerevolution

Presented by Rajini Maski, Senior Software Engineer, Happiest Minds Technologies An important problem with document-search in any content management system (CMS) is the handling of permission-based search requests for each user. In this session, we present an algorithm and framework that allows the Search Engine to plainly index both public and privileged documents without any early binding overhead—thus enforcing document-level security policies only at the time of search. With our late-binding approach for ACL (access control lists) and some custom components, we have achieved reduction in search-time overhead. We will also discuss the order of complexity and execution time for the search overhead.

A Novel methodology for handling Document Level Security in Search Based Appl...

lucenerevolution

Presented by Hien Luu, Technical Lead, LinkedIn Rajasekaran Rangaswamy, LinkedIn For internet companies, marketing campaigns play an important role in acquiring new customers, retaining and engaging existing customers, and promoting new products. The LinkedIn segmentation and targeting platform helps marketing teams to easily and quickly create member segments based on member attributes using nested predicate expressions ranging from simple to complex. Once segments are created, then those qualified members are targeted with marketing campaigns. Lucene is a key piece of technology in this platform. This session will cover how we leverage Hadoop to efficiently build Lucene indexes for a large and growing member attribute data set of 225 million members, and how Lucene is used to create segments based on complex nested predicate expressions. This presentation will also share some of the lessons we learned and challenges we encountered from using Lucene to search over large data sets.

How Lucene Powers the LinkedIn Segmentation and Targeting Platform

lucenerevolution

Presented by Stefan Pohl, Senior Research Engineer, HERE, a Nokia Business Besides the quality of results, the time that it takes from the submission of a query to the display of results is of utmost importance to user satisfaction. Within search engine implementations such as Apache Lucene, significant development efforts are hence directed towards reducing query latency. In this session, I will explain reasons for high query latencies and describe general approaches and recent developments within Lucene to counter them.To make the presented material relevant to a wider audience, I will focus on the actual query processing, as this is at the core of every query and search use-case.

Query Latency Optimization with Lucene

lucenerevolution

10 keys to Solr's Future

lucenerevolution

Presented by Julien Nioche, Director, DigitalPebble This session will give an overview of Apache Nutch. I will describe its main components and how it fits with other Apache projects such as Hadoop, SOLR, Tika or HBase. The second part of the presentation will be focused on the latest developments in Nutch, the differences between the 1.x and 2.x branch and what we can expect to see in Nutch in the future. This session will cover many practical aspects and should be a good starting point to crawling on a large scale with Apache Nutch and SOLR.

Large Scale Crawling with Apache Nutch and Friends

lucenerevolution

Mehr von lucenerevolution (20)

Administering and Monitoring SolrCloud Clusters

Implementing a Custom Search Syntax using Solr, Lucene, and Parboiled

Using Solr to Search and Analyze Logs

Enhancing relevancy through personalization & semantic search

Real-time Inverted Search in the Cloud Using Lucene and Storm

Solr's Admin UI - Where does the data come from?

Schemaless Solr and the Solr Schema REST API

High Performance JSON Search and Relational Faceted Browsing with Lucene

Text Classification with Lucene/Solr, Apache Hadoop and LibSVM

Faceted Search with Lucene

Recent Additions to Lucene Arsenal

Turning search upside down

Spellchecking in Trovit: Implementing a Contextual Multi-language Spellchecke...

Shrinking the haystack wes caldwell - final

The First Class Integration of Solr with Hadoop

A Novel methodology for handling Document Level Security in Search Based Appl...

How Lucene Powers the LinkedIn Segmentation and Targeting Platform

Query Latency Optimization with Lucene

10 keys to Solr's Future

Large Scale Crawling with Apache Nutch and Friends

Kürzlich hochgeladen

Building Digital Trust in a Digital Economy Veronica Tan, Director - Cyber Security Agency of Singapore Apidays Singapore 2024: Connecting Customers, Business and Technology (April 17 & 18, 2024) ------ Check out our conferences at https://www.apidays.global/ Do you want to sponsor or talk at one of our conferences? https://apidays.typeform.com/to/ILJeAaV8 Learn more on APIscene, the global media made by the community for the community: https://www.apiscene.io Explore the API ecosystem with the API Landscape: https://apilandscape.apiscene.io/

Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...

apidays

The 7 Things I Know About Cyber Security After 25 Years | April 2024

Rafal Los

Scaling API-first – The story of a global engineering organization

Radu Cotescu

This presentations targets students or working professionals. You may know Google for search, YouTube, Android, Chrome, and Gmail, but did you know Google has many developer tools, platforms & APIs? This comprehensive yet still high-level overview outlines the most impactful tools for where to run your code, store & analyze your data. It will also inspire you as to what's possible. This talk is 50 minutes in length.

Powerful Google developer tools for immediate impact! (2023-24 C)

wesley chun

CNv6 Instructor Chapter 6 Quality of Service

giselly40

Automating Google Workspace (GWS) & more with Apps Script

wesley chun

Abhishek Deb(1), Mr Abdul Kalam(2) M. Des (UX) , School of Design, DIT University , Dehradun. This paper explores the future potential of AI-enabled smartphone processors, aiming to investigate the advancements, capabilities, and implications of integrating artificial intelligence (AI) into smartphone technology. The research study goals consist of evaluating the development of AI in mobile phone processors, analyzing the existing state as well as abilities of AI-enabled cpus determining future patterns as well as chances together with reviewing obstacles as well as factors to consider for more growth.

Exploring the Future Potential of AI-Enabled Smartphone Processors

debabhi2

The Raspberry Pi 5 was announced on October 2023. This new version of the popular embedded device comes with a new iteration of Broadcom’s VideoCore GPU platform, and was released with a fully open source driver stack, developed by Igalia. The presentation will discuss some of the major changes required to support this new Video Core iteration, the challenges we faced in the process and the solutions we provided in order to deliver conformant OpenGL ES and Vulkan drivers. The talk will also cover the next steps for the open source Raspberry Pi 5 graphics stack. (c) Embedded Open Source Summit 2024 April 16-18, 2024 Seattle, Washington (US) https://events.linuxfoundation.org/embedded-open-source-summit/ https://eoss24.sched.com/event/1aBEx

Raspberry Pi 5: Challenges and Solutions in Bringing up an OpenGL/Vulkan Driv...

Igalia

BooK Now Call us at +918448380779 to hire a gorgeous and seductive call girl for sex. Take a Delhi Escort Service. The help of our escort agency is mostly meant for men who want sexual Indian Escorts In Delhi NCR. It should be noted that any impersonator will get 100 attention from our Young Girls Escorts in Delhi. They will assume the position of reliable allies. VIP Call Girl With Original Photos Book Tonight +918448380779 Our Cheap Price 1 Hour not available 2 Hours 5000 Full Night 8000 TAG: Call Girls in Delhi, Noida, Gurgaon, Ghaziabad, Connaught Place, Greater Kailash Delhi, Lajpat Nagar Delhi, Mayur Vihar Delhi, Chanakyapuri Delhi, New Friends Colony Delhi, Majnu Ka Tilla, Karol Bagh, Malviya Nagar, Saket, Khan Market, Noida Sector 18, Noida Sector 76, Noida Sector 51, Gurgaon Mg Road, Iffco Chowk Gurgaon, Rajiv Chowk Gurgaon All Delhi Ncr Free Home Deliver

08448380779 Call Girls In Greater Kailash - I Women Seeking Men

Delhi Call girls

Enterprise Knowledge’s Urmi Majumder, Principal Data Architecture Consultant, and Fernando Aguilar Islas, Senior Data Science Consultant, presented "Driving Behavioral Change for Information Management through Data-Driven Green Strategy" on March 27, 2024 at Enterprise Data World (EDW) in Orlando, Florida. In this presentation, Urmi and Fernando discussed a case study describing how the information management division in a large supply chain organization drove user behavior change through awareness of the carbon footprint of their duplicated and near-duplicated content, identified via advanced data analytics. Check out their presentation to gain valuable perspectives on utilizing data-driven strategies to influence positive behavioral shifts and support sustainability initiatives within your organization. In this session, participants gained answers to the following questions: - What is a Green Information Management (IM) Strategy, and why should you have one? - How can Artificial Intelligence (AI) and Machine Learning (ML) support your Green IM Strategy through content deduplication? - How can an organization use insights into their data to influence employee behavior for IM? - How can you reap additional benefits from content reduction that go beyond Green IM?

Driving Behavioral Change for Information Management through Data-Driven Gree...

Enterprise Knowledge

Slack Application Development 101 Slides

praypatel2

Histor y of HAM Radio presentation slide

vu2urc

With more memory available, system performance of three Dell devices increased, which can translate to a better user experience Conclusion When your system has plenty of RAM to meet your needs, you can efficiently access the applications and data you need to finish projects and to-do lists without sacrificing time and focus. Our test results show that with more memory available, three Dell PCs delivered better performance and took less time to complete the Procyon Office Productivity benchmark. These advantages translate to users being able to complete workflows more quickly and multitask more easily. Whether you need the mobility of the Latitude 5440, the creative capabilities of the Precision 3470, or the high performance of the OptiPlex Tower Plus 7010, configuring your system with more RAM can help keep processes running smoothly, enabling you to do more without compromising performance.

Boost PC performance: How more available memory can improve productivity

Principled Technologies

🐬 The future of MySQL is Postgres 🐘

RTylerCroy

Bajaj Allianz Life Insurance Company - Insurer Innovation Award 2024

The Digital Insurer

Created by Mozilla Research in 2012 and now part of Linux Foundation Europe, the Servo project is an experimental rendering engine written in Rust. It combines memory safety and concurrency to create an independent, modular, and embeddable rendering engine that adheres to web standards. Stewardship of Servo moved from Mozilla Research to the Linux Foundation in 2020, where its mission remains unchanged. After some slow years, in 2023 there has been renewed activity on the project, with a roadmap now focused on improving the engine’s CSS 2 conformance, exploring Android support, and making Servo a practical embeddable rendering engine. In this presentation, Rakhi Sharma reviews the status of the project, our recent developments in 2023, our collaboration with Tauri to make Servo an easy-to-use embeddable rendering engine, and our plans for the future to make Servo an alternative web rendering engine for the embedded devices industry. (c) Embedded Open Source Summit 2024 April 16-18, 2024 Seattle, Washington (US) https://events.linuxfoundation.org/embedded-open-source-summit/ https://ossna2024.sched.com/event/1aBNF/a-year-of-servo-reboot-where-are-we-now-rakhi-sharma-igalia

A Year of the Servo Reboot: Where Are We Now?

Igalia

Explore 'The Codex of Business: Writing Software for Real-World Solutions,' a compelling SlideShare presentation that delves into digital transformation in healthcare. Discover through a detailed case study how Agile methodologies empower healthcare providers to develop, iterate, and refine digital solutions that address real-world challenges. Learn how strategic planning, user feedback, and continuous improvement drive success in deploying technologies that enhance patient care and operational efficiency. Ideal for healthcare professionals, IT specialists, and digital transformation advocates seeking actionable insights and practical examples of technology making a real difference.

The Codex of Business Writing Software for Real-World Solutions 2.pptx

Malak Abu Hammad

A Domino Admins Adventures (Engage 2024)

Gabriella Davis

Workshop - Best of Both Worlds_ Combine KG and Vector search for enhanced R...

Neo4j

GenCyber Cyber Security Day Presentation

Michael W. Hawkins

Kürzlich hochgeladen (20)

Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...

The 7 Things I Know About Cyber Security After 25 Years | April 2024

Scaling API-first – The story of a global engineering organization

Powerful Google developer tools for immediate impact! (2023-24 C)

CNv6 Instructor Chapter 6 Quality of Service

Automating Google Workspace (GWS) & more with Apps Script

Exploring the Future Potential of AI-Enabled Smartphone Processors

Raspberry Pi 5: Challenges and Solutions in Bringing up an OpenGL/Vulkan Driv...

08448380779 Call Girls In Greater Kailash - I Women Seeking Men

Driving Behavioral Change for Information Management through Data-Driven Gree...

Slack Application Development 101 Slides

Histor y of HAM Radio presentation slide

Boost PC performance: How more available memory can improve productivity

🐬 The future of MySQL is Postgres 🐘

Bajaj Allianz Life Insurance Company - Insurer Innovation Award 2024

A Year of the Servo Reboot: Where Are We Now?

The Codex of Business Writing Software for Real-World Solutions 2.pptx

A Domino Admins Adventures (Engage 2024)

Workshop - Best of Both Worlds_ Combine KG and Vector search for enhanced R...

GenCyber Cyber Security Day Presentation

How is the Government Spending Your Money? How GCE is Using Lucene and the GCE Big Data Cloud

1. Lucene in the Cloud: Leveraging the Power of Search and Big Data to Shed Light on Government Spending Seshubabu Simhadri Chief Technology Officer, GCE Confidential, Do Not Disclose. Property of Global Computer Enterprises, Inc..

2. Background What is USASpending.gov? Moving to Our Big Data cloud Some of the design decisions Tool Selection Cluster Design Hardware Design Limitations and enhancements Overview Confidential, Do Not Disclose. Property of Global Computer Enterprises, Inc..

3. What is USASpending.gov? Confidential, Do Not Disclose. Property of Global Computer Enterprises, Inc..

4. U.S. Government Spending vs. Other Entities Confidential, Do Not Disclose. Property of Global Computer Enterprises, Inc..

5. Distribution of U.S. Government Spending Confidential, Do Not Disclose. Property of Global Computer Enterprises, Inc..

6. • Analytics •  Stats •  Top-K • Free Text Search (With auto Suggestions) • Large Data Feeds • APIs What can users do on the site? Confidential, Do Not Disclose. Property of Global Computer Enterprises, Inc..

7. • Public • Media • Congress • Value Added Resellers Who are the users of the site? Confidential, Do Not Disclose. Property of Global Computer Enterprises, Inc..

8. Leveraging the industry leading open source platform to deliver cost savings and scalability within a Cloud computing model GCE Big Data and Analytics Cloud Confidential, Do Not Disclose. Property of Global Computer Enterprises, Inc..

9. •  Hadoop − For indexing and downloads Start by •  Distributed Solr Looking at − Analytics the Usual − Free text search Suspects •  Drupal static content •  Visualization What’s Inside the GCE Cloud? Confidential, Do Not Disclose. Property of Global Computer Enterprises, Inc..

10. The greatest challenge is how to optimally design a node – which combination of CPUs, memory, and shard size delivers the desired performance? Solr Node Sizing Confidential, Do Not Disclose. Property of Global Computer Enterprises, Inc..

11. Multiple index types Different types of spending Varying sizes Break complete dataset into shards as small as required to meet the response times Choose shard size based on response times Single Core with multiple cores or Multiple Solr instances each with single core? Solr Node Sizing Confidential, Do Not Disclose. Property of Global Computer Enterprises, Inc..

12. How do you design the cluster – which ones are individual nodes and which ones are aggregators? Solr Cluster Design Confidential, Do Not Disclose. Property of Global Computer Enterprises, Inc..

13. Should all shards be treated equal? Userà Aggregator Nodes à Shards Different requirements for nodes collecting the data and nodes serving a specific dataset Aggregator Node 1,2,3 ….m Large Solr Instances, No local index Shard Nodes 1,2,3,..100..n Small Solr Instance with index Solr Cluster Design Confidential, Do Not Disclose. Property of Global Computer Enterprises, Inc..

14. Separate Solr instances Multiple hard drives per server Solid state disks Infiniband What configuration did we choose? Confidential, Do Not Disclose. Property of Global Computer Enterprises, Inc..

15. Enhanced Faceting: Enabling aggregation by more than one field Will be contributed to Solr project Solr Enhancements Confidential, Do Not Disclose. Property of Global Computer Enterprises, Inc..

16. When the shards increase, management of SQLs inside Solr becomes a challenge External Data Importer Using Hadoop Solr Data Importer: Why Not? Confidential, Do Not Disclose. Property of Global Computer Enterprises, Inc..

17. Solr in the Cloud required building a cost effective and high performance infrastructure Small vs. large Commodity servers Utilizing Large Commodity Servers Confidential, Do Not Disclose. Property of Global Computer Enterprises, Inc..

18. Failure of one node results in failure of multiple shards - careful design is required Disadvantages of higher capacity servers Confidential, Do Not Disclose. Property of Global Computer Enterprises, Inc..

19. Sharded architecture Multiple Solr instances per server each handling small datasets Aggregator nodes + shards Hadoop for data indexing and data feeds Large Commodity Servers •  48-core •  256GB RAM •  SSD •  Infiniband Summary Confidential, Do Not Disclose. Property of Global Computer Enterprises, Inc..

20. Come build the future of Big Data GCECloud.com We’re hiring! Confidential, Do Not Disclose. Property of Global Computer Enterprises, Inc..

21. Questions? ssimhadri at GCECloud.com Visit us at www.GCECloud.com

How is the Government Spending Your Money? How GCE is Using Lucene and the GCE Big Data Cloud

Empfohlen

Empfohlen

Weitere ähnliche Inhalte

Was ist angesagt?

Was ist angesagt? (19)

Andere mochten auch

Andere mochten auch (8)

Ähnlich wie How is the Government Spending Your Money? How GCE is Using Lucene and the GCE Big Data Cloud

Ähnlich wie How is the Government Spending Your Money? How GCE is Using Lucene and the GCE Big Data Cloud (20)

Mehr von lucenerevolution

Mehr von lucenerevolution (20)

Kürzlich hochgeladen

Kürzlich hochgeladen (20)

How is the Government Spending Your Money? How GCE is Using Lucene and the GCE Big Data Cloud