Internalizing location services with geo names

•

0 likes•1,146 views

Presented by John Marc Imbrescia, Senior Software Engineer, Etsy.com Etsy recently chose to bring our location services in house. We used the open source GeoNames data set and built the tools we needed to use that data to allow members to select their location, show translations of place names, and to feed data into our search database for local, regional, and country based searches. This talk will cover the implementation details and decisions we made along the way. How we mapped places from our old data set to the GeoNames data. The internal tools we built including a SOLR core for doing location place name autosuggest. Modifications to our Listings Search and Shop Search cores and the different ways we use location based search around the site both distance and region based using GeoNames hierarchy data. There will also be a discussion about choosing to release some of the tools we built for this project open source and the decisions behind the non-search (display etc.) related elements of the project and the tools we chose for them and why.

Education Technology

Internalizing
loca.on
services
with
GeoNames
John Marc Imbrescia
Senior Software Engineer - Etsy
Wednesday, May 1, 13

Internalizing
location services
with GeoNames
May 2 2013
Wednesday, May 1, 13

The world’s online handmade marketplace.
What is Etsy.com?
Wednesday, May 1, 13

What is Etsy.com?
•20 million unique items
•18 million daily item searches
•800,000 sellers
•28 million unique views per month
•Developer blog: codeascraft.etsy.com
•450 worldwide employees
Wednesday, May 1, 13

Our Problem
Location names were only in English
•Search based on English names
•Display and search needed to be
i18n friendly.
•API limits and speed concerns
meant we needed a new solution.
Wednesday, May 1, 13

What do we use Location for?
More than just search
•Display
•Local Search
•No Mapping
•No Bounding boxes
Wednesday, May 1, 13

What do we use Location for?
Item Search
Wednesday, May 1, 13

What do we use Location for?
Location Display
Wednesday, May 1, 13

How did this use to work?
•Yahoo API
•Every lookup was an API call
•Stored user input and API response
•Searched based on text match of API response
•Not radius using lat/lon
•No way to Internationalize
Wednesday, May 1, 13

What Services did Etsy need to
Internalize location services?
•Lookup - Autosuggest
•Update - Scripts to refresh data
•Display - Built into the php stack
•Search - Existing, modiﬁed for new pattern
Wednesday, May 1, 13

What we have now
•GeoNames as a data source
•Feeds “geonamessuggest” Solr Core
•Sqlite database for place name lookup
•GeoName IDs used for local search
•Leverages GeoName hierarchy data
•Built in Internationalization
Wednesday, May 1, 13

How did we get here?
•Mapped old locations to GeoNames
•Added Geoname ID hierarchy to listing search
•Pushed out Sqlite database to webservers
•Slowly transitioned lookup and search services
•Did side by side testing to look for anomalies
Wednesday, May 1, 13

What are the data types?
GeoNames
Wednesday, May 1, 13

Schemas
GeoNames
•775k Entries
•1.4m alternate spellings
Wednesday, May 1, 13

Geonamessuggest
Our autosuggest for place names
•Localized
•GeoIP
•Population
Wednesday, May 1, 13

Geonamessuggest
Schema
Wednesday, May 1, 13

Distance and population come ﬁrst
Sort function
Wednesday, May 1, 13

GeonameId Hierarachy
Local listing search
•Each listing gets a hierarchy of geonameids
•Local search is a ﬁlter on this ID
•Fast & Reliable
•Enables powerful functionality
•Kept old data ﬁelds
Wednesday, May 1, 13

GeonameId Collection
Local listing search
•Each listing gets a hierarchy of geonameids
•Local search is a ﬁlter query on this ID
•Fast & Reliable
•Enables powerful functionality
•Kept old data ﬁelds
Wednesday, May 1, 13

CONFERENCE PARTY
The Tipsy Crow: 770 5th Ave
Starts after Stump The Chump
Your conference badge gets you in the
door
TOMORROW
Breakfast starts at 7:30
Keynotes start at 8:30
CONTACT
John Marc Imbrescia
@thejohnmarc
johnmarc@etsy.com
Wednesday, May 1, 13

More from lucenerevolution

Presented by Daniel Beach, Search Application Developer, OpenSource Connections Solr is a powerful search engine, but creating a custom user interface can be daunting. In this fast paced session I will present an overview of how to implement a client-side search application using Solr. Using open-source frameworks like SpyGlass (to be released in September) can be a powerful way to jumpstart your development by giving you out-of-the box results views with support for faceting, autocomplete, and detail views. During this talk I will also demonstrate how we have built and deployed lightweight applications that are able to be performant under large user loads, with minimal server resources.

Building Client-side Search Applications with Solr

lucenerevolution

Presented by Timothy Potter, Founder, Text Centrix Storm is a real-time distributed computation system used to process massive streams of data. Many organizations are turning to technologies like Storm to complement batch-oriented big data technologies, such as Hadoop, to deliver time-sensitive analytics at scale. This talk introduces on an emerging architectural pattern of integrating Solr and Storm to process big data in real time. There are a number of natural integration points between Solr and Storm, such as populating a Solr index or supplying data to Storm using Solr’s real-time get support. In this session, Timothy will cover the basic concepts of Storm, such as spouts and bolts. He’ll then provide examples of how to integrate Solr into Storm to perform large-scale indexing in near real-time. In addition, we'll see how to embed Solr in a Storm bolt to match incoming tuples against pre-configured queries, commonly known as percolator. Attendees will come away from this presentation with a good introduction to stream processing technologies and several real-world use cases of how to integrate Solr with Storm.

Integrate Solr with real-time stream processing applications

lucenerevolution

Configure your Solr cluster to handle hundreds of millions of documents without even noticing, handle queries in milliseconds, use Near Real Time indexing and searching with document versioning. Scale your cluster both horizontally and vertically by using shards and replicas. In this session you'll learn how to make your indexing process blazing fast and make your queries efficient even with large amounts of data in your collections. You'll also see how to optimize your queries to leverage caches as much as your deployment allows and how to observe your cluster with Solr administration panel, JMX, and third party tools. Finally, learn how to make changes to already deployed collections —split their shards and alter their schema by using Solr API.

Scaling Solr with SolrCloud

lucenerevolution

Presented by Rafal Kuć, Consultant and Software engineer, , Sematext Group, Inc. Even though Solr can run without causing any troubles for long periods of time it is very important to monitor and understand what is happening in your cluster. In this session you will learn how to use various tools to monitor how Solr is behaving at a high level, but also on Lucene, JVM, and operating system level. You'll see how to react to what you see and how to make changes to configuration, index structure and shards layout using Solr API. We will also discuss different performance metrics to which you ought to pay extra attention. Finally, you'll learn what to do when things go awry - we will share a few examples of troubleshooting and then dissect what was wrong and what had to be done to make things work again.

Administering and Monitoring SolrCloud Clusters

lucenerevolution

In a recent project with the United States Patent and Trademark Office, Opensource Connections was asked to prototype the next generation of patent search - using Solr and Lucene. An important aspect of this project was the implementation of BRS, a specialized search syntax used by patent examiners during the examination process. In this fast paced session we will relate our experiences and describe how we used a combination of Parboiled (a Parser Expression Grammar [PEG] parser), Lucene Queries and SpanQueries, and an extension of Solr's QParserPlugin to build BRS search functionality in Solr. First we will characterize the patent search problem and then define the BRS syntax itself. We will then introduce the Parboiled parser and discuss various considerations that one must make when designing a syntax parser. Following this we will describe the methodology used to implement the search functionality in Lucene/Solr. Finally, we will include an overview our syntactic and semantic testing strategies. The audience will leave this session with an understanding of how Solr, Lucene, and Parboiled may be used to implement their own custom search parser.

Implementing a Custom Search Syntax using Solr, Lucene, and Parboiled

lucenerevolution

Many of us tend to hate or simply ignore logs, and rightfully so: they’re typically hard to find, difficult to handle, and are cryptic to the human eye. But can we make logs more valuable and more usable if we index them in Solr, so we can search and run real-time statistics on them? Indeed we can, and in this session you’ll learn how to make that happen. In the first part of the session we’ll explain why centralized logging is important, what valuable information one can extract from logs, and we’ll introduce the leading tools from the logging ecosystems everyone should be aware of - from syslog and log4j to LogStash and Flume. In the second part we’ll teach you how to use these tools in tandem with Solr. We’ll show how to use Solr in a SolrCloud setup to index large volumes of logs continuously and efficiently. Then, we'll look at how to scale the Solr cluster as your data volume grows. Finally, we'll see how you can parse your unstructured logs and convert them to nicely structured Solr documents suitable for analytical queries.

Using Solr to Search and Analyze Logs

lucenerevolution

Enhancing relevancy through personalization & semantic search

lucenerevolution

Building real-time notification systems is often limited to basic filtering and pattern matching against incoming records. Allowing users to query incoming documents using Solr's full range of capabilities is much more powerful. In our environment we needed a way to allow for tens of thousands of such query subscriptions, meaning we needed to find a way to distribute the query processing in the cloud. By creating in-memory Lucene indices from our Solr configuration, we were able to parallelize our queries across our cluster. To achieve this distribution, we wrapped the processing in a Storm topology to provide a flexible way to scale and manage our infrastructure. This presentation will describe our experiences creating this distributed, real-time inverted search notification framework.

Real-time Inverted Search in the Cloud Using Lucene and Storm

lucenerevolution

Like many Web-Applications in the past, the Solr Admin UI up until 4.0 was entirely server based. It used separate code on the server to generate their Dashboards, Overviews and Statistics. All that code had to be maintained and still ... you weren't really able to use that kind of data for the things you needed it for. It was wrapped into HTML, most of the time difficult to extract and changed the structure from time to time w/o announcement. After a short look back, we're going to look into the current state of the Solr Admin UI - a client-side application, running completely in your browser. We'll see how it works, where it gets its data from and how you can get the very same data and wire that into your own custom applications, dashboards and/oder monitoring systems.

Solr's Admin UI - Where does the data come from?

lucenerevolution

Schemaless Solr and the Solr Schema REST API

lucenerevolution

Presented by Renaud Delbru, Co-Founder, SindiceTech In this presentation, we will discuss how Lucene and Solr can be used for very efficient search of tree-shaped schemaless document, e.g. JSON or XML, and can be then made to address both graph and relational data search. We will discuss the capabilities of SIREn, a Lucene/Solr plugin we have developed to deal with huge collections of tree-shaped schemaless documents, and how SIREn is built using Lucene extensibility capabilities (Analysis, Codec, Flexible Query Parser). We will compare it with Lucene's BlockJoin Query API in nested schemaless data intensive scenarios. We will then go through use cases that show how relational or graph data can be turned into JSON documents using Hadoop and Pig, and how this can be used in conjunction with SIREn to create relational faceting systems with unprecedented performance. Take-away lessons from this session will be awareness about using Lucene/Solr and Hadoop for relational and graph data search, as well as the awareness that it is now possible to have relational faceted browsers with sub-second response time on commodity hardware.

High Performance JSON Search and Relational Faceted Browsing with Lucene

lucenerevolution

In this session we will show how to build a text classifier using the Apache Lucene/Solr with libSVM libraries. We classify our corpus of job offers into a number of predefined categories. Each indexed document (a job offer) then belongs to zero, one or more categories. Known machine learning techniques for text classification include naïve bayes model, logistic regression, neural network, support vector machine (SVM), etc. We use Lucene/Solr to construct the features vector. Then we use the libsvm library known as the reference implementation of the SVM model to classify the document. We construct as many one-vs-all svm classifiers as there are classes in our setting, then using the Hadoop MapReduce Framework we reconcile the result of our classifiers. The end result is a scalable multi-class classifier. Finally we outline how the classifier is used to enrich basic solr keyword search.

Text Classification with Lucene/Solr, Apache Hadoop and LibSVM

lucenerevolution

Faceted search is a powerful technique to let users easily navigate the search results. It can also be used to develop rich user interfaces, which give an analyst quick insights about the documents space. In this session I will introduce the Facets module, how to use it, under-the-hood details as well as optimizations and best practices. I will also describe advanced faceted search capabilities with Lucene Facets.

Faceted Search with Lucene

lucenerevolution

Presented by Shai Erera, Researcher, IBM Lucene's arsenal has recently expanded to include two new modules: Index Sorting and Replication. Index sorting lets you keep an index consistently sorted based on some criteria (e.g. modification date). This allows for efficient search early-termination as well as achieve better index compression. Index replication lets you replicate a search index to achieve high-availability, fault tolerance as well as take hot index backups. In this talk we will introduce these modules, discuss implementation and design details as well as best practices.

Recent Additions to Lucene Arsenal

lucenerevolution

As part of their work with large media monitoring companies, Flax has developed a technique for applying tens of thousands of stored Lucene queries to a document in under a second. We'll talk about how we built intelligent filters to reduce the number of actual queries applied and how we extended Lucene to extract the exact hit positions of matches, the challenges of implementation, and how it can be used, including applications that monitor hundreds of thousands of news stories every day.

Turning search upside down

lucenerevolution

Presented by Xavier Sanchez Loro, Ph.D, Trovit Search SL This session aims to explain the implementation and use case for spellchecking in Trovit search engine. Trovit is a classified ads search engine supporting several different sites, one for each on country and vertical. Our search engine supports multiple indexes in multiple languages, each with several millions of indexed ads. Those indexes are segmented in several different sites depending on the type of ads (homes, cars, rentals, products, jobs and deals). We have developed a multi-language spellchecking system using solr and lucene in order to help our users to better find the desired ads and avoid the dreaded 0 results as much as possible. As such our goal is not pure orthographic correction, but also suggestion of correct searches for a certain site.

Spellchecking in Trovit: Implementing a Contextual Multi-language Spellchecke...

lucenerevolution

Shrinking the haystack wes caldwell - final

lucenerevolution

Presented by Mark Miller, Software Developer, Cloudera Apache Lucene/Solr committer Mark Miller talks about how Solr has been integrated into the Hadoop ecosystem to provide full text search at "Big Data" scale. This talk will give an overview of how Cloudera has tackled integrating Solr into the Hadoop ecosystem and highlights some of the design decisions and future plans. Learn how Solr is getting 'cozy' with Hadoop, which contributions are going to what project, and how you can take advantage of these integrations to use Solr efficiently at "Big Data" scale. Learn how you can run Solr directly on HDFS, build indexes with Map/Reduce, load Solr via Flume in 'Near Realtime' and much more.

The First Class Integration of Solr with Hadoop

lucenerevolution

Presented by Rajini Maski, Senior Software Engineer, Happiest Minds Technologies An important problem with document-search in any content management system (CMS) is the handling of permission-based search requests for each user. In this session, we present an algorithm and framework that allows the Search Engine to plainly index both public and privileged documents without any early binding overhead—thus enforcing document-level security policies only at the time of search. With our late-binding approach for ACL (access control lists) and some custom components, we have achieved reduction in search-time overhead. We will also discuss the order of complexity and execution time for the search overhead.

A Novel methodology for handling Document Level Security in Search Based Appl...

lucenerevolution

Presented by Hien Luu, Technical Lead, LinkedIn Rajasekaran Rangaswamy, LinkedIn For internet companies, marketing campaigns play an important role in acquiring new customers, retaining and engaging existing customers, and promoting new products. The LinkedIn segmentation and targeting platform helps marketing teams to easily and quickly create member segments based on member attributes using nested predicate expressions ranging from simple to complex. Once segments are created, then those qualified members are targeted with marketing campaigns. Lucene is a key piece of technology in this platform. This session will cover how we leverage Hadoop to efficiently build Lucene indexes for a large and growing member attribute data set of 225 million members, and how Lucene is used to create segments based on complex nested predicate expressions. This presentation will also share some of the lessons we learned and challenges we encountered from using Lucene to search over large data sets.

How Lucene Powers the LinkedIn Segmentation and Targeting Platform

lucenerevolution

More from lucenerevolution (20)

Building Client-side Search Applications with Solr

Integrate Solr with real-time stream processing applications

Scaling Solr with SolrCloud

Administering and Monitoring SolrCloud Clusters

Implementing a Custom Search Syntax using Solr, Lucene, and Parboiled

Using Solr to Search and Analyze Logs

Enhancing relevancy through personalization & semantic search

Real-time Inverted Search in the Cloud Using Lucene and Storm

Solr's Admin UI - Where does the data come from?

Schemaless Solr and the Solr Schema REST API

High Performance JSON Search and Relational Faceted Browsing with Lucene

Text Classification with Lucene/Solr, Apache Hadoop and LibSVM

Faceted Search with Lucene

Recent Additions to Lucene Arsenal

Turning search upside down

Spellchecking in Trovit: Implementing a Contextual Multi-language Spellchecke...

Shrinking the haystack wes caldwell - final

The First Class Integration of Solr with Hadoop

A Novel methodology for handling Document Level Security in Search Based Appl...

How Lucene Powers the LinkedIn Segmentation and Targeting Platform

Recently uploaded

Food safety_Challenges food safety laboratories_.pdf

Sherif Taha

Activity 01 - Artificial Culture (1).pdf

ciinovamais

ICT Role in 21st Century Education & its Challenges.pptx

AreebaZafar22

This PowerPoint helps students to consider the concept of infinity.

christianmathematics

Python Notes for mca i year students osmania university.docx

Ramakrishna Reddy Bijjam

Unit-IV; Professional Sales Representative (PSR).pptx

VishalSingh1417

Fostering Friendships - Enhancing Social Bonds in the Classroom

Pooky Knightsmith

Kodo Millet PPT made by Ghanshyam bairwa college of Agriculture kumher bhara...

pradhanghanshyam7136

HMCS Max Bernays Pre-Deployment Brief (May 2024).pptx

Esquimalt MFRC

Wizards are very useful for creating a good user experience. In all businesses, interactive sessions are most beneficial. To improve the user experience, wizards in Odoo provide an interactive session. For creating wizards, we can use transient models or abstract models. This gives features of a model class except the data storing. Transient and abstract models have permanent database persistence. For them, database tables are made, and the records in such tables are kept until they are specifically erased.

How to Create and Manage Wizard in Odoo 17

Celine George

Mehran University Newsletter Vol-X, Issue-I, 2024

Mehran University of Engineering & Technology, Jamshoro

Vishram Singh - Textbook of Anatomy Upper Limb and Thorax.. Volume 1 (1).pdf

ssuserdda66b

Making communications land - Are they received and understood as intended? webinar Thursday 2 May 2024 A joint webinar created by the APM Enabling Change and APM People Interest Networks, this is the third of our three part series on Making Communications Land. presented by Ian Cribbes, Director, IMC&T Ltd @cribbesheet The link to the write up page and resources of this webinar: https://www.apm.org.uk/news/making-communications-land-are-they-received-and-understood-as-intended-webinar/ Content description: How do we ensure that what we have communicated was received and understood as we intended and how do we course correct if it has not.

Making communications land - Are they received and understood as intended? we...

Association for Project Management

Graduate Outcomes Presentation Slides - English

neillewis46

Towards a code of practice for AI in AT.pptx

Jisc

Single or Multiple melodic lines structure

dhanjurrannsibayan2

Micro-Scholarship, What it is, How can it help me.pdf

Poh-Sun Goh

Unit-V; Pricing (Pharma Marketing Management).pptx

VishalSingh1417

Salient Features of India constitution especially power and functions

KarakKing

Google Gemini An AI Revolution in Education.pptx

Dr. Sarita Anand

Recently uploaded (20)

Food safety_Challenges food safety laboratories_.pdf

Activity 01 - Artificial Culture (1).pdf

ICT Role in 21st Century Education & its Challenges.pptx

This PowerPoint helps students to consider the concept of infinity.

Python Notes for mca i year students osmania university.docx

Unit-IV; Professional Sales Representative (PSR).pptx

Fostering Friendships - Enhancing Social Bonds in the Classroom

Kodo Millet PPT made by Ghanshyam bairwa college of Agriculture kumher bhara...

HMCS Max Bernays Pre-Deployment Brief (May 2024).pptx

How to Create and Manage Wizard in Odoo 17

Mehran University Newsletter Vol-X, Issue-I, 2024

Vishram Singh - Textbook of Anatomy Upper Limb and Thorax.. Volume 1 (1).pdf

Making communications land - Are they received and understood as intended? we...

Graduate Outcomes Presentation Slides - English

Towards a code of practice for AI in AT.pptx

Single or Multiple melodic lines structure

Micro-Scholarship, What it is, How can it help me.pdf

Unit-V; Pricing (Pharma Marketing Management).pptx

Salient Features of India constitution especially power and functions

Google Gemini An AI Revolution in Education.pptx

Internalizing location services with geo names

1. Internalizing loca.on services with GeoNames John Marc Imbrescia Senior Software Engineer - Etsy Wednesday, May 1, 13

2. Internalizing location services with GeoNames May 2 2013 Wednesday, May 1, 13

3. The world’s online handmade marketplace. What is Etsy.com? Wednesday, May 1, 13

4. What is Etsy.com? •20 million unique items •18 million daily item searches •800,000 sellers •28 million unique views per month •Developer blog: codeascraft.etsy.com •450 worldwide employees Wednesday, May 1, 13

5. Our Problem Location names were only in English •Search based on English names •Display and search needed to be i18n friendly. •API limits and speed concerns meant we needed a new solution. Wednesday, May 1, 13

6. What do we use Location for? More than just search •Display •Local Search •No Mapping •No Bounding boxes Wednesday, May 1, 13

7. What do we use Location for? Item Search Wednesday, May 1, 13

8. What do we use Location for? Item Search Wednesday, May 1, 13

9. What do we use Location for? Item Search Wednesday, May 1, 13

10. What do we use Location for? Location Display Wednesday, May 1, 13

11. How did this use to work? •Yahoo API •Every lookup was an API call •Stored user input and API response •Searched based on text match of API response •Not radius using lat/lon •No way to Internationalize Wednesday, May 1, 13

12. What Services did Etsy need to Internalize location services? •Lookup - Autosuggest •Update - Scripts to refresh data •Display - Built into the php stack •Search - Existing, modiﬁed for new pattern Wednesday, May 1, 13

13. What we have now •GeoNames as a data source •Feeds “geonamessuggest” Solr Core •Sqlite database for place name lookup •GeoName IDs used for local search •Leverages GeoName hierarchy data •Built in Internationalization Wednesday, May 1, 13

14. How did we get here? •Mapped old locations to GeoNames •Added Geoname ID hierarchy to listing search •Pushed out Sqlite database to webservers •Slowly transitioned lookup and search services •Did side by side testing to look for anomalies Wednesday, May 1, 13

15. What are the data types? GeoNames Wednesday, May 1, 13

16. Schemas GeoNames •775k Entries •1.4m alternate spellings Wednesday, May 1, 13

17. Schemas GeoNames Wednesday, May 1, 13

18. Geonamessuggest Our autosuggest for place names •Localized •GeoIP •Population Wednesday, May 1, 13

19. Geonamessuggest Schema Wednesday, May 1, 13

20. Distance and population come ﬁrst Sort function Wednesday, May 1, 13

21. GeonameId Hierarachy Local listing search •Each listing gets a hierarchy of geonameids •Local search is a ﬁlter on this ID •Fast & Reliable •Enables powerful functionality •Kept old data ﬁelds Wednesday, May 1, 13

22. GeonameId Collection Local listing search •Each listing gets a hierarchy of geonameids •Local search is a ﬁlter query on this ID •Fast & Reliable •Enables powerful functionality •Kept old data ﬁelds Wednesday, May 1, 13

23. CONFERENCE PARTY The Tipsy Crow: 770 5th Ave Starts after Stump The Chump Your conference badge gets you in the door TOMORROW Breakfast starts at 7:30 Keynotes start at 8:30 CONTACT John Marc Imbrescia @thejohnmarc johnmarc@etsy.com Wednesday, May 1, 13

Internalizing location services with geo names

Recommended

Recommended

More Related Content

More from lucenerevolution

More from lucenerevolution (20)

Recently uploaded

Recently uploaded (20)

Internalizing location services with geo names