Elasticsearch Introduction at BigData meetup
â˘
13 gefällt mirâ˘26,858 views
Global introduction to elastisearch presented at BigData meetup. Use cases, getting started, Rest CRUD API, Mapping, Search API, Query DSL with queries and filters, Analyzers, Analytics with facets and aggregations, Percolator, High Availability, Clients & Integrations, ...
Empfohlen
Weitere ähnliche Inhalte
Was ist angesagt?
Was ist angesagt? (19)
Andere mochten auch
Andere mochten auch (17)
Ăhnlich wie Elasticsearch Introduction at BigData meetup
Ăhnlich wie Elasticsearch Introduction at BigData meetup (20)
Mehr von Eric Rodriguez (Hiring in Lex)
Mehr von Eric Rodriguez (Hiring in Lex) (10)
KĂźrzlich hochgeladen
KĂźrzlich hochgeladen (20)
Elasticsearch Introduction at BigData meetup
- 1. Introduction to Elasticsearch 27th May 2014 - BigData Meetup Eric Rodriguez @wavyx
- 2. About Me Eric Rodriguez Founder of data.be ! ⢠Web entrepreneur ⢠Data addict ⢠Multi-Language: PHP, Java/ Groovy/Grails, .Net, ⌠be.linkedin.com/in/erodriguez ! github.com/wavyx ! @wavyx
- 3. Elasticsearch - Company ⢠Founded in 2012 => http://www.elasticsearch.com ⢠Professional services ⢠Training ⢠Consultancy / Development support ⢠Production support subscription (3 levels of SLAs)
- 5. (M)ELK Stack ⢠Elasticsearch - Search server based on Lucene ⢠Logstash -Tool for managing events and logs ⢠Kibana -Visualize logs and time-stamped data ⢠Marvel - Monitor your clusterâs heartbeat You Know, for SearchâŚ
- 6. Logstash ⢠Collect, parse, index, and search logs
- 7. Kibana ⢠A versatile dashboard to see and interact with your data
- 8. Marvel ⢠Monitor the health of your cluster⨠cluster-wide metrics, overview of all nodes and indices and events (master election, new nodes)
- 9. real time, search and analytics engine open-source Lucene JSON schema free document⨠store RESTful API documentation scalability high availability distributed multi tenancy per-operation⨠persistence
- 10. Use Cases ⢠Full-Text Search ⢠Data Store ⢠Analytics ⢠Alerts ⢠Ads ⢠âŚ
- 11. Copyright 2014 Elasticsearch Inc / Elasticsearch BV.All rights reserved. Content used with permission from Elasticsearch.
- 12. Copyright 2014 Elasticsearch Inc / Elasticsearch BV.All rights reserved. Content used with permission from Elasticsearch.
- 13. Copyright 2014 Elasticsearch Inc / Elasticsearch BV.All rights reserved. Content used with permission from Elasticsearch.
- 14. Copyright 2014 Elasticsearch Inc / Elasticsearch BV.All rights reserved. Content used with permission from Elasticsearch.
- 15. Copyright 2014 Elasticsearch Inc / Elasticsearch BV.All rights reserved. Content used with permission from Elasticsearch.
- 16. Copyright 2014 Elasticsearch Inc / Elasticsearch BV.All rights reserved. Content used with permission from Elasticsearch.
- 17. Copyright 2014 Elasticsearch Inc / Elasticsearch BV.All rights reserved. Content used with permission from Elasticsearch.
- 18. Elasticsearch core ⢠Apache Lucene is a high-performance, full-featured text search engine library written entirely in Java ⢠Elasticsearch added value: âSimple is bestâ ⢠Simple API (with documentation) ⢠JSON & RESTful ⢠Sharding & Replication ⢠Extensibility: plugins and scripts ⢠Interoperability: clients and integrations
- 19. Terms for DBAs ⢠Index ⢠Type ⢠Document ⢠Fields ⢠Mapping ElasticsearchRDBMs ⢠Database ⢠Table ⢠Row ⢠Column ⢠Schema
- 20. Plug & Play ⢠Zero conďŹguration ⢠4 LoC to get started ;)
- 22. REST ⢠Check your cluster, node, and index health, status, and statistics ⢠Administer your cluster, node, and index data and metadata ⢠Perform CRUD (Create, Read, Update, and Delete) and search operations against your indexes ⢠Execute advanced search operations such as paging, sorting, ďŹltering, scripting, faceting, aggregations, and many others
- 23. Basic Operations 1/3 ⢠Add a document ⢠Create index
- 24. Basic Operations 2/3 ⢠Modify/Replace a document ⢠Delete a document ⢠Delete index
- 25. Basic Operations 3/3 ⢠Update a document
- 26. Mapping 1/2 ⢠DeďŹne how a document should be mapped (similar to schema): searchable ďŹelds, tokenization, storage, .. ⢠Explicit mapping is deďŹned on an index/type level ⢠A default mapping is automatically created
- 27. Mapping 2/2 ⢠Core types: string, integer/long, ďŹoat/double, boolean, and null ⢠Other types:Array, Object, Nested, IP, GeoPoint, GeoShape, Attachment ⢠Example
- 28. Search API 1/2 ⢠Multi-index, Multi-type ⢠Uri search - Google like⨠Operators (AND/OR), ďŹelds, sort, paging, wildcards, âŚ
- 29. Search API 2/2 ⢠Paging & Sort ⢠Fields: selection, scripts ⢠Post ďŹlter ⢠Highlighting ⢠Rescoring ⢠Explain ⢠âŚ
- 30. Query DSL ⢠âSQLâ for elasticsearch ⢠Queries should be used ⢠for full text search ⢠where the result depends on a relevance score ⢠Filters should be used ⢠for binary yes/no searches ⢠for queries on exact values
- 31. Basic Queries
- 32. Basic Filters
- 33. Analysis 1/2 ⢠Analysis is extracting âtermsâ from a given text ⢠Processing natural language to make it computer searchable ⢠ConďŹgurable registry of Analyzers that can be used ⢠to break indexed (analyzed) ďŹelds when a document is indexed ⢠to process query strings
- 34. Analysis 2/2 ⢠Analyzers are composed of ⢠a singleTokenizer (may be preceded by one or more CharFilters) ⢠zero or moreTokenFilters ⢠Default Analyzers⨠standard, pattern, whitespace, language, snowball
- 35. Copyright 2014 Elasticsearch Inc / Elasticsearch BV.All rights reserved. Content used with permission from Elasticsearch.
- 36. Analytics ⢠Aggregation of information: similar to âgroup byâ ⢠Facets ⢠Aggregated data based on a search query ⢠One-dimensional results ⢠Ex:âterm facetsâ return facetcounts for various values for a speciďŹc ďŹeld ⨠Think color, tag, category, ⌠⢠Aggregations (ES 1.0+) ⢠Nested Facets ⢠Basic Stats: mean, min, max, std dev, term counts ⢠SigniďŹcantTerms, Percentiles, Cardinality estimations
- 37. Facets ⢠not yet deprecated, but use aggregations! ⢠Various Facets⨠terms, range, histogram, date, statistical, geo distance, âŚ
- 38. Aggregations ⢠A generic powerful framework that can be divided into 2 main families: ⢠Bucketing⨠Each bucket is associated with a key and a document criterion⨠The aggregation process provides a list of buckets - each one with a set of documents that "belong" to it. ⢠Metric⨠Aggregations that keep track and compute metrics over a set of documents. ⢠Aggregations can be nested !
- 39. Bucket Aggregators ⢠global ⢠ďŹlter ⢠missing ⢠terms ⢠range ⢠date range ⢠ip range ⢠histogram ⢠date histogram ⢠geo distance ⢠geohash grid ⢠nested ⢠reverse nested ⢠top hits (version 1.3)
- 40. Metrics Aggregators ⢠count ⢠stats ⢠extended stats ⢠cardinality ⢠percentiles ⢠min ⢠max ⢠sum ⢠avg
- 41. Search for end users ⢠Suggesters - âDid you meanâ⨠Terms, Phrases, Completion, Context ⢠âMore like thisâ⨠Find documents that are "like" provided text by running it against one or more ďŹelds
- 42. Percolator ⢠Classic ES 1. Add & Index documents 2. Search with queries 3. Retrieve matching documents ⢠Percolator 1. Add & Index queries 2. Percolate documents 3. Retrieve matching queries
- 43. Why Percolate ?! ⢠Alerts: social media mentions, weather forecast, news alerts ⢠Automatic Monitoring: price monitoring, stock alerts, logs ⢠Ads: display targeted ads based on userâs search queries ⢠Enrich: percolate new documents, then add query matches as document tags
- 44. High Availability 1/2 ⢠Sharding - Write Scalability ⢠Split logical data over multiple machines & Control data ďŹows ⢠Each index has a ďŹxed number of shards ⢠Improve indexing performance ⢠Replication - Read Scalability ⢠Each shard can have 0-many replicas (dynamic setup) ⢠Removing SPOF (Single Point Of Failure) ⢠Improve search performance
- 45. High Availability 2/2 ⢠Zen Discovery ⢠Automatic discovery of nodes within a cluster and electing a master node ⢠Useful for failover and replication ⢠SpeciďŹc modules:Amazon EC2, Microsoft Azure, Google Compute Engine ⢠Snapshot & Restore module
- 46. Cluster Management ⢠Marvel - http://www.elasticsearch.org/overview/marvel/ ⢠BigDesk - http://bigdesk.org/ ⢠Paramedic - https://github.com/karmi/elasticsearch- paramedic ⢠KOPF - https://github.com/lmenezes/elasticsearch-kopf/ ⢠Elastic HQ - http://www.elastichq.org/
- 47. Clients & Integration ⢠Ecosystem: Kibana, Logstash, Marvel, Hadoop integration ⢠API Clients: Java, Javascript, Groovy, PHP, Perl, Python, .Net, Ruby, Scala, Clojure, Go, Erlang, ⌠⢠Integrations: Grails, Django, Play!, Symfony2, Carrot2, Spring, Drupal,Wordpress, ⌠⢠Rivers: CouchDB, JDBC, MongoDB, Neo4j, Redis, RabbitMQ, ActiveMQ,Amazon SQS, File System,Twitter,Wikipedia, RSS, âŚ
- 48. Fast & Furious Evolution Version 1.1⨠March 25, 2014 ⢠Cardinality Agg ⢠Percentiles Agg ⢠SigniďŹcantTerms Agg ⢠SearchTemplates ⢠Cross ďŹelds search ⢠Alias for indices & templates Version 1.2⨠May 22, 2014 ⢠Java 7 ⢠Indexing & Merging performance ⢠Aggregations performance ⢠Context suggester ⢠Deep scrolling ⢠Field value factor Benchmark API coming in 1.3 Version 1.0⨠Feb 12, 2014 ⢠Aggregations ⢠Snapshot & Restore ⢠Distributed Percolator ⢠Cat API ⢠Federated search ⢠Doc values ⢠Circuit breaker
- 49. Resources ⢠http://www.elasticsearch.org/guide/ ⢠http://www.elasticsearch.org/videos/ ⢠http://www.elasticsearchtutorial.com/ ⢠http://exploringelasticsearch.com/ ⢠http://joelabrahamsson.com/elasticsearch-101/ ⢠http://belczyk.com/2014/01/elasticsearch-recomended-learning-materials/ ⢠http://www.elasticsearch.org/guide/en/elasticsearch/reference/1.x/modules- plugins.html
- 50. Books ⢠Elasticsearch Server⨠http://www.packtpub.com/ elasticsearch-server-2e/book ⢠Elasticsearch in Action⨠http://www.manning.com/ hinman/
- 51. Books ⢠Elasticsearch Cookbook⨠http://www.packtpub.com/ elasticsearch-cookbook/book ⢠Mastering Elasticsearch⨠http://www.packtpub.com/ mastering-elasticsearch- querying-and-data-handling/ book
- 52. Books ⢠Elasticsearch -The DeďŹnitive Guide⨠http://www.elasticsearch.org/blog/elasticsearch-deďŹnitive-guide/
- 53. Thank you! eric@data.be - @wavyx be.linkedin.com/in/erodriguez - github.com/wavyx http://www.meetup.com/ElasticSearch-User-Group-Belux-Belgium-Luxembourg/