Elasticsearch presentation at the MVP Western Europe Community Open Day 2016

1. Titulo
Subtitulo
Ricardo Peres
@rjperes75

2. Introduction
NoSQL database for indexing JSON contents
Documents are indexed as they are added (< 1s)
Schema-less (kind of…)
Distributed
High performance
REST semantics
Graph capabilities
Based on Lucene
Part of the ELK stack
Open source!

3. Cluster
A collection of servers (nodes) running
Elasticsearch
Single master
Multicast based discovery (can be explicit)

4. Shards
Indexes are distributed by shards – default is 5
shards and 1 replica (cluster)
Defined at index creation time
Transparent to the user
It is possible to define a hashing function

5. Indexes
A collection of types
Similar to a database

6. Types
Collection of documents
Has a schema (implicit or explicit)
Similar to a table

7. Documents
Self-contained data
Exist in a type
Have an id
Have a version
Have a schema
Can have expiration

8. Fields
Documents are structured in fields
Special fields: _id, _uid, _index, _type
_timestamp, _all, _source, _ttl, _meta,
_parent, _routing are optional

9. Data Types
string
long, integer, short, byte,
double, float
date
boolean
binary
geo_point
geo_shape
object
nested
ip
completion
token_count
arrays

10. Creating a Document
Auto Id
POST /website/blog
{
"title" : “My Blog",
"url" : "http://my/blog",
"tags" : [ "development" ]
}
Explicit Id
POST /website/blog/1
{
"title" : "My Blog",
"url" : "http://my/blog",
"tags" : [ " development " ]
}

11. Updating a Document
Partial
POST
/website/blog/1/_update
{
"doc" :
{
"tags" : [ "testing" ],
"views": 1
}
}
Full
POST
/website/blog/1/_update
{
"title" : "My Blog",
"url" : "http://my/blog",
"tags" : [ "testing" ]
}

12. Deleting a Document
Single
DELETE /website/blog/1
Index
DELETE /website

13. Mappings
Created at index or type level implicitly or explicitly
Cannot modify, only add
Can enforce schema or not
PUT website
{
"mappings": {
"blog": {
" _timestamp”: {
"enabled" : true
},
"dynamic" : "strict",
"properties": {
"title": {
"type": "string",
"analyzer": "standard"
}
}
}
}
}

14. Mapping Templates
Automatically apply mappings to new types
PUT website
{
"mappings": {
"post": {
"dynamic_templates": [ {
"timestamp": {
"date_detection": true,
"dynamic_date_formats": [ "yyyy-MM-dd HH:mm", "yyyy-MM-dd" ],
"match": "timestamp",
"match_mapping_type": "date",
"mapping": {
"type": "date",
"format" : "yyyy-MM-dd HH:mm"
}
}
} ]
}
}
}

15. Query and Filter Context
Queries: scoring of the results
Filters determine what appears in the results
Are cached

16. Querying
Search API
 Uses the URL
Starting with <index> and <type>
is optional
/<index>/<type>/_search?q=som
ething
/<index>/<type1>,<type2>/_sear
ch?q=something
_search?q=something
_search?q:field:value
_search?q=+firstname(john
mary)&-surname:smith
Query DSL
 Query and filter context
 simple_query_string,
query_string, match, term,
terms, range, multi_match,
match_phrase, missing,
exists, regexp, fuzzy, prefix,
ids
 bool, dis_max
 more_like_this, script,
template

17. Pagination, Sorting and Projection
size, from
sort
fields
POST website/post/_search
{
“size”: 10,
“from”: 0,
“sort”: {
“timestamp”: {
“order”: “desc”
}
},
“fields”: [ “title”, “_id” ]
}

18. Percolator
 Search in reverse: first define the
query, then add documents to it
 Querying a document gives all
percolator queries that it matches

19. Relations
No joins, but some alternatives
Parent/child: has_child, has_parent
Nested objects
Terms filter lookup: terms with type and id

20. Relevance
Term Frequency/Inverse Document
Frequency/Field Length Norm
Custom scores
A match hit/miss can be explained

21. Indexing
Stemming
Tokenization
Normalization

22. Index Aliases
Used to refer to one or more indexes, one or more types, possibly with a filter
Useful for "moving indexes" (month, year, country, etc)
POST /_aliases
{
"actions" : [ {
"add" : {
"indices" : [ "social-2015", "social-2016" ],
"alias" : "social-testing",
"filter" : {
"term" : {
"tag" : "testing"
}
}
}
} ]
}

23. Alias Templates
 Creates an alias when a type is created
POST /_template/social
{
"order": 0,
"template": "social-*",
"settings": {
"index": {
"refresh_interval": "5s"
}
},
"mappings": {},
"aliases": {
"social": {}
}
}

24. Bulk Operations
 Perform multiple operations (index, update, delete) at once
POST bulk/data/_bulk
{ "index" : { "_id" : "1" } }
{ "field1" : "value1" }
{ "index" : { "_id" : "2" } }
{ "field1" : "value1" }
{ "index" : { "_id" : "3" } }
{ "field1" : "value1" }
{ "update" : { "_id" : "2" } }
{ "doc": { "field2": "value2" } }
{ "delete" : { "_id" : "3" } }

25. Analytics
 Aggregations
 Can be nested
 Can use scripts
GET /megacorp/employee/_search
{
"aggs": {
"all_interests": {
"terms": {
"field": “feature“
},
“aggs”: {
“average_price”: {
“field”: “price”
}
}
}
}
}

26. APIs
 REST (native)
 .NET
 JavaScript/Node.js
 Python
 Java
 Groovy
 PHP
 Perl
 Ruby

27. Plugins
Marvel
Sense
Watcher
Graph
Shield
ES-Hadoop
Head
Kopf
Elasticsearch-SQL

28. Kibana
Reporting
Dashboards

29. Logstash
 Collect and transform data
 Input – Filters – Outputs
 Sources/destinations:
 Elasticsearch
 File
 Syslog
 Windows Eventlog
 Redis
 RabbitMQ
 GitHub
 HTTP
 Beats
 Twitter
 WebSocket
 …

30. References
https://www.elastic.co
https://www.gitbook.com/book/allen8807/elasti
csearch-definitive-guide-en/details
https://github.com/elastic/cookbook-
elasticsearch
https://github.com/elastic/elasticsearch-net
https://github.com/elastic/kibana
https://github.com/elastic/logstash
https://github.com/elastic/elasticsearch
http://joelabrahamsson.com/elasticsearch-101

31. Thank you!
Thank you for attending!
@rjperes75
rjperes@hotmail.com
http://weblogs.asp.net/ricardoperes