1. Real-time search in Drupal.
Meet Elasticsearch
By Alexei Gorobets
asgorobets

2. Elasticsearch
Flexible and powerful open
source, distributed real-time
search and analytics engine
for the cloud

3. Why use
Elasticsearch?

4. ● RESTful API
● Open Source
● JSON over HTTP
● based on Lucene
● distributed
● highly available
● schema free
● massively scalable

6. Setup in 2 steps:
1. Extract the archive
2. > bin/elasticsearch

7. How to use it?

8. > curl -XGET localhost:9200/?pretty

9. > curl -XGET localhost:9200/?pretty
{
"ok" : true,
"status" : 200,
"name" : "Infinity",
"version" : {
"number" : "0.90.1",
"snapshot_build" : false,
"lucene_version" : "4.3"
},
"tagline" : "You Know, for Search"
}

10. > curl -XGET localhost:9200/?pretty
action (verb)

11. > curl -XGET localhost:9200/?pretty
node + port

12. > curl -XGET localhost:9200/?pretty
path

13. > curl -XGET localhost:9200/?pretty
query string

14. Let's index some data

15. > PUT /index/type/id
Where?
It's very similar to
database in SQL

16. > PUT /index/type/id
What?
Table
Content type,
Entity type,
any kind of type you decide

17. > PUT /index/type/id
Which?
Node ID,
Entity ID,
any kind of serial ID

18. > PUT /mysite/node/1 -d
{
"nid": "1",
"status": "1",
"title": "Hello elasticsearch",
"body": "First elasticsearch document"
}

19. > PUT /mysite/node/1 -d
{
"nid": "1",
"status": "1",
"title": "Hello elasticsearch",
"body": "First elasticsearch document"
}
{
"ok":true,
"_index":"mysite",
"_type":"node",
"_id":"1",
"_version":1
}

20. Let's GET some data

21. > GET /mysite/node/1
{
"_index" : "mysite",
"_type" : "node",
"_id" : "1",
"_version" : 1,
"exists" : true,
"_source" : {
"nid":"1",
"status":"1",
"title":"Hello elasticsearch",
"body":"First elasticsearch document"
}

22. > GET /mysite/node/1?fields=title,body
Get specific fields

23. > GET /mysite/node/1?fields=title,body
Get specific fields
> GET /mysite/node/1/_source
Get source only

24. Let's UPDATE some data

25. > PUT /mysite/node/1 -d
{
"status":"0"
}

26. > PUT /mysite/node/1 -d
{
"ok":true,
"_index":"mysite",
"_type":"node",
"_id":"1",
"_version":2
}
{
"status":"0"
}

27. UPDATE = DELETE + PUT

28. Let's DELETE some data

29. > DELETE /mysite/node/1

30. > DELETE /mysite/node/1
{
"ok":true,
"found":true,
"_index":"mysite",
"_type":"node",
"_id":"1",
"_version":3
}

31. Distributed, Highly Available

33. > PUT /new_index -d '{
"settings" : {
"number_of_shards" : 3,
"number_of_replicas" : 2
}
}'

34. Concurrency, Version control

35. > PUT /myapp/node/1?version=1
{
"title": "hi girl"
}

36. > PUT /myapp/node/1?version=1
{
"title": "hi girl"
}
{
"_index": "myapp",
"_type": "node",
"_id": "1",
"_version": 1,
"created": false
}

37. > PUT /myapp/node/1?version=1
{
"title": "hey boy"
}
# 200

38. > PUT /myapp/node/1?version=1
{
"title": "hey boy"
}
# 409
> version conflict, current [2], provided [1]

39. Let's SEARCH for something

40. > GET /_search

41. > GET /_search
{
"took" : 32,
"timed_out" : false,
"_shards" : {
"total" : 20,
"successful" : 20,
"failed" : 0
},
"hits" : { results... }
}

42. Let's SEARCH in multiple
indices and types

43. > GET /index/_search
> GET /index/type/_search
> GET /index1,index2/_search
> GET /myapp_*/type, entity_*/_search

44. Let's PAGINATE results

45. > GET /_search?size=10&from=20
size = results per page
from = starting from

46. Let's search oldschool

47. > GET /_search?q=title:elasticsearch
> GET /_search?q=nid:60

48. +title:awesome
+status:1
+created:[1369917354 TO *]

49. ?q=title:awesome%20%2Bcreated:
[1369917354%20TO%20*]%2Bstatus:1
+title:awesome
+status:1
+created:[1369917354 TO *]
The ugly encoding =)

50. Query DSL style

51. > GET /_search -d
{
"query": {
"match": "awesome"
}
}

52. > GET /_search -d
{
"query": {
"match" : {
"title" : {
"query" : "+awesome -poor",
"boost" : 2.0,
}
}
}
}

53. Mappings and types

54. Core types
* string
* number
* date
* boolean

55. Complex types
* array type
* object type
* nested type
Others:
ip type
geo point
geo shape
attachments

56. Define type mapping

57. > PUT /myapp/node -d
{
"node" : {
"properties" : {
"message" : {
"type" : "string",
"store" : true
}
}
}
}

58. Indexed fields

59. Full text
analyzed
== is splitted into
terms
Term
not analyzed
== is stored as is

60. > PUT /myapp/node -d
{
"node" : {
"properties" : {
"name" : {
"type" : "string",
"store" : true,
“index”: “not_analyzed”
}
}
}
}

61. Dynamic mapping

62. Analysis and indexing

63. Inverted index
1. “The quick brown fox
jumped over the lazy
dog”
2. “Quick brown foxes
leap over lazy dogs in
summer”
Term Doc_1 Doc_2
-------------------------
Quick | | X
The | X |
brown | X | X
dog | X |
dogs | | X
fox | X |
foxes | | X
in | | X
jumped | X |
lazy | X | X
leap | | X
over | X | X
quick | X |
summer | | X
the | X |

64. Analyzer
Tokenizers
● standard
● keyword
● whitespace
● ngram
TokenFilters
standard
lowercase
stop
truncate
snowball

65. > GET /_analyze?analyzer=standard -d
'this is a test baby'
{
"tokens" : [ {
"token" : "test",
"start_offset" : 10,
"end_offset" : 14,
"type" : "<ALPHANUM>",
"position" : 4
}, {
"token" : "baby",
"start_offset" : 15,
"end_offset" : 19,
"type" : "<ALPHANUM>",
"position" : 5
} ]
}

66. Autocomplete fields

67. Queries & Filters

68. Queries & Filters
full text search
relevance score
heavy
not cacheable
exact match
show or hide
lightning fast
cacheable

69. Combine Filters & Queries

70. > GET /_search -d
{
"query": {
"filtered": {
"query": {
"match": { "title": "awesome" }
},
"filter": {
"term": { "type": "article" }
}
}
}
}

71. and Sorting

72. > GET /_search -d
{
"query": {
"filtered": {
"query": {
"match": { "title": "awesome" }
},
"filter": {
"term": { "type": "article" }
}
}
}
"sort": {"date":"desc"}
}

73. Relevance. Explain API

76. Term frequency
How often does the term appear in the field?
The more often, the more relevant.
Inverse document frequency
How often does each term appear in the
index? The more often, the less relevant. T
Field norm
How long is the field? The longer it is, the less
likely it is that words in the field will be
relevant.

77. and Facets

78. Facets on Amazon

79. > GET /_search -d
{
"facets": {
"home_team": {
"terms": {
"field": "field_home_team"
}
}
}
}

80. > GET /_search -d
{
"facets": {
"home_team": {
"terms": {
"field": "field_home_team"
}
}
}
}
Give your facet a name

81. > GET /_search -d
{
"facets": {
"home_team": {
"terms": {
"field": "field_home_team"
}
}
}
}
Your facet filter can be:
● Terms
● Range
● Histogram
● Date Histogram
● Filter
● Query
● Statistical
● Terms Stats
● Geo Distance

82. "facets" : {
"home_team" : {
"_type" : "terms",
"missing" : 203,
"total" : 100,
"other" : 42,
"terms" : [ {
"term" : "hou",
"count" : 8
}, {
"term" : "sln",
"count" : 6
}, ...

83. STOP! I want this in Drupal?

84. Available modules:
Elasticsearch
Elasticsearch Connector
Search API elasticsearch

85. Development directions:
1. Search API implementation
2. Field Storage API
3. Alternative backends
Available modules:
Elasticsearch
Elasticsearch Connector
Search API elasticsearch

86. Field Storage API implementation
Elasticsearch field storage sandbox by Damien Tournoud
Started in July 2011

87. Field Storage API implementation
Elasticsearch field storage sandbox by Damien Tournoud
Started in July 2011
Elasticsearch EntityFieldQuery sandbox
https://drupal.org/sandbox/asgorobets/2073151

88. Let's DEMO

89. Let the Search be with you