In this presentation, we are going to discuss how elasticsearch handles the various operations like insert, update, delete. We would also cover what is an inverted index and how segment merging works.

1. Deep Dive Into Elasticsearch
Kunal Kapoor
Software Consultant
Knoldus Software LLP

2. AGENDA
● What is Elasticsearch
● Getting Started
● Key Terminologies
● CRUD Operations
● Understanding the physical layout
● What happens when you index a document
● How to make an inverted index mutable
● How per-segment search works
● How a delete operation works
● Segment Merging

3. What is Elasticsearch
● Search engine based on Lucene.
● Provides near real-time search
● Distributed
● Fault Tolerant
● Notable users:
– Facebook
– Github
– CERN
– LinkedIn

4. Getting Started
● Download the elasticsearch distribution from
https://www.elastic.co/downloads/elasticsearch
● To start the elasticsearch server run the following
command from within the extracted directory
– ./bin/elasticsearch
● Once the server or node is created you can check the
health of your cluster by running
– curl 'localhost:9200/_cat/health?v'

5. Key Terminologies
● Node - A node is a single server that is part of your
cluster, stores your data, and participates in the cluster’s
indexing and search capabilities
● Cluster - A cluster is a collection of one or more nodes
(servers) that together holds your entire data and
provides indexing and search capabilities across all
nodes.
● Index - An index is a collection of documents that have
somewhat similar characteristics.

6. Key Terminologies
● Type - A type is a logical category/partition of your index.
It is defined for documents that have a set of common
fields.
● Shard – A shard is basically an lucene index. Contains
the documents and various data structures that help in
searching.

7. CRUD Operations
● Indexing a document
– curl -XPOST 'localhost:9200/test/test/1?pretty' -d '{"text":"Hello World"}'
● Updating a document
– curl -XPOST 'localhost:9200/test/test/1?pretty' -d '{"text":"Hello"}'
● Delete a document
– curl -XDELETE 'localhost:9200/test/test/1?pretty'
● Search a document
– curl -XPOST 'localhost:9200/test/_search?pretty' -d '{"query":
{ "match": {"text": "hello" }}}'

8. Understanding the physical layout

9. SHARDS
(Lucene indices)

10. SEGMENTS

12. Inverted Index
● Data structure storing a mapping, from content such as
words or numbers, to its locations in a database file, or a
set of documents.
● Provides full-text search
● Consists of 2 parts
– Sorted Dictionary
– Postings
● Immutable

13. What happens when you index a
document?
● The node that receives the request becomes the
controller for that request.
● That node determines the shard in which the document
should reside on the basis of the documents Id.
– shard = hash(document_id) % number_of_primary_shards
● The request is then forwarded to the appropriate node
which contains the shard.
● The node forwards the request to the appropriate shard.

14. What happens when you index a
document?
● The shard performs analysis on the document and creates the
appropriate inverted index which is helpful for searching.
● The request is then sent to the replica shards.
● The documents are analyzed by the standard analyzer by
default.
● It split the documents on white-space and lowercases the
documents.
● The documents are then ready to be inserted in the inverted
index

15. What happens when you index a
document?
{
“text”:”Elasticsearch is an awesome search engine”
}
{
“text:”Elasticsearch is not a database”
}

16. elasticsearch, is, an, awesome, search, engine, elasticsearch,not, a, database

17. Terms Document1 Document2
elasticsearch ✓ ✓
is ✓ ✓
an ✓ -
awesome ✓ -
search ✓ -
engine ✓ -
not - ✓
a - ✓
database - ✓

18. How to make an inverted index
mutable?
● Earlier, the whole inverted index would be rewritten to
disk with the changes.
● Very costly approach
● Lucene introduced the concept of per-segment search.
● Now a Lucene index would mean a collection of
segments plus a commit point.
● A commit point is a file that contains the list of segments
that are ready for search.

19. How per-segment search works?
● New documents are collected in an in-memory buffer.
● Every so often, the buffer is commited (refresh)
– A new supplementary segment with a commit point is
written to file-system cache.
– The transaction log is updated with the request for a
full commit later.
● The buffer is cleared and the segment is made available
for search.

20. In-memory Buffer
COMMIT POINT
Transaction Log

21. How a delete operation works?
● Every shard in the Elasticsearch node maintains a .del file
along with the commit point.
● .del file lists which documents in which segments have
been deleted.
● When a delete request is encountered, the appropriate
document is marked as deleted.
● The deleted document will still match the search query but
will be filtered later on.
● Later the document is purged from the file-system while
segment merging.

22. Segment Merging
● Each segment consumes file handles, memory, and CPU
cycles.
● The more the number of segments, the slower the search
will be.
● Elasticsearch solves this problem by merging segments
in the background.
● Small segments are merged into bigger segments.
● This is the moment when those old deleted documents
are purged from the file-system.

23. Segment Merging
● This is how the merge process works:-
– The merge process selects a few segments of similar
size and merges them into a new bigger segment in
the background.
– The new segment is flushed to disk.
– A new commit point is written that includes the new
segment and excludes the old, smaller segments.
– The new segment is opened for search.
– The old segments are deleted.

24. References
● The Definitive Guide by Clinton Gormley and Zachary
Tong
● https://www.elastic.co/guide/en/elasticsearch/reference/current/

25. Questions?

26. References
● The Definitive Guide by Clinton Gormley and Zachary
Tong
● https://www.elastic.co/guide/en/elasticsearch/reference/current/

27. THANK YOU