Elasticsearch is a highly scalable system if you use it to its full potential. In this talk we discuss the most important parts to make it scale with your Java applications: Monitoring your setup for performance. Optimize with bulk requests. Distribute the load in your cluster. Pick the right index and shard strategy.

1. Scale
for Your Java Applications
Philipp Krenn @xeraa
@xeraa

2. Developer
@xeraa

3. @xeraa

4. Terminology
Cluster, Node, Index, Shard
@xeraa

6. Disclaimer
This is not a benchmark
@xeraa

7. Injector
@xeraa

8. Download
http://download.elastic.co/workshops/basic-kibana/injector/
injector-7.0.jar
@xeraa

9. Use
$ java -jar injector-7.0.jar
--nb 1000 --bulk 100
--debug
--elasticsearch --es.host http://localhost:9200
--es.user elastic --es.pass changeme
--es.index person
--console --cs.pretty
@xeraa

11. {
"name": "Mamadou Colombe",
"dateofbirth": "2009-12-26",
"gender": "male",
"children": 4,
"marketing": {
"cars": null,
"shoes": null,
"toys": 1688,
"fashion": null,
"music": 243,
"garden": null,
"electronic": 1222,
"hifi": null,
"food": null
},
"address": {
"country": "France",
"zipcode": "75000",
"city": "Paris",
"countrycode": "FR",
"location": {
"lon": 2.3250817954678284,
"lat": 48.857148516731385
}
}
}
@xeraa

12. Monitoring
@xeraa

13. time
$ time java -jar injector-7.0.jar ...
@xeraa

14. Stack Monitoring
@xeraa

16. Bulk Requests
@xeraa

17. 1,000 Docs & Bulk Size 1
Executed in 37.87 secs fish external
usr time 13.08 secs 135.00 micros 13.08 secs
sys time 1.00 secs 752.00 micros 1.00 secs
@xeraa

18. 1,000 Docs & Bulk Size 100
Executed in 4.01 secs fish external
usr time 7.49 secs 134.00 micros 7.49 secs
sys time 0.43 secs 738.00 micros 0.43 secs
@xeraa

19. 1,000 Docs & Bulk Size 100
Executed in 3.91 secs fish external
usr time 6.34 secs 138.00 micros 6.34 secs
sys time 0.38 secs 843.00 micros 0.38 secs
@xeraa

20. Optimal Bulk Size?
@xeraa

21. It Depends
Document size (100B, 1KB, 100KB, 1MB,...)
Number of nodes & shards
Node size
Load on the cluster
@xeraa

22. Refresh
@xeraa

23. index.refresh_interval
Default 1s
30s or -1
@xeraa

24. 10,000 Docs & Refresh 1s
Executed in 10.23 secs fish external
usr time 11.50 secs 124.00 micros 11.50 secs
sys time 0.77 secs 845.00 micros 0.77 secs
@xeraa

25. 10,000 Docs & Refresh 30s
Executed in 10.02 secs fish external
usr time 11.87 secs 120.00 micros 11.87 secs
sys time 0.74 secs 730.00 micros 0.74 secs
@xeraa

26. Load
Distribution
@xeraa

27. Nodes
Master
Data
Coordinating
@xeraa

28. Topology
Sniffing
Cloud Proxy
@xeraa

29. Distribution
Round Robin
Adaptive Replica Selection for searches
@xeraa

30. Index & Shard
Strategy
@xeraa

31. Defaults
7.0: 1 primary, 1 replica
Before: 5 primary, 1 replica
@xeraa

32. Index Patterns
Static vs Time-Based
@xeraa

33. Time-Based
Don't _delete_by_query
Do daily / weekly / ... patterns
Or even better Rollover
@xeraa

34. Shard Size
>10GB, maybe ~50GB
It depends
@xeraa

35. Shards per GB Heap
<20
@xeraa

36. Pagination
@xeraa

37. Pagination
GET person/_search
{
"from" : 0,
"size" : 10,
"sort": [
{
"dateofbirth": {
"order": "asc"
}
}
]
}
@xeraa

38. Deep Pagination
GET person/_search
{
"from" : 5000,
"size" : 10,
"sort": [
{
"dateofbirth": {
"order": "asc"
}
}
]
}
@xeraa

39. Error
GET person/_search
{
"from" : 10000,
"size" : 10,
"sort": [
{
"dateofbirth": {
"order": "asc"
}
}
]
}
@xeraa

40. For Your Protection
index.max_result_window: 10000
Search After
Scroll
@xeraa

41. Profiler
@xeraa

43. Benchmarks
@xeraa

44. Rally
Macrobenchmarking Framework for
Elasticsearch
https://github.com/elastic/rally
@xeraa

45. Nightly Benchmarks
https://benchmarks.elastic.co
@xeraa

49. Conclusion
@xeraa

50. Topics
Monitoring
Bulk & Refresh
Load Distribution
@xeraa

51. Topics
Index & Shard Strategy
Pagination
Benchmarks
@xeraa

52. It Depends
@xeraa

53. Free & Open
@xeraa

54. Scale
for Your Java Applications
Philipp Krenn @xeraa
@xeraa