Finding Cars and Hunting Down Logs - ElasticSearch @AutoScout24

1. Finding Cars and Hunting Down Logs:
Elasticsearch @ AutoScout24
AutoScout24
24 Nov 2016
Philipp Garbe Lead developer (philipp.garbe@scout24.com)
Juri Smarschevski Team lead (juri.smarschevski@scout24.com)

2. Search
AutoScout24 search journey in nutshell
2

3. Who we are ?
Unique Monthly Visitors in Europa
3
…10 more

4. Some numbers
Search index contains ~2.6M classifieds
4
Unique visitors (monthly): ~10M
Search requests per day: ~36M
Index update rate per day ~400.000 classifieds

5. Status quo. March 2013.
Endeca used as a search engine
5
Use case: providing search results and facets for the entire AS24
platform
Problems:
• New product requirements, performance of Endeca becomes slower
• Time to market of our required features is not sufficient
• Maintenance is complex / expensive

6. Possible candidates
Solr ?
• <feeling> too complex installation / configuration </feeling>
6
Sphinx ?
• Support situation is unclear
Elasticsearch ?
• Fresh buzzword
• From beginning on built for distributed systems (rumors)
• Easy installation / configuration (fact)

7. POC
Goals
• Performance should be comparable with Endeca
• The solution should be scalable
7

8. 8
Rollout plan. 03.2013 - 11.2013
07.2013 11.201302.2013 03.2013 05.2013
POC
Implementation & migration
Training
Go live phase
#real_project_picture_squeezed

9. 9
Endeca Elasticsearch
(0.9.x)
Amount of machines 60 20
[Re]index time ~180 min ~45 min
Deploy to Live up to 2 days < 3 hours
Effort for testing an
issue on local machine
4 h 1 h
Performance = =
Product / dev guys
satisfaction
:( :)
300%
400%
1000%
400%
% ?
Results after 8 months of working.

10. No problems after migration ?
Cluster split brain
Has in fact nothing to do with Elasticsearch, is more related to learn phase at AS24
10
Deep pagination
Elasticsearch 5.x release notes: “Deep pagination of search results is now possible with the search_after
feature, which efficiently skips over previously returned results to return just the next page.“

11. 11
Status quo. November 2014.
Project “Tatsu” has started
.NET => JVM
C# => Scala
IIS / Windows => Play / Linux
Local data center => AWS
Monolith => Micro services
Windows workstations => Mac notebooks
... => ...

12. 12
Status quo. November 2014.
Project “Tatsu” has started
.NET => JVM
C# => Scala
IIS / Windows => Play / Linux
Local data center => AWS
Monolith => Micro services
Windows workstations => Mac notebooks
? => ?
=> 2015

13. 13
Elasticsearch clusters “lift & shift” to AWS ?
AWS Elasticsearch Service ?
Elasticsearch as a service (SaaS) ?
Own hosting in AWS ?

14. 16
Rolling update in details (possible scenario).
Time
1
Initial state

15. 17
Rolling update in details (possible scenario).
Node has
been replaced
Time
1 2
Initial state
~ 60 sec

16. 18
Rolling update in details (possible scenario).
Master has
been killed
Node has
been replaced
Time
1 2 3
Initial state

17. 19
Rolling update in details (possible scenario).
Master has
been killed
Node has
been replaced
Master election
Time
1 2 3 4
Initial state

18. 20
Rolling update in details (possible scenario).
Master has
been killed
Node has
been replaced
Master election
Time
1 2 3 4 5
Initial state Last node has
been replaced

19. 21
Rolling update findings
Master has
been killed
?Outage=

20. 22
Rolling update findings

21. Logging
Continuously deployed, immutable and stateful
23

22. 7.4 billion
documents
Some numbers
36 TB
EBS
18 nodes á
m4.4xlarge
(64GB / 53.5
cpu units)

23. Unified Logs
25

24. Challenge: Deployment time

25. Rolling updates
27

31. Challenge: Costs

32. First setup
● 18x m4.4xlarge
● 18x 2TB gp2
● 3TB/day cross-zone traffic

33. Cost/Usage Optimized Setup
● 15x m4.x2large
● 15x 384GB gp2
● 6x SpotFleet
● 6x 4TB st1
● 9TB/day cross-zone traffic
Savings: ~40%

34. Future. What next ?
Percolator (saved search)
36
Elastic Graph (recommendations)
Freetext search

35. 37
Conclusion
Here is a simple question - if we had the
possibility to go back in the time and start the
same journey with Elasticsearch,
would we do it the same way ?

36. Q & A
38