1. ELASTICSEARCH
AT
THE NORWEGIAN TAX SERVICES
Oslo Enterprise Search Meetup
Magnus Haug - magnus.haug@bekk.no
Jørgen Braseth - jorgen.braseth@bekk.no
Teknologihuset
18.02.2014

2. AGENDA
MAG & Search
Examples of use
How we did it
Tools
Conclusion

3. Background
MAG & SEARCH

4. MAG
Replacing 19 existing systems
Data for tax returns
~ 200M documents each year
At least 10 years data retention

5. «We want a Google search»

6. OUR ALTERNATIVES

7. Designed for horizontal scaling
Lucene
HTTP or Binary

8. Example of use
FINDING REPORTED INFORMATION

10. {CICS}

20. Example of use
IDENTIFICATION

22. Architecture
HOW WE DO IT

23. HOW WE DO: INDEXING
Document store
ATOM
ATOM
ATOM
Indexing process
Elasticsearch

24. HOW WE DO: COMBINING DATA SOURCES
Person info
Company info
Document store
ATOM
ATOM
ATOM
Indexing process
Elasticsearch

25. HOW WE DO: SECURING DATA
Search API
Read unfiltered
Indexing process
Read filtered
Document Index
Write

26. HOW WE «DO»: ROBUSTNESS
Rack 1
Rack 2
Rack 3
ES Node 1
ES Node 3
ES Node 5
ES Node 2
ES Node 4

27. HOW WE DO: REINDEXING
Person info
Company info
ATOM
Document store
ATOM
ATOM
ATOM
Indexing process
Elasticsearch

28. HOW WE DO: HOT SWAPPING INDICES
Search API
Indexing process
Read
Write
Old index
New index

29. Everyday maintenance
TOOLS

30. Head

31. Big Desk

32. Elasticsearch HQ

33. Kibana

34. Marvel

35. Marvel

36. What do we think?
IMPRESSIONS

37. PERFORMANCE
100 million docs
Two ES-nodes
< 1000 ms
http://www.sxc.hu/photo/1430721

38. DEVELOPER FRIENDLY
Easy setup
Easy testing
Easy debugging
cURL!

39. INTEGRATION TESTING
@RunWith( ElasticsearchRunner.class )
public class MinSystemTest {
@ElasticsearchNode( name = "iris-test", clusterName = "iris-test" )
private static Node node;
@ElasticsearchClient( nodeName = "iris-test" )
private static Client client;
...
SearchRequestBuilder searchRequestBuilder =
client.prepareSearch( INDEKSNAVN ).setQuery( QueryBuilders.matchAllQuery() );
return client.search( searchRequestBuilder.request() ).actionGet();
...

40. UNUSED FEATURES
Percolate
Highlighting
Suggestions
Rivers
REST-API

41. FUTURE PLANS FOR ELASTICSEARCH AT MAG
Upgrade to v1.0
Used as a cache
As a service in our PaaS

42. ARE WE HAPPY?

43. ?