Video available at http://www.youtube.com/watch?v=gkdfNl0WL-A Original slides at http://presentations.found.no/berlin-buzzwords-2013/ This talk covers some of the lessons we've learned from securing and herding hundreds of Elasticsearch clusters. It is applicable whether you operate Elasticsearch in your own infrastructure, in the cloud, or if you're a developer who wants a better understanding of Elasticsearch's various failure modes. Elasticsearch easily lets you develop amazing things, and it has gone to great lengths to make Lucene's features readily available in a distributed setting. However, when it comes to running Elasticsearch in production, you still have a fairly complicated system on your hands: a system with high expectations on network stability, a huge appetite for memory, and a system that assumes all users are trustworthy. Instead of delving deeply into a few specifics, we give a brief overview of problems you are likely to run into and suggested solutions to these problems. We cover topics that are applicable to both developers and users with Elasticsearch clusters of every shape and size – with an emphasis on resiliency and security. Basic familiarity with Elasticsearch is assumed.

1. Elasticsearch in production
Alex Brasetvik
@alexbrasetvik

2. How marketing thinks our users feel

3. How we developers sometimes feel

4. Who?
Co-founder of Found AS
7+ years of search, 2+ Elasticsearch
We manage hundreds of Elasticsearch
clusters
… on Amazon's cloud

5. Agenda
Memory (and stability)
Security (and multi-tenancy)
Networking (and reliability)
Client (and resiliency)

6. Memory
Search engines crave memory
Caches, caches, caches
Field- and filter caches
Page cache
Index building

7. PostgreSQL
Verifies resource usage
Safe >>> fast
Uses disk if necessary

8. Elasticsearch trusts you
Built for speed
It'll jump if you ask it to
What could possibly go wrong?

9. OutOfMemoryError
Woah there
I ate all the memories
Your cluster may or may not work any more

10. May or may not work?
What else was happening at the time?
Corrupt cluster state, crashed Netty, …
In short: Don't end up there

11. Warning signs?
Monitor cache sizes and heap space
Outgrowing page cache: gradual slowdown
Outgrowing heap space: sudden crash

12. Understand the memory profile
Test realisticly
Bound cache sizes and flush thresholds
v0.90+ takes you longer with field filters, etc.

13. Large heaps are expensive to garbage collect
Keep heap < 32GiB (But test!)
Lots of page cache is good, though!

14. Security
Elasticsearch trusts everyone
Not its job to do auth(z)
You're the gatekeeper

15. _search
Read only?
Limit indexes / wrap with filters?
Protect the field caches

16. Arbitrary code execution
Elasticsearch has powerful scripting
Not sandboxed
On by default

17. Any website can reach your machine
http://127.0.0.1:9200/_search?callback=capture&source=…
Run in a virtual machine

18. Networking
Elasticsearch is distributed
Easy (for a distributed system)
Supports many usage patterns.

19. Quite common topology
High availability, right?

20. Obey or risk split brains …
… and irrecoverable data-loss

21. +1 is a "tie breaker"

22. Stormy clouds
Zone vs instance failure
Thundering herds
Optimizing MTTR is not HA

23. Client considerations
Idempotent/retry-able requests
Use a connection pool.
_bulk / _msearch

24. Have enough memory
Have a majority of nodes
Don't allow arbitrary search requests
Use retryable requests

26. Alex over Trondheim, Tore Helgedagsrud
Elephant, Roy Costello
Wingsuit, Richard Schneider
Lightning Storm and Stars, Justin Ennis
Wingsuit flock, Richard Schneider
Oh salad, you so funny, Eatliver