Brahe mass scale flexible indexing
•
0 gefällt mir•1,174 views
Presented by Ben Brown, Software Architect, Cerner Corporation Our team made their first foray into Solr building out Chart Search, an offering on top of Cerner's primary EMR to help make search over a patient's chart smarter and easier. After bringing on over 100 client hospitals and indexing many tens of billions of clinical documents and discrete results we've (thankfully) learned a couple of things. The traditional hashed document ID over many shards and no easily accessible source of truth doesn't make for a flexible index. Learn the finer points of the strategy where we shifted our source of truth to HBase. How we deploy new indexes with the click of a button, take an existing index and expand the number of shards on the fly, and several other fancy features we enabled.
Empfohlen
Empfohlen
Weitere ähnliche Inhalte
Ähnlich wie Brahe mass scale flexible indexing
Ähnlich wie Brahe mass scale flexible indexing (20)
Mehr von lucenerevolution
Mehr von lucenerevolution (20)
Kürzlich hochgeladen
Kürzlich hochgeladen (20)
Brahe mass scale flexible indexing
- 1. Brahe - Flexible Indexing At Scale Ben Brown Software Architect, Cerner Corporation
- 2. Who I Am • Ben Brown Software Architect • Cerner Healthcare IT Company • Semantic Solutions Team of 10 Search Services Fun Stuff NLP, Medical Ontologies, ML
- 3. Chart Search Taking This Photo: http://bit.ly/Y7kTJt
- 4. Chart Search Turning it into this
- 5. Chart Search Does • Faceting • NLP • Semantic Concept Markup Makes for a heavy record (Especially on Solr 1.4)
- 6. Where We Started Started Major Engineering in 2009 IBM Dev Works: http://ibm.co/14ZrtqX
- 7. Where We Started Started Major Engineering in 2009 IBM Dev Works: http://ibm.co/14ZrtqX
- 8. Scale • Clusters partitioned by client • Raw and processed data in HDFS • All processing & indexing done through map reduce
- 9. Shard Size Limiting Factor ~26 Million Discrete Results Per Shard Average of 35 Shards Per Client Range 5 to 140
- 11. Query Touch Points One User Action ~ 4 Queries 35 Shards - 432 Touch Points 140 Shards - 1692 Touch Points • Works, but not efficient • Chance for variance killing performance • Failure is a massive config headache
- 12. Growth • Hashed ID does not play well with resizing • Deploy Again • Reindex Everything Document Hash modulo Shard Count Doc One:Hash(abc123) = 15 Doc Two: Hash(efg456) = 8 Doc Three: Hash(hij789) = 7 3 Shards Doc One -> Shard 0 Doc Two -> Shard 2 Doc Three -> Shard 1 4 Shards Doc One -> Shard 3 Doc Two -> Shard 0 Doc Three -> Shard 3
- 13. We Have a Problem Painful Growth Lots of Deploys Variance Risk Image: http://bit.ly/Y7oBD6
- 14. What Would Be Better? Load Balance at the Client Automated Failover Easy Deployments Simplified Splitting Minimized Touch Points Disconnected Stages
- 15. Solution Shift Master to HBase Image: http://bit.ly/ZXO2na
- 16. Why HBase? Lexically organized keys Efficient key range scans Efficient time based scans We're pretty good at operating it
- 17. Coordinate With ZooKeeper |-- Index name |-- Version |-- Solr Schema/Config |-- Table Name + Connection Info |-- Shard Number |-- Shard Boundary Info |-- Replica Number |-- Ephemeral Claim |-- Solr Connection Info |-- Ephemeral Online
- 18. Custom Core Admin Work with ZooKeeper for claim process Creates solr core after claims Controls pulling data from HBase
- 19. Claim Process
- 20. Claim Process
- 22. Claim Process
- 23. Claim Process
- 24. Claim Process
- 25. Claim Process
- 26. Claim Process
- 27. Claim Process
- 28. Claim Process
- 29. Claim Process
- 30. Claim Process
- 31. Claim Process
- 32. Claim Process
- 33. Claim Process
- 34. Claim Process
- 35. Claim Process
- 36. Claim Process
- 37. Claim Process
- 38. Claim Process
- 39. Claim Process
- 40. Claim Process
- 41. Claim Process
- 42. Claim Process
- 43. Claim Process
- 44. Claim Process
- 45. Claim Process
- 46. Claim Process
- 47. Claim Process
- 48. Claim Process
- 49. Claim Process
- 50. Coordinate With ZooKeeper |-- Index name |-- Version |-- Solr Schema/Config |-- Table Name + Connection Info |-- Shard Number |-- Shard Boundary Info |-- Replica Number |-- Ephemeral Claim |-- Solr Connection Info |-- Ephemeral Online
- 51. Queries • Client inspects ZooKeeper • Finds online nodes o Only for the keyspace it cares about o Issues distributed queries if necessary • Balances in the Client • Retries if queries fail
- 52. Ends Thoughts • Keep things simple • Disconnect your stages • Keep your touchpoints at a minimum • Organize your data around your queries • Use what you’re good at
- 53. CONTACT Ben Brown http://linkd.in/ZZIBK4 @b_brown ENGINEERING BLOG https://engineering.cerner.com/ WE’RE HIRING! http://www.cerner.com/About_Cerner/Careers/
- 54. Bonus Slides!