Elastic{ON} 2017 Recap
•
7 gefällt mir•1,291 views
Elastic{ON} is the big conference organized by Elastic where new features and roadmap are announced. In this session, we will explore what's new in the Elastic stack
Empfohlen
Empfohlen
Weitere ähnliche Inhalte
Was ist angesagt?
Was ist angesagt? (20)
Ähnlich wie Elastic{ON} 2017 Recap
Ähnlich wie Elastic{ON} 2017 Recap (20)
Mehr von Matias Cascallares
Kürzlich hochgeladen
Kürzlich hochgeladen (20)
Elastic{ON} 2017 Recap
- 1. Elastic @mcascallares Elastic{ON} 2017 Recap Matias Cascallares
- 6. 6 • Faster • Friendlier • Smaller • Smarter • Safer 26 October 2016 Elasticsearch 5.0.0
- 7. Throughput with one replica on two nodes, with auto-generated IDs Append-only indexing 7 0 8 15 23 30 v2.4.2 v5.2.1 master K docs/s
- 8. What’s new in 5.x?
- 9. Mappings
- 10. Range Fields & Queries
- 11. 11 What’s on at Elasticon tomorrow between 11am and 2pm?
- 12. 12
- 13. 13 Wednesday 11am - 2pm
- 14. 14 Wednesday 11am - 2pm - INTERSECTS
- 15. 15 Wednesday 11am - 2pm - CONTAINS
- 16. 16 Wednesday 11am - 2pm - WITHIN
- 18. 18 { "city": { "type": "string", "index": "analyzed", "fields": { "city.keyword": { "type": "string", "index": "not_analyzed" } } } }
- 19. 19 { "city": { "type": "text" "fields": { "city.keyword": { "type": "keyword" } } } }
- 20. 20 { "city": { "type": "text" "fields": { "city.keyword": { "type": "keyword" } } } } Full text queries Full text analysis
- 21. 21 { "city": { "type": "text" "fields": { "city.keyword": { "type": "keyword" } } } } Keyword queries Aggregations Sorting
- 22. 22 { "city": { "type": "text" "fields": { "city.keyword": { "type": "keyword" } } } } No analysis
- 23. 23 San Francisco SAN FRANCISCO san francisco San franciscO
- 24. 24 San Francisco SAN FRANCISCO san francisco San franciscO san francisco Normalizer
- 27. 27 NY NYC New York New York City } Synonyms
- 28. Phrase query: “NYC is OLD!”
- 29. 29 Synonym Filter: (ny|nyc|new), (is|york), (old,city)
- 30. 30 Synonym Filter: (ny|nyc|new), (is|york), (old,city)
- 31. 31 Synonym Filter: (ny|nyc|new), (is|york), (old,city) ny is old nyc new york city Synonym Graph Filter:
- 33. 33 Query Optimizations • Smarter query caching • Faster geo, range, and nested queries • Unified highlighter • Field collapsing • Cancellable searches • Partitioned term aggs
- 35. 35 When your cluster is RED… /_cat/allocation /_cat/indices /_cat/nodes /_cat/recovery /_cat/shards /_cluster/health /_cluster/state /{index}/_shard_stores /_cluster/settings /_node/stats /{index}/_settings /_node
- 36. 36 When your cluster is RED… /_cluster/allocation/explain
- 37. 37 /_cluster/allocation/explain … "allocate_explanation" : "cannot allocate because allocation is not permitted to any of the nodes”, … { "decider" : "filter", "decision" : "NO", "explanation" : "node does not match index setting [index.routing.allocation.include] filters [_name:"non_existent_node"]" } …
- 38. 38 /_cluster/allocation/explain … "unassigned_info" : { "reason" : "NODE_LEFT", "at" : "2017-01-04T18:03:28.464Z", "details" : "node_left[OIWe8UhhThCK0V5XfmdrmQ]", "last_allocation_status" : "no_valid_shard_copy" }, "can_allocate" : "no_valid_shard_copy", "allocate_explanation" : "cannot allocate because a previous copy of the primary shard existed but can no longer be found on the nodes in the cluster" …
- 39. 39 /_cluster/allocation/explain … "rebalance_explanation" : "cannot rebalance as no target node exists that can both allocate this shard and improve the cluster balance", "node_allocation_decisions" : [ { "node_id" : "oE3EGFc8QN-Tdi5FFEprIA", "node_name" : "node_t1", "transport_address" : "127.0.0.1:9401", "node_decision" : "worse_balance", "weight_ranking" : 1 } …
- 40. Java REST Client
- 41. 41 Java REST Client - behind the scenes • Came late to the party… • Isn’t nearly as extensive as the Transport Client • Maintaining a transport protocol based client causes a massive engineering overhead • It’s a “second” entry point into the system • Complicates distinguishing between clients and nodes
- 42. 42 Java low-level HTTP client • Released in 5.0.0 • JSON strings only • Resilient, but not user friendly due to the lack of a higher level API
- 43. 43 Java high-level HTTP client • IDE friendly • Similar API to Transport Client - easy migration • Based on low-level REST client • Support CRUD & Search • Previews in 5.5 • Depends on elasticsearch-core
- 44. Tribe Node
- 45. 45 How the Tribe Node works Cluster Sales Master Nodes Data Node Data Node Data Node Cluster R&D Master Nodes Data Node Data Node Data Node
- 46. 46 How the Tribe Node works Cluster Sales Master Nodes Data Node Data Node Data Node Tribe Node Cluster R&D Master Nodes Data Node Data Node Data Node tribe: t1: cluster.name: sales t2: cluster.name: r_and_d
- 47. 47 How the Tribe Node works Cluster Sales Master Nodes Data Node Data Node Data Node Tribe Node t1 Node Client Cluster R&D Master Nodes Data Node Data Node Data Node
- 48. 48 How the Tribe Node works Cluster Sales Master Nodes Data Node Data Node Data Node Tribe Node t1 Node Client Cluster R&D Master Nodes Data Node Data Node Data Node t2 Node Client
- 49. 49 How the Tribe Node works Cluster Sales Master Nodes Data Node Data Node Data Node Tribe Node t1 Node Client Cluster R&D Master Nodes Data Node Data Node Data Node t2 Node ClientCluster State Cluster State
- 50. 50 How the Tribe Node works Cluster Sales Master Nodes Data Node Data Node Data Node Tribe Node t1 Node Client Cluster R&D Master Nodes Data Node Data Node Data Node t2 Node Client Cluster State Cluster State
- 51. 51 How the Tribe Node works Cluster Sales Master Nodes Data Node Data Node Data Node Tribe Node t1 Node Client Cluster R&D Master Nodes Data Node Data Node Data Node t2 Node Client Merged Cluster State
- 52. 52 How the Tribe Node works Cluster Sales Master Nodes Data Node Data Node Data Node Tribe Node t1 Node Client Cluster R&D Master Nodes Data Node Data Node Data Node t2 Node Client Merged Cluster State Kibana
- 53. 53 Problems With How the Tribe Node works Cluster Sales Master Nodes Data Node Data Node Data Node Tribe Node t1 Node Client Cluster R&D Master Nodes Data Node Data Node Data Node t2 Node Client Merged Cluster State Kibana
- 54. 54 Problems With How the Tribe Node works Cluster Sales Master Nodes Data Node Data Node Data Node Tribe Node t1 Node Client Cluster R&D Master Nodes Data Node Data Node Data Node t2 Node Client Merged Cluster State Kibana Static Configuration tribe: t1: cluster.name: sales t2: cluster.name: r_and_d
- 55. 55 Problems With How the Tribe Node works Cluster Sales Master Nodes Data Node Data Node Data Node Tribe Node t1 Node Client Cluster R&D Master Nodes Data Node Data Node Data Node t2 Node Client Kibana Merged Cluster State Connections to All Nodes
- 56. 56 Problems With How the Tribe Node works Cluster Sales Master Nodes Data Node Data Node Data Node Tribe Node t1 Node Client Cluster R&D Master Nodes Data Node Data Node Data Node t2 Node Client Kibana Merged Cluster State Frequent cluster state updates
- 57. 57 Problems With How the Tribe Node works Cluster Sales Master Nodes Data Node Data Node Data Node Tribe Node t1 Node Client Cluster R&D Master Nodes Data Node Data Node Data Node t2 Node Client Kibana Merged Cluster State Index names must be unique
- 58. 58 Problems With How the Tribe Node works Cluster Sales Master Nodes Data Node Data Node Data Node t1 Node Client Cluster R&D Master Nodes Data Node Data Node Data Node t2 Node Client Merged Cluster State Tribe Node Kibana No master node No index creation
- 59. 59 Problems With How the Tribe Node works Cluster Sales Master Nodes Data Node Data Node Data Node Tribe Node t1 Node Client Cluster R&D Master Nodes Data Node Data Node Data Node t2 Node Client Merged Cluster State Kibana Reduce results from many shards
- 60. The Tribe Node is Dead
- 62. Minimal viable solution to supersede tribe 62
- 63. Reduces the problem domain to query execution 63
- 64. Cluster related information is reduced to a namespace 64
- 65. 65 How Cross-Cluster search works Cluster Sales Master Nodes Data Node Data Node Data Node Cluster R&D Master Nodes Data Node Data Node Data Node
- 66. 66 How Cross-Cluster search works Cluster Sales Master Nodes Data Node Data Node Data Node Cluster R&D Master Nodes Data Node Data Node Data Node Any node can perform cross-cluster search
- 67. 67 How Cross-Cluster search works Cluster Sales Master Nodes Data Node Data Node Data Node Optional dedicated cross-cluster search cluster Master/Data Node Cluster R&D Master Nodes Data Node Data Node Data Node Master/Data Node
- 68. 68 How Cross-Cluster search works Cluster Sales Master Nodes Data Node Data Node Data Node Master/Data Node Cluster R&D Master Nodes Data Node Data Node Data Node Master/Data Node PUT _cluster/settings { "transient": { "search.remote": { "sales.seeds": "10.0.0.1:9300", “r_and_d.seeds”: "10.1.0.1:9300" } } } Dynamic settings Optional dedicated cross-cluster search cluster
- 69. 69 How Cross-Cluster search works Cluster Sales Master Nodes Data Node Data Node Data Node Master/Data Node Cluster R&D Master Nodes Data Node Data Node Data Node Master/Data Node No cluster state updates Optional dedicated cross-cluster search cluster
- 70. 70 How Cross-Cluster search works Cluster Sales Master Nodes Data Node Data Node Data Node Master/Data Node Cluster R&D Master Nodes Data Node Data Node Data Node Master/Data Node Kibana Optional dedicated cross-cluster search cluster
- 71. 71 How Cross-Cluster search works Cluster Sales Master Nodes Data Node Data Node Data Node Master/Data Node Cluster R&D Master Nodes Data Node Data Node Data Node Master/Data Node Kibana Can create indices Optional dedicated cross-cluster search cluster
- 72. 72 How Cross-Cluster search works Cluster Sales Master Nodes Data Node Data Node Data Node Master/Data Node Cluster R&D Master Nodes Data Node Data Node Data Node Master/Data Node Kibana Optional dedicated cross-cluster search cluster
- 73. 73 How Cross-Cluster search works Cluster Sales Master Nodes Data Node Data Node Data Node Master/Data Node Cluster R&D Master Nodes Data Node Data Node Data Node Master/Data Node Kibana Few lightweight connections Optional dedicated cross-cluster search cluster
- 74. 74 How Cross-Cluster search works Cluster Sales Master Nodes Data Node Data Node Data Node Master/Data Node Cluster R&D Master Nodes Data Node Data Node Data Node Master/Data Node Kibana Index namespacing GET sales:*,r_and_d:logs*/_search { "query": { … } } Optional dedicated cross-cluster search cluster
- 75. 75 How Cross-Cluster search works Cluster Sales Master Nodes Data Node Data Node Data Node Master/Data Node Cluster R&D Master Nodes Data Node Data Node Data Node Master/Data Node Kibana With many shards Batched Reduce Phase Optional dedicated cross-cluster search cluster
- 78. v6 and beyond
- 79. Doc Values v2.x
- 80. 80 Doc Values • Columnar store • Fast access to a field’s value for many documents. • Used for aggregations, sorting, scripting, and some queries • Written to disk at index time. • Cached in the file-system cache
- 81. 81 Doc Values - Dense Values Segment 1 Docs Field 1 Field 2 1 One A 2 Two B 3 Three C Segment 2 Docs Field 1 Field 2 1 Four D
- 82. 82 Doc Values - Dense Values Segment 1 Docs Field 1 Field 2 1 One A 2 Two B 3 Three C Segment 2 Docs Field 1 Field 2 1 Four D Merged Segment 3 Docs Field 1 Field 2 1 One A 2 Two B 3 Three C 4 Four D
- 83. 83 Doc Values - Sparse Values Segment 1 Docs Field 1 Field 2 1 One A 2 Two B 3 Three C Segment 2 Docs Field 3 Field 4 Field 5 1 Foo Bar Baz
- 84. 84 Doc Values - Sparse Values Segment 1 Docs Field 1 Field 2 1 One A 2 Two B 3 Three C Segment 2 Docs Field 3 Field 4 Field 5 1 Foo Bar Baz Merged Segment 3 Docs Field 1 Field 2 Field 3 Field 4 Field 5 1 One A Null Null Null 2 Two B Null Null Null 3 Three C Null Null Null 4 Null Null Foo Bar Baz
- 85. Sparse Doc Values Lucene 7
- 87. 87 Index sorting • Sort index by e.g. weight, recency, or popularity • Ultra-fast search - can terminate once enough hits found
- 88. 88 Index sorting • Sort index by e.g. weight, recency, or popularity • Ultra-fast search - can terminate once enough hits found • Even helps with total count and aggregations • Sort index by low cardinality terms - faster search • Better sparse index compression • Slower indexing, good for static indices
- 90. 90 Sequence Numbers • Internal Feature • Every operation gets a sequence number • In 6.0: Fast replica recovery on active indices • Lays groundwork for: • Primary-Replica syncing when Primary fails • Cross Data-Centre Recovery • Changes API
- 91. Upgrading
- 93. 93 Rolling Upgrades • Upgrade from 5.latest to 6.latest, without a full cluster restart • Why now and not earlier? • Testing needs to be ready • The team and the code must be ready • Growing user-base and faster release cycles required less painful upgrades
- 94. 94 Rolling Upgrades • What is 5.latest? • It’s the latest release of 5.x that is GA once 6.0.0 goes GA • All 6.x releases will allow upgrading from that 5.x release • There might be subsequent 5.x releases that are also eligible for upgrades to 6.x
- 95. 95 Rolling Upgrades • Caveats: • If using security, must have TLS enabled • Reserve the right to require full cluster restart in the future - but only if absolutely necessary • All nodes must be upgraded to 5.latest in order to upgrade • Indices created in 2.x still need to be reindexed before upgrading to 6.x
- 96. Cross Major Version Search v6.0.0
- 97. 97 Cross Major Version Search v5.2.0 Kibana Master Nodes Data Node Data Node
- 98. 98 Cross Major Version Search v5.2.0 Kibana v6.0.0 Master Nodes Data Node Data Node Master Nodes Data Node Data Node
- 99. 99 Cross Major Version Search v5.2.0 Master Nodes Data Node Data Node v6.0.0 v5.latest Kibana Master Nodes Data Node Data Node
- 100. 100 Cross Major Version Search v5.2.0 Master Nodes Data Node Data Node v6.0.0 Kibana Master Nodes Data Node Cross Cluster Client v5.latest
- 101. Elasticsearch SQL
- 102. 102 Elasticsearch SQL • Support Elasticsearch features as much as possible • Support standard SQL as much as possible • Same experience out of the box as Elasticsearch, which means • Lightweight & fast • Easy to pick up • Still Elasticsearch :)
- 103. Kibana
- 104. Visualizations
- 105. 105 Tag Cloud
- 106. 106 Heatmaps
- 107. 107 Vector Maps
- 110. 110 Advanced View
- 111. 111 Query Profiler
- 112. Kibana Canvas
- 113. 113 Logstash Monitoring • New visualization application on top of Elasticsearch data • Use Case: • live infographics • presentations with live data feeds • highly customized reports • Currently, in the prototyping phase • Release date: TBD
- 115. Beats
- 116. 116 Beats Packetbeat Network data Filebeat Log files Winlogbeat Windows Event Logs Heartbeat Uptime monitoring +40 community Beats Metricbeat Metrics
- 117. New Beat: Heartbeat
- 118. 118 Heartbeat host! ! !Your app! OS! TCP/TLS connect! ICMP ping! HTTP/S request!
- 119. Metricbeat
- 120. 120 Metricbeat modules MySQL Memcac he PHP- FPM CEPH Zoo keeper GolangDocker Apache Kafka HAProx y System Redis Couchb ase NGINX Postgre s Prometh eus Jolokia
- 121. Filebeat
- 123. Logstash
- 124. Persistent Queues
- 125. 125 Persistent Queues • Survive (temporary) machine failures • Limited impact on performance • View stats in Monitoring UI
- 126. Pipeline Visualizer
- 128. www.elastic.co