Elastic Meetup Belgium - December 2018

{
”host” : “data.be”,
“topic” : “What’s new in the Elastic Stack”,
“speaker” : “Arthur Eyckerman”,
“location” : “@betacowork”
}

Arthur Eyckerman
Sr. Solutions Architect
Living in Turnhout, center of NL | BE | LU
[Consultant, Freelancer, Architect]
Joined Elastic in May 2018
tuurleyck

ELK
Feb
2010
Elasticsearch
first release
Kibana
joins
Logstash
joins
January
2013
August
2013
Feb
2014
Jan
2015
Feb
2015
May
2015
June
2015
Feb
2014
Sept
2016
Oct
2016
May
2017
May
2017
Sept
2017
Oct
2017
Feb
2018
April
2018
July
2017
May
2018
https://www.elastic.co/about/history-of-elasticsearch

Elasticsearch
Heart of the Elastic Stack
Distributed & Scalable Highly Available Multi-tenancy
Developer Friendly Real-time, Full-text Search Aggregations

Logstash
Data processing pipeline
Ingest data of all shapes,
sizes, and sources
Parse and dynamically
transform data
Transport data to any
output
Secure and encrypt data
inputs
Build your own pipeline More than 200+ plugins

Kibana
Window into the Elastic Stack
Visualize and analyze Geospatial Customize and Share
Reports
Graph Exploration UX to secure and manage
the Elastic Stack
Build Custom Apps

Elastic Stack
Feb
2010
Elasticsearch
first release
Kibana
joins
Logstash
joins
Monitoring
feature first
release (XP)
Found acquired;
now Elastic Cloud
Packetbeat joins;
Beats introduced
Elasticsearch
1.0
January
2013
August
2013
Feb
2014
Jan
2015
Feb
2015
May
2015
June
2015
Feb
2014
Sept
2016
Prelert
acquired for ML
capabilities
Elastic Stack
5.0 first release
with X-Pack
Elastic Cloud
Enterprise
(ECE) release
Oct
2016
May
2017
May
2017
Sept
2017
Oct
2017
Feb
2018
April
2018
July
2017
May
2018
Security
feature first
release (XP)
Alerting
feature first
release (XP)

Beats
Lightweight data shippers
Ship data from the source Ship and centralize in
Elasticsearch
Ship to Logstash for
transformation and parsing
Single command modules to
visualize data
Libbeat: API framework to
build custom beats 70+ community Beats

Elastic Stack + Solutions
Feb
2010
Elasticsearch
first release
Kibana
joins
Logstash
joins
Monitoring
feature first
release (XP)
Found acquired;
now Elastic Cloud
Packetbeat joins;
Beats introduced
Elasticsearch
1.0
January
2013
August
2013
Feb
2014
Jan
2015
Feb
2015
May
2015
June
2015
Feb
2014
Sept
2016
Prelert
acquired for ML
capabilities
Elastic Stack
5.0 first release
with X-Pack
Elastic Cloud
Enterprise
(ECE) release
Opbeat
acquired for
APM
GCP on Elastic
Cloud release
Swiftype
acquired
APM first
release
Opening of X-
Pack code
Machine learning
first release (XP)
Oct
2016
May
2017
May
2017
Sept
2017
Oct
2017
Feb
2018
April
2018
July
2017
App Search
first release
May
2018
Security
feature first
release (XP)
Alerting
feature first
release (XP)

SaaS
Elastic Cloud
Metrics
Logging
APM
Site
Search
App
Search
Business
Analytics
Elastic Stack
Kibana
Elasticsearch
Beats Logstash
Self Managed
Elastic Cloud
Enterprise
Standalone
Visualize & Manage
Store, Search, & Analyze
Ingest
Enterprise
Search
Security
Analytics
Future
Deployment
Solutions

Elastic was in the perfect place to capitalize on this industry
shift. Elasticsearch, together with a couple of smaller but
strategically critical software components - Logstash and
Kibana - were quickly becoming the industry standard known
as the ELK Stack, and today called the Elastic Stack. What
proved to be so useful about the Elastic Stack was its
enormous flexibility and ease of use. At its heart, Elasticsearch
is a search engine, which means that it can read structured
and unstructured documents (often referred to as “schema-
free” in the industry lingo), index them (an index is like a map
of where the content rests), find the right content, and return a
search result. What makes Elasticsearch special is the fact
that it’s distributed - it allows clusters to act like one big
system; it’s multi-tenant - so many developers of parts of an
organization can use it without interfering with each other; and
it has a programmable web (HTTP) interface - so that
developers can write programs that query Elasticsearch rather
than just respond to typed searches.
https://www.indexventures.com/blog/elastic-the-evolution-of-open-source
TL;DR - Elastic -- The Evolution of Open Source

Community, Open Source .. and SaaS
What do you expect ?

Why use open source ?
• Lots of resources - discuss.elastic.co
• Open Code - github.com/elastic
• Meetups - and great people
• Free, no license required.
• Get started now.
• Your laptop looks better with stickers !

10 DECEMBER 2018 NEWS
Elastic doubles down on cloud native with
Helm charts and CNCF membership
docker.elastic.co
https://www.elastic.co/blog/alpha-helm-charts-for-elasticsearch-kibana-and-cncf-membership
https://raw.githubusercontent.com/cncf/trailmap/master/CNCF_TrailMap_latest.png
helm.elastic.co
Logs
Metrics
APM
Distributed & Scalable
Joining Cloud Native
Computing Foundation
(CNCF) is a natural
evolution for us, given our
open source history of
building products and
ongoing efforts around
integrating with cloud
native technologies.

Helm Charts
To get started with Elasticsearch and Kibana Helm charts:
1. Add the Elastic Helm Chart Repo:
helm repo add elastic https://helm.elastic.co
2. Install Elasticsearch:
helm install --name elasticsearch elastic/elasticsearch
3. Install Kibana:
helm install --name kibana elastic/kibana

Bring Events + Traces + Metrics together for
100% Observability in IT Operations
TRACES
(APM)
EVENT
S
METRICS

Getting started with Elasticsearch
Everybody loves CURL. Right ?
# When you need to index a lot of docs, you should use the bulk API
curl -XPOST "https://39d98b388e194025888558780eaf266b.europe-west1.gcp.cloud.es.io:9244/inspections/_doc/_bulk" -H 'Content-Type: application/json' -d’
{ "index": { "_id": 1 }}
{"business_address":"315 California St","business_city":"San
Francisco","business_id":"24936","business_latitude":"37.793199","business_location":{"type":"Point","coordinates":[-122.400152,37.793199]},"business_longitude":"-
122.400152","business_name":"San Francisco Soup Company","business_postal_code":"94104","business_state":"CA","inspection_date":"2016-06-
09T00:00:00.000","inspection_id":"24936_20160609","inspection_score":77,"inspection_type":"Routine - Unscheduled","risk_category":"Low
Risk","violation_description":"Improper food labeling or menu misrepresentation","violation_id":"24936_20160609_103141"}
{ "index": { "_id": 2 }}
{"business_address":"10 Mason St","business_city":"San
122.409061","business_name":"Soup Unlimited","business_postal_code":"94102","business_state":"CA","inspection_date":"2016-11-
23T00:00:00.000","inspection_id":"60354_20161123","inspection_type":"Routine", "inspection_score": 95}
{ "index": { "_id": 3 }}
{"business_address":"2872 24th St","business_city":"San
122.409752","business_name":"TIO CHILOS GRILL","business_postal_code":"94110","business_state":"CA","inspection_date":"2016-07-
Risk","violation_description":"Unclean nonfood contact surfaces","violation_id":"1797_20160705_103142"}
{ "index": { "_id": 4 }}
{"business_address":"1661 Tennessee St Suite 3B","business_city":"San Francisco Whard
Restaurant","business_id":"66198","business_latitude":"37.75072","business_location":{"type":"Point","coordinates":[-122.388478,37.75072]},"business_longitude":"-
122.388478","business_name":"San Francisco Restaurant","business_postal_code":"94107","business_state":"CA","inspection_date":"2016-05-
27T00:00:00.000","inspection_id":"66198_20160527","inspection_type":"Routine","inspection_score":56 }
{ "index": { "_id": 5 }}
{"business_address":"2162 24th Ave","business_city":"San
122.481299","business_name":"Soup House","business_phone_number":"+14155752700","business_postal_code":"94116","business_state":"CA","inspection_date":"2016-09-
Risk","violation_description":"Unapproved or unmaintained equipment or utensils","violation_id":"5794_20160907_103144"}
{ "index": { "_id": 6 }}
{"business_address":"2162 24th Ave","business_city":"San
122.481299","business_name":"Soup-or-Salad","business_phone_number":"+14155752700","business_postal_code":"94116","business_state":"CA","inspection_date":"2016-09-
Risk","violation_description":"Unapproved or unmaintained equipment or utensils","violation_id":"5794_20160907_103144"}'

Data Visualizer for Files
Experimental | Basic (free)
To get the most out of Machine
Learning you need to understand your
data
Data Visualizer now provides the
ability to upload files (up to 100MB)
and use ML File Structure API for field
identification and to create indices
and index patterns
Works with CSV, text delimited, or
JSON files

ODBC
REST
Translate API
CLI
JDBC
ODBC
SQL Endpoint
Basic (free)

sql>
SELECT description FROM features WHERE product =
'elasticsearch' AND version >= '6.3.0';
description
---------------
SQL
Data Rollups
Java 10 & 11 Support

• API for creating an Elasticsearch process
to periodically store aggregate statistics
• Primary benefit is space savings
‒ Faster queries
‒ Potentially less nodes to manage
‒ Smaller snapshots
‒ Longer retention times
‒ etc.
• Query rolled up data and “live” data
together in a single query.
Data Rollups
Basic (free)

Raw Minute Hour Day
Docs: 9,041,000 1,448,285 49,554 8,447
Size: 2.23gb 1.25gb 48.40mb 9.10mb
Docs % change: -83.98% -99.45% -99.91%
Size % change: -43.68% -97.84% -99.59%
(avg ~200 docs per minute, 32 days of data, single host)
(20 grouping fields, 62 numerics @ min/max/avg == 186
metrics)
Rolling up Metricbeat dataRolling up metricbeat data

Data Rollups API
// Define rollup job, start/stop
/_xpack/rollup/job/{job_id}
/_xpack/rollup/job/{job_id}/_start
/_xpack/rollup/job/{job_id}/_stop
// rollup jobs configured for an index or index pattern
/_xpack/rollup/data/{index}/
// Search endpoint using standard ES Query DSL
/{index}/_rollup_search

Cluster 1
Cluster 2
_source Only Snapshot
Basic (free)
_source only snapshots can be 50%
smaller than full snapshots
Requires a reindex to make the data
searchable again
Trades off restore time for smaller
storage space / costs
source
only
snapshot
full
snapshot
Snapshot
Restore to a different
cluster
Restore to the
same cluster

Java Updates
OSS
Adds support for Java 11
Adds support for G1 garbage collector
(supported on Java 10+)
Even with G1, we don’t recommend
extremely large heaps

Canvas: Create live pixel-perfect presentations
Beta | Basic (free)

Canvas
Beta | Basic (free)
Showcase your data, live & pixel-perfect
Tell the story of your data, your way
Showcase your work on the big screen
Support Elasticsearch SQL
Automate reporting (adios copy/paste)
Built for the community, extend Canvas with
plugins

Spaces
Basic (free) / Gold
Organize Kibana visuals,
dashboards, etc into separate,
independent spaces
Control user access to spaces
using role-based access control
Simplify Kibana multitenant use
Use Cases:
● Organization
● Phasing (dev, stage, prod, etc)
● Security (restrict access)

Rollups in Kibana
Beta | Basic (free)
Automatically roll up data into coarser
time buckets as it ages
- Save on storage space & costs
- Smaller indices = faster analytics
6.3 - Rollups API in Elasticsearch
6.5 - Rollups support in Kibana
- Job management UI
- Visualize rolled up indices
Aggregation functions:
- Avg, min, max, sum, count

Beats Central Management
Beta | Gold
Centrally manage your fleet of
Beats
• Enroll & unenroll Beats
• Add, modify & delete
configs
Manage via UI and APIs
Currently supports:
• Filebeat (inputs, modules)
• Metricbeat (modules)
Metricbeat
Metricbeat
Filebeat
Filebeat
Filebeat
Beats
Admin
Metricbeat

Beats Central Management
Beta | Gold
Centrally manage your fleet of
Beats
• Enroll & unenroll Beats
• Add, modify & delete
configs
Manage via UI and APIs
Currently supports:
• Filebeat (inputs, modules)
• Metricbeat (modules)

Functionbeat
Beta | Basic (free)
New Beat type that deploys as
a function in serverless
platforms
Easily collect and stream cloud
data to Elasticsearch
6.5 release supports AWS
Lambda
Collect events from Cloudwatch
Logs, SQS, and more to come
soon
Automate via single AWS CLI
command

Heartbeat GA
OSS
New config reload provide an
easy way to create, update,
delete Heartbeat monitors
Autodiscovery for Docker &
Kubernetes
Add Data tutorials in Kibana

Multiple Pipelines● Building on Multiple Pipelines
● Reduce repetition, isolate blockages and
simplify pipelines
● Usage patterns (also in docs!)
○ Distributor
○ Output Isolator
○ Forked Path
○ Collector
● Don’t create cycles
Logstash
JDBC Pipeline
Netflow Pipeline
Apache Pipeline
Logstash Inter-pipeline Communication
Beta

Inter-connected Pipelines: Distributor
Logstash
JDBC Pipeline
Netflow Pipeline
Apache Pipeline
Distributor
Pipeline
OR
OR
Beta

Inter-connected Pipelines: Output Isolation
Logstash
S3-Out Pipeline
HTTP-Out Pipeline
ES-Out Pipeline
Apache Pipeline
AND
AND
Beta

Inter-connected Pipelines: Forking
Logstash
Redacted Pipeline
Normal Pipeline
Apache Pipeline AND
Beta

Inter-connected Pipelines: Collector (Common Output)
Beta
Logstash
JDBC Pipeline
Netflow Pipeline
Apache Pipeline
Output Pipeline

Azure Monitoring Module
OSS
• Easily monitor your Azure deployments
with the new Azure monitoring module
• Consume from Azure Event Hubs with a
new input plugin
‒ Basic configuration generally recommended
‒ Advanced configuration enables more
granular tuning when consuming from multiple
Event Hubs
• Amazon Linux OS support
input {
azure_event_hubs {
event_hub_connections => ["Endpoint=sb://
threads => 8
decorate_events => true
consumer_group => "logstash"
storage_connection => "DefaultEndpointsPr
}
}

SNMP Input Plugin
OSS
Centrally poll SNMP devices like
routers, switches, phones, and
printers.
Great for network monitoring &
management use cases
Support SNMP v1, v2c, v3 over
TCP/UDP
Servers, Switches,
Routers, Gateways, etc
SNMP
Agents

Infrastructure Solution
Beta | Basic (free)
Curated experience for infra
operators
Bird’s eye view of 10K+ infra
elements
Native support for Kubernetes,
Docker
Drill down to metrics, logs, APM
views
Ad hoc and structured search

Logs Solution
Beta | Basic (free)
Compact log viewer optimized
for live log event troubleshooting
Console-like display
Live log streaming (like tail -f)
Infinite scroll for historical logs
Ad hoc and structured search

Supported Languages & Frameworks

Track key
application
metrics
65
● Response time for requests
● Unhandled errors & exceptions
● Visualize call hierarchy (waterfall chart)
● Identify code bottlenecks
● Drill down to the code level

66
APM App (UI)
● Tailored for APM
● Made for developers
● Free (Basic license)
● Integrated Search Bar

Mix APM with
other data &
features
67
● APM data is just another
Elasticsearch index
● Customize dashboards with other
visuals to show what YOU want
● Mix with other Elastic Stack features,
such as machine learning, alerting…
● Built-in integration with ML & Alerting

RUM (Real User Monitoring)
The only way to measure actual end-user experience
Browser
interactionAgent
Response
from web
server
Agent
Request to
web serverAgent
Browser
renderingAgent
Time to interaction: ~6s

69
RUM
● See where the browser
spends its time
● Similar waterfall view
● Annotations at key DOM
events

Java & Go Agents GA
OSS
Java support: Java 7-10,
Several
technologies/frameworks:
Servlet API, Spring Web MVC,
Spring Boot, Tomcat, WildFly,
Jetty, Websphere, JDBC & more
Go support: Go 1.8+, httprouter,
Echo, Gin,gorilla/mux,
database/sql, GORM, gocql,
gRPC

Distributed Tracing
Beta | Basic (free)
Consolidated waterfall showing
all instrumented services
Ability to jump to full trace
context when in a sub-context
OpenTracing compatible

Elastic APM Monitoring
Basic (free)
Track the health of your Elastic
APM deployments
Monitoring stats include rates,
events, and requests.

Become an Elastic Pioneer
7.0 is shipping with many highly
requested features, and we welcome
your testing and feedback during
the preview cycles. When our
community gets involved, good
things happen.
Install a 7.0 preview version, try
new features, and provide feedback
via GitHub to become an Elastic
Pioneer. Learn more about the
Elastic Pioneer Program.
Oh, and Elastic Pioneers may win
some limited edition Elastic swag
(and bragging rights, of course).

Elastic Meetup Belgium - December 2018

Elastic Meetup Belgium - December 2018

Empfohlen

Empfohlen

Weitere ähnliche Inhalte

Was ist angesagt?

Was ist angesagt? (20)

Ähnlich wie Elastic Meetup Belgium - December 2018

Ähnlich wie Elastic Meetup Belgium - December 2018 (20)

Kürzlich hochgeladen

Kürzlich hochgeladen (20)

Elastic Meetup Belgium - December 2018