The recent technological improvements have make it really difficult to properly monitor application performances. Luckily great tools such as Elastic APM are taking care of the problem efficiently. Learn in this presentation how to diagnose, analyse and monitor your applications using Elastic APM.
Application performance monitoring with Elastic APM and the ELK stack
1. Senacor Technologies AG 04.05.2019 1
Application performances
monitoring with
Elastic APM
And the ELK stack
Alain Lompo
Senacor Technologies AG
#DevOps PRO Europe alain.lompo@senacor.com @alainlompo
2. Senacor Technologies AG 04.05.2019 2
Plan
Motivation
Introducing the ELK Stack
Setting up the ELK Stack
Introducing Elastic APM
Setting up Elastic APM
Monitoring Spring-boot apps with Elastic APM
4. Senacor Technologies AG 04.05.2019 4
The increased complexity of software development
Programming
Language(s)
Methodology
Team
IDE
Application server
Frameworks
And Dev technologies
Integration with
External services
Orchestration
Virtualisation
Containairization
Build tools
Versioning system
Qualimetry tools
Dev Environment
Commit Environment
Acceptance Environment
PRE-PROD Environment
PROD Environment
Solution
architecture
Application
architecture
5. Senacor Technologies AG 04.05.2019 5
Monitoring complex apps is…complex
Internet
Web server Email server Database server Search server
Application server Application server
Application server Application server
SAN
?
?
?
9. Senacor Technologies AG 04.05.2019 9
Introducing Elasticsearch
Elasticsearch is an open source search server based
on the Apache Lucene server
It is cross-platform (build in Java)
It is designed from origin to be scalable and distributed
It is data source agnostic (can take any data source and
search through it)
10. Senacor Technologies AG 04.05.2019 10
Elasticsearch’s role in the ELK stack
Kibana Elasticsearch Logstash / Beats
UI for Datas and
Dashboard visualisation
Powerful and flexible
Search server
Collection, transformation
And shipment of Log datas
http (S) http (S)
/ REST
Elastic
Transport
11. Senacor Technologies AG 04.05.2019 11
Other interesting Elasticsearch features
Real time analytics
Full text search
Document oriented
Restful API
16. Senacor Technologies AG 04.05.2019 16
Starting simple
GET _search
{
"query": {
"match_all": {}
}
}
17. Senacor Technologies AG 04.05.2019 17
Starting simple
POST /talks/devops
{
"title": "Application monitoring with ELK stack
and Elastic APM",
"speaker": "Alain Lompo",
"category": "APM",
"tags": ["Elastic", "Search", "Monitoring", "APM"],
"duration_in_minutes": 45
}
18. Senacor Technologies AG 04.05.2019 18
Sample configuration for logstash (in a .conf file)
input {
stdin {}
}
output {
elasticsearch {
hosts => ["http://localhost:9200"]
}
stdout { codec => rubydebug }
}
19. Senacor Technologies AG 04.05.2019 19
Connecting Kibana to Elasticsearch server (in kibana.yml)
elasticsearch.hosts: ["http://localhost:9200"]
20. Senacor Technologies AG 04.05.2019 20
Connecting apm-server to elasticsearch server (in apm-server.yml)
output.elasticsearch:
hosts: ["localhost:9200"]
22. Senacor Technologies AG 04.05.2019 22
On top of the stack…
Elastic APM is built on top of the ELK stack:
Uses Elasticsearch as its datastore
Uses logstash to collect, parse and transform logs
Uses Kibana to render and visualise all kinds
of metrics related dashboards
23. Senacor Technologies AG 04.05.2019 23
Other interesting features of Elastic APM
Elastic APM…
Can monitor in real time the performances
of thousands of applications.
Automatically gathers performance relevant
informations.
Automatically instrumentalise our apps and
measure response times for requests
24. Senacor Technologies AG 04.05.2019 24
How does Elastic APM fits into the Elastic ecosystem?
Logs
Metrics
Packets
Beats
Logstash
Datastore JMX
Elasticsearch Kibana
APM agent APM server
27. Senacor Technologies AG 04.05.2019 27
What’s an agent?
APM agents are…
Open source libraries written in the same
language as our application.
Agents are hooks in your application that
collect performance metrics and errors.
All collected datas are sent to the APM server
28. Senacor Technologies AG 04.05.2019 28
Deep diving into the APM server…
The APM server is…
An application written in GO that runs on your
server just like the elasticsearch server.
Listens by default on port 8200.
Expose a simple json based Restfull http Api
29. Senacor Technologies AG 04.05.2019 29
Deep diving into the APM server…
The APM server …
Receives datas from agents periodically.
Builds Elasticsearch documents from the
datas collected by agents.
The documents are stored in the
Elasticsearch cluster
30. Senacor Technologies AG 04.05.2019 30
Extended visualisation capabilities with Kibana and Xpack…
Using Kibana you can…
Visualize the collected datas
Create Dashboards
Detect anomalies and create alerts
31. Senacor Technologies AG 04.05.2019 31
Drilling down into traces…
With Kibana’s APM UI you can…
Drill-down into transaction traces
to see executed methods
33. Senacor Technologies AG 04.05.2019 33
Analysing results in Kibana
Elastic APM will provide us out of the box informations
about:
The web request
The background jobs
The errors
34. Senacor Technologies AG 04.05.2019 34
Analysing results in Kibana
Elastic APM will provide us out of the box informations
about:
The average for any measure
The 95th percentile
The 99th percentile
The number of requests per minute
35. Senacor Technologies AG 04.05.2019 35
Analysing results in Kibana
It is also possible to:
Create customized dashboard
Choose your indexes
Choose the datas to visualize
37. Senacor Technologies AG 04.05.2019 37
Handling identified issues
The analysis result:
Will highlight bottlenecks, errors and issues that are
deeply rooted in the code
Elastic APM allows us to zoom in and pinpoint precisely
the origin of problems
39. Senacor Technologies AG 04.05.2019 39
Setting up notifications and alerts
Notifications and alerts are very usefull and:
Will periodically inform us when a metrics value exceeds
a specified threshold
Elastic APM and the ELK stack allows us to configure
them with a great degree of flexibility
41. Senacor Technologies AG 04.05.2019 41
Wrapping up
Software development and lifecycle has grown complex
over time
We can properly monitor our application with Elastic APM
Elastic APM makes APM simple and beautful
It is built on top of the ELK stack: which provides a
powerful and flexible search server, datasource agnostic
data collecting and transformation pipeline and a
fantastic report and dashboard generation capabilities
Hello and welcome to this presentation about Application performances monitoring with Elastic APM and the ELK stack.
Okay, here is what will be covered in this presentation
What do we actually need to do in order to build an enterprise application today?
A modern enterprise application deployment infrastructure.
Elasticsearch is by definition cross-platform, scalable and distributed and datasource agnostic
Elasticsearch has a central role in the ELK stack.
Real time analytic: it is the use of, or the capacity to use, data and related resources as soon as they enter the system. The real-time adjective itself refers to the impression of immediate or near-immediate responsivity of the system by the user.
In full-text search it is possible to query and to perform linguistic searches against documents. It can include single or multiple words or phrases and returns documents that match search conditions. ElasticSearch is a full text search engine based on Apache Lucene, a free and open-source information retrieval software library.
Logstash is a server-side data processing pipeline that can fetch datas from a multitude and various sources simultaneously. It can transform them before sending them to a “stash” server (in our case Elasticsearch, of course)
Kibana lets you visualize your Elasticsearch data and navigate the Elastic Stack, so you can do anything from learning why you're getting paged at 2:00 a.m. to understanding the impact rain might have on your quarterly numbers.
Kibana is the UI component of the ELK stack. It allows you to visualize your Elasticsearch datas, create dashboard, reports, etc. You can do anything from learning why your Jms queue is not reacting as fast as usual to understanding the impact an train delay might have on your eCommerce Apps optimale sales.
A simple way to understand the ELK stack would be to compare it to a three tiers application
Kibana would be the presentation tier
Elasticsearch would be the business tier
Logstash would be the data tier
