1.
>>> Realtime Monitoring
Storing 2TB of logs a day in
Elasticsearch
@aliostad
Ali Kheyrollahi, ASOS

2.
@aliostad

3.
@aliostad
The joy of hitting F5

4.
@aliostad
The joy of a single process

5.
@aliostad
The joy of a having a
production-size database
locally

6.
@aliostad
The joy of having a
dev machine build
running all services

7.
@aliostad
/// What if your systems
is “Microservices”?

8.
@aliostad

9.
@aliostad
- +40 platform teams
- 1oos of microservices
- some services >10k rps

10.
@aliostad
> stackoverflow
> £1.5 bln
global fashion
destination
> 35% year-on-year

11.
@aliostad
/// elements of
observability

12.
@aliostad
/// observability
>>> Control Theory
“a measure for how well internal states of a
system can be inferred by knowledge of its
external outputs”

13.
@aliostad
Logging
Telemetry
Tracing
/// mixed concerns

14.
@aliostad
Tracing
Logging
Credit: Peter Bourgon
Events
Aggregations
Request Scope
Telemetry
Alerting
/// Scope

15.
@aliostad
Logging Telemetry Tracing Alerting
Log4Net ✓
Time-series DBs ✓ ✓ ✓
Zipkin ✓ ✓ ✓ ✓
Prometheus ✓ ✓ ✓
Elasticsearch ✓ ✓ ✓ ✓
New Relic* ✓ ✓ ✓
Circonus* ✓ ✓ ✓
* paid services
/// comparison

16.
@aliostad
1
2
3
At source (perf counters)
At the storage (Circonus)
In the visualisation tool (Kibana)
/// aggregations
4 In the pipeline (Riemann)

17.
@aliostad
/// data
sources

18.
@aliostad
/// use cases
• Metrics (Visualisation)
• CPU, number of errors
• Response time percentiles
• Full-text search capability (logs and errors)
• Correlating across services
• Alerting when there is an SLO breach

19.
@aliostad
/// azure logs
• Azure Diagnostics (WADLogs table)
• IIS logs
• VM Windows Event Logs
• Performance Counters (standard + custom)

20.
@aliostad
/// application logs
Microservice
ETW
SLAB Azure
Table Sink
ETW
Application
Logs
EC
e.g. CRIT_ORD_API_DatabaseDown

21.
@aliostad
/// instrumentation logs
Microservice Perf Counters
SLAB Azure
Table Sink
ETW
Instrumentation
Logs
Azure Performance
Counter Logs
PerfIt
Azure
Agent

22.
@aliostad
/// ingest
process

23.
@aliostad
/// pull vs push

24.
@aliostad
/// logstash
QUEUE
VM
Logstash
collectd
syslog
Logstash
app logs
nginx
To
Elasticsearch
UDP
File-tailing

25.
@aliostad
/// ConveyorBelt
Performance
Counters
ConveyorBelt
Azure
WAD logs
ETW Logs
Elasticsearch
Instrumentation
Logs
IIS Logs
Woodpecker
Outputs (Pull Logs)
Sources Config
Up to 2TB/day

26.
@aliostad
/// ConveyorBelt
Source
Source
Source Config
Scheduler
Parser
units of
work
To Elasticsearch
Source
Actor
Actor
Actor

27.
@aliostad
/// Woodpecker
Source
Source
Source Config
Pull
Telemetry
record
Azure
Table
(Regular
Intervals)
Source

28.
@aliostad
/// elasticsearch
intro

29.
@aliostad
/// elasticsearch
• Linearly-scalable and HA* search (and visualisation)
• ELK Stack
• Open Source (enterprise features require license)
• Speaks JSON
• REST API and very developer-friendly

30.
@aliostad
/// cluster
• Cluster: No ZK
• Gossip / discovery
• Node type:
• master - leader election
• data
• client

31.
@aliostad
/// data hierarchy
• Index
• Shard
• Replica
• Type/Mapping
• Document: JSON, immutable,
versioned
INDEX
MAPPING
MAPPING
MAPPING
…
Document
Document
Document
…

32.
@aliostad
/// data types
• JSON data types: bool, long, float, string*, datetime
• Array?
• String tokenisation/analysers
• Best of both world?
• Object
• nested
{
“a”: {
“b” : {
“c”: 42
}
}
}

33.
@aliostad
/// doc operations
• Upsert
• Delete
• Partial Update
• Search (JSON-based query DSL)

34.
@aliostad
/// DEMO 1

35.
@aliostad
/// more
advanced

36.
@aliostad
/// index shard/replica
Write Read Master
Shard Shard
Replica Replica
Index Index

37.
@aliostad
/// index
• Daily indices
• Hot/Cold with index alias
• Creation => templates
• Settings:
• refresh_interval

38.
@aliostad
/// mapping/type
• Schema
• How many mappings per index?
• Dynamic mapping
• Operations
• Upsert
• Delete

39.
@aliostad
/// templates
PUT https://es_cluster:9200/_template/my_template
{
“template”: “my_index_*”,
“settings”: {…}
“mappings”: {
“mapping_1”: {…},
“mapping_2”: {…}
}
}

40.
@aliostad
/// bulk api
• Always use Bulk API to index documents
• Batches of 1K-5K documents
• Watch-out for error 429 and back-off pattern
• Check bulk rejects [change bulk queue length]

41.
@aliostad
/// DEMO 2

42.
@aliostad
/// physical
architecture

43.
@aliostad
/// resources
node type
• Data: Disk, RAM, CPU, Network
• Master: CPU, Network, (RAM)
• Client: Network, CPU, (RAM)
• Kibana: CPU, Network, RAM

44.
@aliostad
/// simple
data/master/kibana
• CPU
• RAM
• Disk
• Network

45.
@aliostad
/// next level
d a t a / m a s t e r
kibana

46.
@aliostad
c l i e n t
k i b a n a
m a s t e r
d a t a
traffic
traffic

47.
@aliostad
3x client
2x kibana
20x data (hot)
traffic
traffic
10x data (warm)
/// our setup
3x master
ARM
Template
Desire State
Configuration

48.
@aliostad
/// hot/warm
• Hot => CPU, Warm => Memory
• Index Allocation/Routing
• At the index:
"index.routing.allocation.require.box_type" : "warm"
• At the node (elasticsearch.yml)
box_type: warm

49.
@aliostad
/// security
• x-pack: SSL + username/password security (basic,
Kerberos)
• No Federated Authentication
• Proxy (nginx, apache, etc)
• IP-whitelisting

50.
@aliostad
/// administration
• Like all: logs, slow query logs, etc
• top, htop, iostat
• collectd + local logstash
• two clusters, each watching the other
• curator for hot/cold and deleting old indices

51.
@aliostad
/// tracing

52.
@aliostad
/// ActivityId
Microservice
Id
IdId Thread Local Storage
Id
To Other APIs
Id
Event

53.
@aliostad
/// alerting

54.
@aliostad
/// watcher
• Trigger
• Input
• Condition
• Action

55.
@aliostad
/// watcher notes
• All watches get executed on the active master
• Use Action Throttling to limit alerts
• Use watch templates when you see common patterns
• Use transforms and metadata to include context in
actions/emails

56.
@aliostad
/// lessons
learnt

57.
@aliostad
/// Do you speak CAP?
• Consistency? Treat all data dispensable. Back up data
that gets mastered in Elasticsearch. Not a document db.
• Highly-Available? For >99.9% availability use redundancy
• Partition-Intolerance? Node intercommunication highly
chatty, ideally keep in the same data centre and even in
the same VPC (aws)/VNet (azure)

58.
@aliostad
/// Beware
• Split brain common
• Data corruption possible
• Backup data that gets mastered in ES (kibana indices)
• It seems safest High Availability is redundancy
(expensive)

59.
@aliostad
Thank you :)
Questions…?

60.
@aliostad
Credits
• Picture: Embroidery thread macro - https://www.flickr.com/photos/39908901@N06/
• Picture: Calculate Red - https://www.flickr.com/photos/93277085@N08/10398245145
• Picture: 1950's wristwatch workings - https://www.flickr.com/photos/
134832191@N08/27301612554/
• Picture: Tokyo Tower_58 - https://www.flickr.com/photos/ajari/2756645901
• Picture: 1Bamboo and Rust - https://www.flickr.com/photos/hammershaug/5816522126/
• Picture: IMG_1899 - https://www.flickr.com/photos/johnas/9650255412/
• Picture: fan2 https://www.flickr.com/photos/sidelong/444054290/
• Picture: Glass jar filled with pasta https://www.flickr.com/photos/76588981@N02/16766079567/
• Picture: Rusty cogs https://www.flickr.com/photos/paperpariah/25375888671/
• Picture: Do you see the world in different colours? https://www.flickr.com/photos/luopl/6012467435/
• Picture: danger https://www.flickr.com/photos/armydre2008/9650951334/
• Link: ETW equivalent for Linux http://blogs.microsoft.co.il/sasha/2017/04/02/tracing-net-core-on-linux-
with-usdt-and-bcc/