Describes a new logger written in Scala.

1. Logging in Scala
John Nestor 47 Degrees
www.47deg.com
August 9, 2016
147deg.com

2. 47deg.com © Copyright 2016 47 Degrees
Outline
• Introduction
• Architecture
• Features
• Implementation
• Demos
• Cross Service Log Aggregation
2

3. Introduction
3

4. 47deg.com © Copyright 2016 47 Degrees
Existing Scala Loggers
• Java loggers (used in Scala code)
• Log4j (used in Spark)
• SLF4J and Logback
• Scala loggers
• Abandoned: Configgy, Logula (Coda Hale)
• Log4s (wraps SLF4J)
• scala-logging (Heiko Seeberger) (wraps Log4j)
• Special Scala loggers
• Akka actor logging (can route to Logback)
• Play logging (uses Logback)
4

5. 47deg.com © Copyright 2016 47 Degrees
Why a New Logger?
• Pure Scala. (No Java code)
• Richer messages (Json rather than text strings)
• Aggregate logs across multiple (micro-)services
• Clean design with rich features
5

6. 47deg.com © Copyright 2016 47 Degrees
New Scala Logger History
• Whitepages Logger - part of a new Scala web service
framework
• Persist Logger (work supported by 47 Degrees)
• clean up and improve API
• removed Whitepages specific stuff
• added complete documentation
• Full source and documentation on Github
• Maven central
"com.persist" % "persist-logging_2.11" % "1.2.4"
• Apache 2 license
6

7. 47deg.com © Copyright 2016 47 Degrees
Rich Logging Example
log.error(s”Too many values Size:${vals.size} Max:${vals.max}”)
log.error(map(“msg”->”Too many values”,
”Size”->vals.size,
”Max”->vals.max),
id=currentId)
{“msg”:”Too Many Values”, “Size”: 10, “Max: “33}
7

8. Architecture
8

9. 47deg.com © Copyright 2016 47 Degrees
Persist Logger Architecture
9
logging.conf
logback.xml
Scala
Logger API
Slf4j API
Akka Actor
Logging API
Custom
Logback
Appender
LogBack
Custom Akka
Logging Actor
Logging Actor
Stdout
Appender
File
Appender
Kafka
Appender
Other
Appenders
Logger
State
Scala
Logger
Control API

10. Features
10

11. 47deg.com © Copyright 2016 47 Degrees
API Log Call
• Level
• Rich Message
• Id (optional)
• Per request level control
• Per request log aggregation
• Exception (optional)
• use RichException for a RichMessage
11

12. 47deg.com © Copyright 2016 47 Degrees
Rich Messages
• String
• Int, Long, Float, Double, BigDecimal
• null, Boolean
• Seq[RichMessage]
• Map[String,RichMessage]
12

13. 47deg.com © Copyright 2016 47 Degrees
Why Json?
• Easy to support rich metadata in addition to the
message
• Easier to parse in downstream tools
• No need to manually escape strings
• Naturally supports multiline messages
• Standard text format
• But many internal possibilities
• This logger uses Persist Json (only internally)
• Rich messages use standard Scala types for in
internal API
13

14. 47deg.com © Copyright 2016 47 Degrees
Kind of Logs
• Error log
• Alternative logs
• GC
• Timing (for fine grain timing)
• Server (request-response, duration)
• Client (request, response, duration)
• User defined
14

15. 47deg.com © Copyright 2016 47 Degrees
Standard Fields
• @category (error, gc, …)
• @host
• @service
• @severity (info, debug, warn, error, …)
• @timestamp (msec)
• class, file, line, actor
• msg
• trace (for exceptions, Json)
15

16. 47deg.com © Copyright 2016 47 Degrees
Appenders
• Stdout
• no server/service specific fields
• color (optional)
• summary
• File (daily rotation)
• Kafka
• Custom, user defined
• Control destination
• Control contents and format
16

17. 47deg.com © Copyright 2016 47 Degrees
Level Control
• logging.conf (default options), can override
• uses Typesafe config
• levels: API, SLF4J, Akka
• Change level via API
• Per request custom level
• Custom filters
• Based on level, and content
17

18. 47deg.com © Copyright 2016 47 Degrees
Timing
• Fine grain timing
• Uses request ids to aggregate
• Time in microseconds
• Time log
• Can be used with concurrency (futures and actors)
18

19. 47deg.com © Copyright 2016 47 Degrees
Other Loggers
• Captures messages from other legacy loggers that
might be used in new project libraries
• SLF4J (captured via Logback appender)
• Akka Actor Logging (captured by custom Akka log
handing actor)
19

20. 47deg.com © Copyright 2016 47 Degrees
Look At Documentation
• logging.conf
• Overview
• API
20

21. Implementation
21

22. 47deg.com © Copyright 2016 47 Degrees
Persist Logger Architecture
22
logging.conf
logback.xml
Scala
Logger API
Slf4j API
Akka Actor
Logging API
Custom
Logback
Appender
LogBack
Custom Akka
Logging Actor
Logging Actor
Stdout
Appender
File
Appender
Kafka
Appender
Other
Appenders
Logger
State
Scala
Logger
Control API

23. 47deg.com © Copyright 2016 47 Degrees
Dealing with Concurrency
• The logging actor provides fully generalized sync
• Serializes output
• Handles filters
• But asking the actor on every message can be slow
• @volatile booleans for each log level associated with
logger
• Note the logger itself is global state
• One copy for entire app, no need to pass a parameter
everywhere: ClassLogging and ActorLogging traits.
• Must start before any messages and stop after any
messages (but there is a development workaround)
• Contains log level booleans
23

24. 47deg.com © Copyright 2016 47 Degrees
Source Location
• Could find using exception stack trace at run-time
• Better solution is to use Scala macro and reflection at
compile-time
24

25. Demos
25

26. 47deg.com © Copyright 2016 47 Degrees
Demos
• Simple
• Actor
• Exceptions
• Request Id
• Alternative
• Other (Actor, SLF4J)
• Timing
• Filter
• Appender
26

27. Cross Service Log
Aggregation
27

28. 47deg.com © Copyright 2016 47 Degrees
The Problem
• Applications are being split into ever more micro-
services
• Much harder to answer questions?
• What is the overall control flow for various requests?
• What was the root cause of an error?
• Where are the performance bottlenecks?
• In this talk we look at a prototype system that can be
used to answer these kinds of questions
• Work in progress
28

29. 47deg.com © Copyright 2016 47 Degrees
Related Work
• Google Dapper
• Zipkin
• Akka Tracing (wraps Zipkin)
29

30. 47deg.com © Copyright 2016 47 Degrees
Approach
• Use Scala as much as possible
• Aggregate logs for each independent request across a set of
Scala services
• Provide a basis for both
• Near real-time
• Alerts
• View aggregated logs for any request
• More comprehensive batch analysis
• Failure statistics
• Overall patterns
• Trends over time
30

31. 47deg.com © Copyright 2016 47 Degrees
Overall Architecture
31
Kafka
Spark
Streaming
(Aggregation)
NoSql DB
REST Log
API *
(Merge, Filter)
Services *
* Uses: Persist Service Framework
Persist Service Framework Uses:
Akka HTTP
Persist Json
Persist Logger
Spark Batch
Analytics

32. 47deg.com © Copyright 2016 47 Degrees
Scala REST Service
32
Server
Log
Service
(can be lots of these)
Error
Log
Client
Log
... To Other
Services
Kafka
(only one cluster/topic for all
logs and services)
... From Other
Services

33. 47deg.com © Copyright 2016 47 Degrees
Services
33
Serial
A
Driver
E
D
C
B
Parallel

34. 47deg.com © Copyright 2016 47 Degrees
Aggregation (via Spark Streaming)
• Pass from client to server
• Client name
• Tracking Id
• same for a single request across all micro services
• Span Id
• unique for each call for a given Client-Server pair
• Match Tracking Id and Span Id across
• Client log
• Server log
34

35. 47deg.com © Copyright 2016 47 Degrees
Cross Service Matching
35
Id1
Span1
X
A
Id1
Span1
X
Id1
Span2
A
Id1
Span3
A
A
Id1
Span1
Span2
B
B
Id1
Span2
A
B
Id1
Span3
A
A
Id1
Span1
Span3
B
Match
Match
Match
Match
Service A Service B

36. 47deg.com © Copyright 2016 47 Degrees
Look at Sample Output
• Array of log messages
• Tree of log messages
• Filtered tree with timings
36

37. Questions
37