Modern log aggregation & search tools provide significant new capabilities for teams building, testing, and running software systems. By treating logging as a core system component, and using techniques such as unique event IDs, transaction tracing, and structured log output, we gain rich insights into application behaviour and health. This talk explains why it is valuable to test aspects of logging and how to do this with modern log aggregation tooling.

1. Why and how to test logging
DevOps Showcase North / Testing Showcase North
Manchester, UK - Unicom
Thursday 11th February 2015
Matthew Skelton
Skelton Thatcher Consulting

2. confession:
I am a big fan of logging

4. use logging as a
channel/vector to
make distributed
systems more testable

5. transforming technology
and teams
high impact expertise
Cloud, Agile, DevOps

8. Why we should test logging
How to test logging effectively

11. exceptional situations
edge cases
metrics
analytics
‘audits’
…
@evanphx

12. logging as an execution trace

13. reduce time-to-detect
increase team engagement
increase configurability
enhance DevOps collaboration
#operability

15. (event sourcing)
(structured logging)
(CQRS)

17. using logging mainly for errors
inconsistent use of logging
“logging slows things down”

18. “logging pollutes my domain model”
“logging is just for Ops people”
log aggregation only in Production

19. logging assumed to be free
($0) to implement
logs not available to Tester or
Developers

25. distinct event IDs
transaction tracing
instrumentation
log aggregation & search

26. cross-component visibility
increased test coverage
programmatic assertions
deep insights

27. stories for testing logging

28. BasketItemAdded
Search: BasketItem

29. Continuous event IDs

30. How many distinct event
types (state transitions) in
your application?

32. represent distinct states

33. enum
Human-readable sets:
unique values, sparse, immutable
C#, Java, Python, node
(Ruby, PHP, …)

34. public enum EventID
{
// Badly-initialised logging data
NotSet = 0,
// An unrecognised event has occurred
UnexpectedError = 10000,
ApplicationStarted = 20000,
ApplicationShutdownNoticeReceived = 20001,
PageGenerationStarted = 30000,
PageGenerationCompleted = 30001,
MessageQueued = 40000,
MessagePeeked = 40001,
BasketItemAdded = 60001,
BasketItemRemoved = 60002,
CreditCardDetailsSubmitted = 70001,
// ...
}

35. Technical
Domain
public enum EventID
{
// Badly-initialised logging data
NotSet = 0,
// An unrecognised event has occurred
UnexpectedError = 10000,
ApplicationStarted = 20000,
ApplicationShutdownNoticeReceived = 20001,
PageGenerationStarted = 30000,
PageGenerationCompleted = 30001,
MessageQueued = 40000,
MessagePeeked = 40001,
BasketItemAdded = 60001,
BasketItemRemoved = 60002,
CreditCardDetailsSubmitted = 70001,
// ...
}

36. BasketItemAdded = 60001

37. BasketItemAdded = 60001
BasketItemRemoved = 60002

38. BasketItemAdded = 60001
BasketItemRemoved = 60002

39. represent distinct states

41. OrderSvc_BasketItemAdded

43. Monolith to microservices:
debugger does not have the full view

44. Even with remote debugger, it’s
boring to attach and detach

46. Storage I/O
Worker Job
Queue
Upload

50. Transaction tracing

51. ‘Unique-ish’ identifier for each request
Passed through downstream layers

52. Unique-ish ID

53. Testing logging

54. use logging as a
channel/vector to
make distributed
systems more testable

55. track a parcel online

57. search by
event ID

58. transaction
trace

59. Web App
Doc store
Audit service
Msg queue

60. Transactional
sub-system
Log aggregation
sub-system

61. An action here… …results in log entry here

62. trace events in a
distributed,
asynchronous system

63. stories for testing logging

64. AuditRecordCreated
Search: AuditRecordCreated

65. AuditRecordCreated
Search: AuditRecordCreated

66. Feature: User trades stocks
Scenario: User requests a sell before close of trading
Given I have 100 shares of MSFT stock
And I have 150 shares of APPL stock
And the time is before close of trading
When I ask to sell 20 shares of MSFT stock
Then I should have 80 shares of MSFT stock
And I should have 150 shares of APPL stock
And a sell order for 20 shares of MSFT stock should have been executed
http://martinfowler.com/bliki/GivenWhenThen.html - Pete Hodgson

67. Given I run a scenario as a Lawyer
And I create a document
[And I wait 5 seconds]
When I search the logs API
Then I should find a recent entry
for “AuditRecordCreated”

69. Recap

70. use logging as a
channel/vector to
make distributed
systems more testable

71. unique-ish event type IDs
transaction tracing
log aggregation & search
Stories: expected search results

72. cross-component visibility
increased test coverage
programmatic assertions
deep insights

73. trace events in a
distributed,
asynchronous system

74. ‘Structured Logging’
TW: “Adopt” (May 2015)
https://www.thoughtworks.com/radar/techniques/structured-logging
http://gregoryszorc.com/
.NET: http://serilog.net/
Java: https://github.com/fluent/fluent-logger-java

75. More
Ditch the Debugger and Use Log
Analysis Instead
Matthew Skelton
https://blog.logentries.com/2015/07/ditch-
the-debugger-and-use-log-analysis-instead/

76. Thank you
http://skeltonthatcher.com/
enquiries@skeltonthatcher.com
@SkeltonThatcher
+44 (0)20 8242 4103
@matthewpskelton