When you're starting out, staying nimble is paramount. This talk will help you push back infrastructure decisions and keep your options open by storing machine-readable JSON logs, not relational data, and processing them as an event stream to construct your data model on the fly.

1. Staying Lean With
Application Logs
@SamirTalwar and @sleepyfox
2016-09-22T11:10:00+0200
Essay form at
https://talks.samirtalwar.com/

2. Reactive,
Event-Sourced, and Lean
(also with logs)
@SamirTalwar and @sleepyfox
2016-09-22T11:10:00+0200
Essay form at
https://talks.samirtalwar.com/

3. We're making a
game for Xbox Two.

5. Trials of Anubis
One of the most popular game modes.
You have two teams, Alpha and Bravo, each of three players.
Each team is trying to defeat the other one.
If you take another player down, they don’t revive in the
same round.
Players can bring their teammates back up, but it’s time-
consuming.
If all the players on the team are taken down, that team
loses.
Best out of three rounds wins the match.

6. Let's get down to
business.
I'm here today to talk about
infrastructure, not graphics.

7. Phase One

9. Phase One
This is a monolith.
It's easy to reason about.
But it doesn't scale past a few games.

10. Phase Two

12. Phase Two
Deploy many monoliths.
It's still easy to reason about.
The components are unbalanced.
They still have to share data.

13. Phase Three

15. Phase Three
We've split into services.
Because microservices are a thing now.
Communication via the database went
well…
for about 20 seconds, before we hit
consistency issues.
We're bottlenecked on the database.

16. Phase Four

18. Phase Four
Services manage their own data storage.
We do communication over HTTP, because
that's apparently lightweight now.

20. Phase Four
Four out of six service components are
just managing HTTP.
Every service needs the same kind of
components.
None of the HTTP stuff makes us money.
It’s just waste.

21. We can do better.
Let's make something that works with our
architecture instead.

22. Event Sourcing
What if, rather than consuming from
other services, each service published
every single event?
• no HTTP servers
• no HTTP logs; just publish to a
store
• still need the clients, but they're
simpler

23. Event Sourcing
Sounds like a solution.
Just one problem.
We'd have to rewrite everything.

24. Event Sourcing
Wait. We already publish events.
To STDOUT. 😮
We log everything already.
Push them to Fluentd or Logstash.

25. A log-powered
event bus.
It'll never work.
Let's do it.

26. impressive
transition
goes here

27. What's in a log?
Machine-readable first.
Human-readable second.
If you want to read your logs,
process them first.
Kibana will help a lot.

28. What's in a log?
We don't care about:
• filtering by severity,
• log rotation,
• multiple formats,
• or even logging to a file.
Those are ops concerns, not dev.

29. What's in a log?
Every event has a type.
The structure of the event is
determined by this type.
That structure is a public contract…
Just like a Java class.
Versioning is probably required.

30. ygt/microservice-logging
const Logger = require('./logger')
const log = new Logger({
now: Date.now,
output: console,
events: {
startup: 'startup',
httpRequest: 'HTTP request',
database: 'database'
}
}).with({service: 'my super service'})

31. ygt/microservice-logging
log.startup.info({
message: 'Ready to go.',
port: 8080
})
// later on
const requestLog = log.with({
request_id: '126bb6fa-28a2-470f-b013-eefbf9182b2d'})
requestLog.database.error({
message: 'Connection failed.'})
requestLog.httpRequest.info({
request: {method: 'GET'},
response: {status: 500}})

32. ygt/microservice-logging
{"timestamp":1474443152917,
"event_type":"startup",
"severity":"INFO",
"service":"my super service",
"message":"Ready to go.",
"port":8080}
{"timestamp":1474443152920,
"event_type":"database",
"severity":"ERROR",
"service":"my super service",
"request_id":"126bb6fa-28a2-470f-b013-eefbf9182b2d",
"message":"Connection failed."}
{"timestamp":1474443152921,
"event_type":"HTTP request",
"severity":"INFO",
"service":"my super service",
"request_id":"126bb6fa-28a2-470f-b013-eefbf9182b2d",
"request":{"method":"GET"},
"response":{"status":500}}

33. ygt/microservice-logging
Check it out.
https://github.com/ygt/microservice-logging
npm install microservice-logging

34. So.
We have a log of all events in the system.
There’s real-time reactive behaviour across services.
The stateless services are completely scalable…
… and there’s patterns for scaling the stateful ones.
Adding new services that derive information from the
data requires no modifications to the existing ones.
We have a model for handling catastrophic failure.
Reporting and analytics are practically free now.

35. So.
And my favourite:
we don't have a database at all.

36. Scaling out
Load-balance, with sticky sessions based on IDs if
services keep state in memory.
Avoid in-memory state; outsource to Memcached or Redis.
On failure, respawn and replay.
Push events from Fluentd/Logstash to Kafka or Kinesis.
Flag non-JSON logs for manual review.
Store events in Elasticsearch or Cassandra.
When you need a real database, keep your events, and
populate it from there (write model and read model).

37. Log everything
Who's got questions?

38. Thank you.
@SamirTalwar
samir@noodlesandwich.com
https://talks.samirtalwar.com/
https://github.com/SamirTalwar
/logs-as-the-event-source