Eventing and streaming open a world of compelling new possibilities to our software and platform designs. They can reduce time to decision and action while lowering total platform cost. But they are not a panacea. Understanding the edges and limits of these architectures can help you avoid painful missteps. This talk will focus on event driven and streaming architectures and how Apache Kafka can help you implement these. It will also discuss key tradeoffs you will face along the way from partitioning schemes to the impact of availability vs. consistency (CAP Theorem). Finally we’ll discuss some challenges of scale for patterns like Event Sourcing and how you can use other tools and even features of Kafka to work around them. This talk assumes a basic understanding of Kafka and distributed computing, but will include brief refresher sections.

1. Hard Truths About
Eventing and Streaming
@DanRosanova
Group Principal Program Manager
Microsoft Azure Messaging

2. Disclaimer
All photos are owned by the speaker and free for non-commercial use

3. A brief history of messaging
• Old school messaging (System/360 QTAM and TCAM)
• IBM MQ (yes there was Teknekron too)
• Rabbit MQ
• Service Bus
• ActiveMQ™
• ZeroMQ
• Apache Kafka®
• NATS

4. What exactly
is a messaging
queue

5. A simple queue
Sender sends message to queue
Queue ACKs receipt
Receiver connects to queue & retrieves message
Receiver ACKs complete (or other action)

6. What queues are good
at and why

7. The queue is the arbiter of truth – which
simplifies many other aspects
EACH READER CAN JUST SAY ‘GIVE
ME THE NEXT’
MESSAGES ARE ACKED /
COMPLETED INDIVIDUALLY
THE QUEUE IS A BUFFER TO
IMPROVE SCALE AND
PERFORMANCE

8. Messaging and Queues are
about applications more than
they are about data

9. Competing Consumer: The Server-Side Cursor

10. The message is the
Unit of Work

11. Public Subscribe: Topics & Subscriptions
Sender only knows about Topic
Receivers only know about Subscriptions
Filters perform routing on Subscriptions

12. Publish
Subscriber
opens
interesting
possibilities
FAN OUT MULTI-CAST
WIRE TAP FORWARDING WILL
ALLOW MULTI-LAYER
TOPOLOGIES (WARNING)

13. But there are some
things queues aren’t so
good at

14. Replay

15. Strict ordering

16. Strict ordering

17. Strict ordering

18. Eventually competing
consumer models
break down

19. Competing Consumer: not all competition is healthy

20. When a wall is too high to go
over and too thick to go
through it is best to go around

21. Enter the Partitioned
Consumer model

22. How is a partitioned consumer different than
a queue?
Data
Apache Kafka® implements a partitioned consumer model

23. There’s
something
else that
resembles
this

24. There’s
something
else that
resembles
this
RECORDS A STREAM RECODING MOVES
FORWARD ONLY
YOU CAN PLAY THE
TAPE OVER AND OVER
AGAIN
A CASSETTE TAPE
ACTUALLY HAS LEFT
AND RIGHT CHANNELS
WHEN YOU PRESS
RECORD, THEY BOTH
RECORD
BUT THE DATA ON
EACH CHANNEL IS
DIFFERENT
IN KAFKA THESE
CHANNELS ARE
CALLED PARTITIONS

25. A bit more on
the partition
concept
Partition is essentially
append only
Reads are performed
using a client side curor
Reads are
nondestructive

26. In a stream the partition is the
Unit of Work
Streams are processed differently from batch data – normal functions cannot
operate on streams as a whole, as they have potentially unlimited data, and
formally, streams are codata (potentially unlimited), not data (which is finite).

27. Ultimate
example for
streams
By Danielpr85 based on Graphviz source of TuukkaH - Own work, Public Domain,
https://commons.wikimedia.org/w/index.php?curid=687268

28. What streams
like Kafka are
very good at
Scale
Low cost
Replay
Order

29. Why partitioned consumer scales so well

30. Why partitioned consumer scales so well
Maximum Degree of
Parallelism

31. Low cost
• There are no expensive indexes to maintain
• Because each partition is independent there is
no cross broker coordination necessary (other
than optional replication)
• Client-side cursor avoids the overhead of
traditional message brokers
• Data replication and ACK level is a choice of the
sender

32. Replay

33. This may lead you to believe you have found Zen

34. But there are clouds on the horizon

35. Fan out and routing
• Partitioned streams (like Kafka) don’t offer
server-side filtering
• Every reader must read all the data
• As more readers want the data a network
imbalance develops
• Parse.ly Kafkapocalypse
10MBps
10MBps
10MBps
10MBps
N MBps

36. Streams are not queues
• The Unit of Work is not an individual message
• This means processing individual messages
gets complicated
• Cursor management becomes a big challenge
• There is no inherent dead letter capability
• People start adding these ‘features’ in and end
up recreating a queue

37. CAP Theorem
In theoretical computer science the CAP theorem states that
it is impossible for a distributed computer system to
simultaneously provide all three of the following guarantees:
Consistency, Availability, Partition tolerance

38. What does CAP mean for streams?
Consistency: Data should
produce the same results
when read multiple times –
i.e. it should be stable and
durable
Availability: The place data is
written to should always be
available to write to
Partition tolerance: the
ability to continue
functioning when one part of
the system becomes
separated from another

39. Or put another way
when a network partition happens, which over
time is inevitable, then you must make a
choice...

40. This is your last chance.
After this, there is no turning back...
Consistency

41. You must decide which of these two is most
significant
Consistency Availabilty

42. Two stats to
remember
99.999% availability still
means 10 of every 1 million
fails
TCP/IP makes it almost
impossible to detect a dead
socket in less than 21
seconds

43. If consistency is so hard why
bother doing it?

44. Locality of data: moving average processor
• In financial applications a
simple moving average
(SMA) is the unweighted
mean of the previous n
data
• For prices PM, PM-1,…
PM-(n-1)
https://en.wikipedia.org/wiki/Moving_average

45. This is perhaps easier in code
simpleMA(float[] prices, int period)
float result;
for(int i=0;i<period;i++)
result += prices[prices.length – i];
return result/period

46. What would
you do with
this?

47. Value of
consistency:
your bank
account
Beginning Balance of 100.00
(5.00)
12.00
4.00
(22.00)
Ending Balance: 89.00
https://en.wikipedia.org/wiki/Double-entry_bookkeeping_system

48. Partitioning schemes
Not all keys are created
equal
You need to be careful to
avoid hot keys
It’s not always something
you can avoid

49. To key or not to key…
that is the question

50. Adding partitions
You’ve identified a hot
partition
You add more partitions
to handle the scale
The result is a data split
Partition 1
Partition 2
1 2 3 4
5 6 7

51. Sharing data across partitions will slow you
down
Using a partitioned consumer model breaks if the end consumer needs
to join data across partitions
It also reintroduces availability issues

52. Pause

53. Failure to plan is planning to
fail

54. Strategies for dealing with failures in
messaging and streaming
Stop Drop
Retry Deadletter

55. Stop
• Simply stop reading – or writing the stream
• Wait until someone elsewhere has fixed the
problem and then resume
• Appropriate for some scenarios, but not all
• Probably a good idea to include a notification

56. Drop
• If the messages aren’t that important, just drop
them
• Up to a certain point they may not matter
• This is a good strategy for non-mission critical
streams
• But not so good for scenarios requiring strong
consistency guarantees
• Definitely a good idea to include a notification

57. Retry
• Try again and see if it works
• Perhaps the error is transient
• Be aware of impact on downstream systems -
idempotence

58. Deadletter
• Put the data somewhere off your hot path so
that you can go back and handle it later
• Does not interrupt your flow
• Works for poisoned messages

59. Combining strategies
• Often no one strategy will exactly match your
needs
• You can combine these to achieve the policy
that is right for you
• E.G. Retry three times, then deadletter

60. Another
Pause

61. What are event driven architectures
• Events are notifications that something
happened
• This is different than traditional messages,
which are the thing (the command)
• Event Driven Architectures are reactive in
nature
• State is derived from an event log or stream

62. Event Sourcing
• Add head
• Add body
• Add left arm
• Add right arm
• Add left leg
• Add right leg

63. Event Sourcing
• Add head
• Add body
• Add left arm
• Add right arm
• Add left leg
• Add right leg

64. Capabilities we’ve gain from Event Sourcing
• Complete rebuild
• Temporal query
• Event replay

65. What cool things can you do now?
• Add head
• Add body
• Add left arm
• Add right arm
• Add left leg
• Add right leg

66. You can have your cake and eat it too!

67. This sounds interesting…

68. Obvious shortcomings of Event Sourcing and
how to overcome them
TIME TO PROCESS THE LOG:
CHECKPOINTING ON A REGULAR BASIS
HOW TO QUERY THE STATE: BUILDING
A MATERIALIZED VIEW

69. Event Sourcing leads to divergent models for
read and write
This is often addressed with Command Query Responsibility Separation (CQRS)
Despite these benefits, you should be very cautious about using CQRS. Many
information systems fit well with the notion of an information base that is updated
in the same way that it's read, adding CQRS to such a system can add significant
complexity. I've certainly seen cases where it's made a significant drag on
productivity, adding an unwarranted amount of risk to the project, even in the
hands of a capable team.
-Martin Fowler

70. KStreams can help you do Event Sourcing
BASICALLY A WAY TO DO EVENT
SOURCING WITHOUT BEING AN
ARCHITECTURAL ASTRONAUT
PROVIDES MATERIALIZED VIEW
(USES ROCKSDB INTERNALLY TO
HOLD THE TABLE)
EACH APPLICATION CAN NOW
HAVE ITS OWN VIEW OF THE
STREAM

71. One more warning
about stream
processing

72. Sometimes consistency
is overrated

73. Sometimes good enough is… good enough
It can be more useful to have an answer right now than to have a
perfect answer at some point in the future (think Napoleon at
Waterloo)

74. If your application
requires perfect data
then it cannot be real-
time and distributed

75. THEOREM

76. If anyone tells you that
you can have it all in a
distributed system

77. In closing
Pick the right tool for
the job
You may need
multiple tools
Be realistic about
your expectations
Experiment and learn
- continuously
Share your learnings
in contributions,
blogs, etc.
Be an active member
of the Apache Kafka
community!

78. Thank You!