Presentation on Apache Kafka and Microservices. Presented in Melbourne and Perth Australia meetup events.

1. Microservices in a
Streaming World
Modern distributed applications and infrastructure

2. About me
Hans Jespersen
hans@confluent.io
https://www.linkedin.com/in/hansjespersen
https://github.com/hjespers
U of Waterloo – Punch Cards, COBOL, Assembler, RJE, Screen Scraping
AT&T – Unix, Client/Server, Tuxedo & Transactions, Pre-Web Internet
Sun – Solaris, ONC RPC, CORBA
TIBCO – Yahoo!Quotes, Pub/Sub, Multicast, JMS, ESB, SOA, WS-*
Solace – Messaging HW, MQTT ,AMQP, REST
Confluent – Kafka, Event Stream Processing
Personally - active in IoT, open source, MQTT

3. Not going to talk about...
Meaningless, vendor marketing,
definition of Micro Services

4. Let’s parse this out together…

5. Love Martin Fowler’s Work

7. https://youtu.be/_RgUxUTuxH4 – or Google “Kafka Summit Martin Fowler”

8. I am going to talk about...
Patterns in Modern Distributed
Architecture and Application
Infrastructure

9. You’ve probably heard of Microservices.

10. But what are they really?

11. Are they SOA all over again?
SOA Microservices

12. Or are they an evolution?
SOA

13. We used to build monolithic applications.

14. These applications came in a variety of different forms.

15. Companies built lots of them.

16. But the teams in these companies were grouped into business areas. Silos.

17. So the applications would also form into silos along their business lines.

18. This led to a lot of duplication across different applications, in different silos.

19. The duplication made the IT function expensive to run.

20. Then someone had an idea.

21. What if they pulled out the common pieces into reusable “services”?
Service

22. Less code in each silo seemed like a good idea.
Service

23. Then, they realized they could build whole companies
based on these reusable services.

24. This seemed like a very good idea!

25. The trick was to rearrange the people to match the services.

26. So instead of sitting in silos…

27. The teams would match the architecture that they wanted to end up with.

28. SoA
This pattern was termed a
‘Service Oriented Architecture.’

29. But - there was a problem.

30. Refactoring companies turned out to be slow and difficult.
Services

31. So people took the same pattern designed for big companies…

32. …and applied it to break up monolithic applications...

33. …knowing that with this architecture, the companies would be able to grow…

34. …and grow…

35. …and grow.

36. This was termed Microservices:
Change through small,
well-defined services
that are easy to reuse.

37. So silos were broken down…

38. …and monoliths were broken up.

39. Evolving incrementally towards contemporary service estates.

40. But things were not entirely rosy yet.

41. While silos seemed expensive and wasteful, they had one big advantage...

42. Each application was free to handle change independently of
the applications around it.

43. Every application was, in some sense, an island.

44. Microservices are built with HTTP, REST, or some other protocol
made from requests and replies.
Request
Reply

45. This works well when ecosystems are small…
Buying
Widgets

46. But gets harder as they grow more complex and more interconnected.

47. Services are tightly coupled. No islands here!

48. So if one service fails…

49. …or even just runs slowly…
Buy Widgets

50. The fall-out could be much larger.

51. Others end up feeling that pain.
Buy Widgets

52. Yet, at company scale, the majority of processes run in the background anyway.
They are asynchronous to one another.
Online
Billing Inventory Fulfillment Fraud
Offline

53. So it makes sense to DECOUPLE services from one another.
Billing Inventory Fulfillment Fraud
Offline
Online
Decouple

54. Apache Kafka™ helps with this as it provides a data backbone for your services.
Billing Inventory Fulfillment Finance Fraud
HTTP etc
Offline
Online

55. This connects services together.

56. It also connects their DATA together.
All Your Data

57. The three tenets of messaging… embedded into a layer of permanence.
Decoupling Notification Data Transfer
Permanence

58. Messaging that remembers.

59. So every service gets the data it needs.
All Your Data

60. Unlike typical service frameworks, it also DECOUPLES services from one another,
so they can evolve independently.

61. This makes it easier to move away from legacy architectures,
to evolve away from the past…
All Your Data
LEGACY

62. …towards a better-factored future, whatever that may look like.
All Your Data
LEGACYNew Services
Analytics

63. So wherever your business ends up…
Cloud
Another Device
Another
Geography

64. Apache Kafka provides the Service Backbone built to handle today’s
data-centric world...
Big Data Ready

65. In a way that can adapt to your company's future, wherever you might take it.

66. Services built on the POWER and IMMEDIACY of an Event Streaming Platform.
Event Streaming Platform

68. So what is this Kafka thing you speak of so highly?

69. How do you see Kafka?

70. Traditional Messaging Functionality
Decoupling of Producers and Consumers

71. Message Exchange Patterns (MEP)
Topic = Publish/Subscribe - N of N delivery Queues = Point-to-Point - 1 of N delivery

72. Message Exchange Patterns (MEP)
Request/Reply
Content-Based Router

73. Message Delivery Semantics
At-Most-Once
“Best effort”
“Reliable”
QoS 0
At-Least-Once
“Guaranteed”
“Certified”
QoS 1
Exactly Once
“Once-and-Only-Once”
“Transactional”
QoS 2

74. Wide Spectrum of Messaging Offerings
Ultra- low Latency (often no broker in the middle)
High Volume (Persistent or Non-Persistent)
Highly Available (Clustered and Fault Tolerant)
Embedded Messaging (inside apps)
Cross Datacenter / Organizational / B2B
Enterprise Message Bus
Messaging-as-a-Service
Web / IoT Messaging
Instant Messaging

75. “publish-subscribe messaging rethought as a distributed commit log”

76. Kafka is a Mashup
Mashup of some well proven concepts into something even greater and easier to use:
EAI + ETL
Messaging Middleware + Big Data
Batch + Real-time
Data Movement + Data Processing
Log Data Streams + Structured Database Tables

77. + Distributed clustered storage
Kafka is a blend of messaging, stream processing, ETL and
modern database designs built around a distributed log
+ Streaming platform
Pub/Sub
Messaging
ETL
Connectors
Spark
Flink
Beam
IBM MQ
TIBCO
RabbitMQ
Mulesoft
Talend
Informatica
Kafka is much more than messaging
+ Exactly Once
+ Designed for the Cloud
+ Inter DC replication
+ Schema evolution
Stream
Processing
Confluent Confidential

78. What’s different about Kafka? Topics are also Queues
Consumers can share one copy of the data
• Independent consumers share the same log
• Inter-dependent consumers share the same log
• No need for Topic/Queue bridging or multiple
copies of the data
Message processing is greatly simplified
- There is no “head’ of the queue
- Writes are sequential, distributed, and
parallel

79. What’s different about Kafka? Messages are not deleted when
consumed
Messages in the commit logs are persistent and immutable
Slow Consumers are (very) decoupled from Fast Producers
Batch and real-time are unified
Message Replay, Replication, and Auditing are built-in (for free)
All production messaging deployment need some form of these
Message Retention is not a waste of disk space
You need to size for offline/disconnected consumers anyway
Distributed State can always be recreated from a common commit log
Makes distributed HA apps much easier to build

80. What’s different about Kafka? Topic Partitions and Keyed Messages
- Topics/Queues are not the smallest unit of
scalability
- Topics partitions are distributed across
brokers for parallel in-order consumption
- This is very different from a cluster of
traditional message brokers
- [graphic of topic partitions with parallel
Producers, Brokers, and Consumers]
- Sometime you can just use more keys
instead of more topics
- Eg. don’t create a new topic for every user,
or IoT device, create unique keys
- This is proven to scale to many millions of
connected users, cars and IoT devices
- [graphic to show Keyed messages get
distributed across topic partitions]

81. From an event stream / transaction log we can derive all of the following
database centric features:
- Replication
- Secondary Indexing
- Caching
- Materialized Views
What’s different about Kafka? Duality of streams and databases
Duality of a message streams and database tables is a key design point
=

82. (Good) Microservices avoid shared mutable state
Shared, mutable state

83. Old World: REST Based Microservices Interconnect
GUI
UI Service Order
s
Returns
Pay Fulfilment Stock
Confluent Confidential
Each Microservice has to maintain their own stateful
nature by using their own databases
1. Difficult to Enforce Same REST API standards
across many languages and micro-services.
2. Rest APIs Inherently Slow: Limited to Thousands
calls/sec.
3. Inter Service Dependencies are Messy.
4. Each Service Needs to Maintain State.
5. Difficult to enforce consistent security standards.
6. Logging is distributed between services.
7. Version compatibility between services is difficult.

84. Streaming Microservices with Kafka
GUI
UI
Service
Orders
Service
Returns
Service
Fulfilment
Service
Payment
Service
Stock
Service
Confluent Confidential
Database Sources Now Centralized on the Kafka Bus for all microservices
1. Service inter-communication standard enforce by Kafka Schema Registry.
2. Millions of messages per second on cheap hardware.
3. No Inter-Service Dependency: just depend on Kafka.
4. Each service can be stateless: Kafka maintains state.
5. Security can be enforced by ACLs from Kafka.
6. Logs can be aggregated into Kafka.
7. Version compatibility can be enforced by Scheme Registry.
8. Kafka is inherently HA, horizontal scalable: still no central point of failure.

85. What’s different about Kafka? Ecosystem and Adoption
The Kafka ecosystem is flourishing and developer adoption continues to grow
• Confluent Platform additions (REST Proxy, Schema Registry, KSQL etc.)
• Third Party Connectors ( Confluent Hub)
• Open Source contributions from individuals, corporations, vendors, consulting organizations
• Inside and outside of Big Data/Stream Processing

86. Adoption of Event Streaming
60%Fortune 100 Companies
Using Apache Kafka

87. Event Streaming at the Heart of the Enterprise

90. Embrace a Distributed World