Viktor Gamov, Confluent, Developer Advocate + Jason "Jay" Smith, Google Cloud, Customer Engineer When it comes time to choose a distributed messaging system, the answer is increasingly - Apache Kafka®. What if you need more than publish and subscribe messaging, but also long-term storage (spoiler: Core Kafka), a flexible integration framework (another spoiler: Kafka Connect), and a means of deploying real-time stream processing applications (another one: Kafka Streams) at scale? The answer is still Apache Kafka - a next-generation event streaming platform. This session introduces the foundations of Apache Kafka® and walks through some code examples. Jay and Viktor will demonstrate how to begin using Kafka on Google Cloud with Confluent Operator on GKE and Serverless Kafka via Confluent Cloud. We will also demonstrate streaming applications using Google Cloud Run and Knative Eventing with Kafka. https://www.meetup.com/KafkaBayArea/events/270303178/

1. Serverless Eventings
with Apache® Kafka
and Cloud Run

2. Viktor Gamov
Jason (Jay) Smith

3. By 2025, more than
a quarter of data
created in the global
datasphere will be real
time in nature.

4. What’s driving
the growth in
real time data?
Digital communications
Social media posts, photos, email, collaboration
platforms, chat boards, B2B/B2C outreach
Digital devices/connected industry
Next-gen manufacturing, industrial sensors,
smartphones, wearables, cars/navigation
Digital commerce
Online interactions, purchase histories,
clickstreams, advertising, inventories, ledgers
Digital media consumption
Newspapers, magazines, videos,
games, television, movies

5. I want to create a data driven organization.

7. I want to create a data driven organization.
event

8. What are the events that go into a transaction?
Need
Recognition
Information
Search
Evaluation of
alternatives
Purchase
Decision
Post-purchase
behavior
Views digital ad
Talks to friend
Views influencers
with product
Views peer with
product
Online category
Online views of
substitutes/
alternatives
Visits product pages
Social community
outreach
Views price across
websites
Visits consumer
reports
Visits product pages
Visits online forums
Best price determined
Quality/cost trade
offs made
Vendor / distributor
selected
Social posts
Written reviews
Customers service
calls
Usage Patterns

9. Event Streaming Enables New Outcomes
• Call for driver availability
• No knowledge of driver arrival
• No data on feature usage
• Real-time driver-rider match
• Real-time ETA
• Real-time sensor diagnostics
• Nightly updated account balance
• Batch fraud checks
• Batch regulatory reporting
• Real-time account updates
• Real-time credit card fraud alerts
• Real-time regulatory reporting
• Post-order “out of stock” emails
• No upsell through
personalization
• Batch point-of sale reports
• Real-time inventory
• Real-time recommendations
• Real-time sales reporting
Without Event Streaming With Event Streaming
Banking
Auto/
Transport
Retail

10. Kafka Stores Events in a Log
An immutable sequence of Events
Older
Messages
Append New
Messages
Time

11. The Stream is spread over many machines
An immutable sequence of Events

12. Operations: It’s hard

13. It doesn’t have to be that way...
Confluent Operator It can be hard to deploy a complete event streaming
platform.
And even harder to secure and evolve the platform
over time.
Confluent wants to make it easy.
Confluent Operator takes our operational knowledge
and leverages Kubernetes to automate it all for you.
Every cluster in Confluent Cloud is powered by the
Confluent Operator.

14. It doesn’t have to be that way...
● Kubernetes* scales workers elastically
● Confluent Operator scales brokers elastically
● Self-Balancing rebalances data automatically
● Tiered Storage makes it much faster
$ git diff -- values.yaml
...
kafka:
- brokers: 5
+ brokers: 10
...
$ helm upgrade kafka
confluent-operator
--set kafka.enabled=true
-f values.yml

15. Kubernetes isn’t the endgame

16. Do developers want to use Kubernetes directly?
Have to do
Write code
Build docker image
Upload image to registry
Deploy service
Expose to the internet
Set up monitoring
Set up autoscaling
Want to do
Write code

17. Serverless usage models
developers
perspective
No
Servers
Pay only
for usage
Event-driven
Portable

18. 18
Knative
Building blocks for serverless
workloads on Kubernetes

19. 19
What Knative is
● An open source project
● Set of building blocks to construct your own FaaS/PaaS
○ abstracts common tasks through custom Kubernetes API objects
● An abstraction on top of Kubernetes
○ It's still Kubernetes: Runs containers at the end of the day.

20. 20
What Knative is not
● It's not a Google product.
● It's not FaaS.

21. 21
Knative Serving
Benefits
● Seamlessly scale up and down
● Built-in traffic splitting between
revisions
● Integrates networking and
service mesh automatically
● Easy to reason about object
model
Pluggable
● Connect to your own logging
and monitoring platform, or
use the built-in system
● Auto-scaler can be tuned or
swapped out for custom code

22. 22
Knative eventing
Benefits
● Declaratively bind between event producers and deployed
services
● Scales from just few events to live streams
● Custom event pipelines to connect with your own existing
systems

23. 23
Knative Eventing

24. 24
Knative Eventing

25. 25
Cloud Run
Google’s Managed Version of
Knative Serving

26. 26

27. 27
DEMO

28. 28
Knative Eventing

29. Anthos is our
platform for
application
modernization
Managed Operations
Service Management
Marketplace
Other CloudsOn-premGoogle Cloud

30. Anthos gives
you freedom to
modernize without
being locked in
Developer
Knative Cloud Run
Networking
Istio Anthos Service Mesh
Infrastructure
Kubernetes Anthos GKE

31. Anthos
lets you…

32. Confluent lets you deploy on any k8s Platform
including Anthos
Kubernetes Engine
Build-your-own
Kubernetes
Enterprise
Distributions
Cloud
Services

33. Events streams connect different environments
together
Private Cloud Hybrid Cloud Public Cloud /
Multi-Cloud
• Deploy on premises with
Confluent Platform
• Deploy on Kubernetes
with Operator
• Deploy a consistent platform
across on-prem and cloud
• Build a persistent bridge
between datacenter and
cloud with Replicator
• Run self-managed with
Confluent Platform
• Leverage a fully managed
service with Confluent
Cloud

34. Two Ways to Deploy
Confluent
Self-Managed Software Fully-Managed Service
Confluent Platform
The Enterprise Distribution
of Apache Kafka
Confluent Cloud
Apache Kafka Re-engineered
for the Cloud
VM

35. Viktor Gamov
Jason (Jay) Smith

36. Thank You

37. ORIGIN
AL
SLIDES

38. 38
Serverless Eventings
with Kafka and Cloud
Run

39. 39
Jason (Jay) Smith
@thejaysmith
Viktor Gamov
@gamussa
Hosts

40. By 2025, more than a
quarter of data created in
the global datasphere will
be real time in nature.
*IDC

41. What’s driving
the growth in
real time data?
Digital communications
Social media posts, photos, email, collaboration
platforms, chat boards, B2B/B2C outreach
Digital devices/connected industry
Next-gen manufacturing, industrial sensors,
smartphones, wearables, cars/navigation
Digital commerce
Online interactions, purchase histories,
clickstreams, advertising, inventories, ledgers
Digital media consumption
Newspapers, magazines, videos,
games, television, movies

42. I want to create a data driven organization.
event

43. What are the events that go into a transaction?
Need
Recognition
Information
Search
Evaluation of
alternatives
Purchase
Decision
Post-purchase
behavior
Views digital ad
Talks to friend
Views influencers
with product
Views peer with
product
Online category
Online views of
substitutes/alternatives
Visits product pages
Social community
outreach
Views price across
websites
Visits consumer reports
Visits product pages
Visits online forums
Best price determined
Quality/cost trade
offs made
Vendor / distributor
selected
Social posts
Written reviews
Customers service calls
Usage Patterns

44. 44
Kafka Represents Events as a Stream
An immutable sequence of Events
44
Older
Messages
Append New
Messages

45. 45
The Stream is spread over many machines
45

46. 4646
Auto / Transport
Without Event Streaming With Event Streaming
Event Streaming Enables New Outcomes
Call for driver availability
No knowledge of driver arrival
No data on feature usage
Real-time driver-rider match
Real-time ETA
Real-time sensor diagnostics
Banking Nightly updated account balance
Batch fraud checks
Batch regulatory reporting
Real-time account updates
Real-time credit card fraud alerts
Real-time regulatory reporting
Retail Post-order “out of stock” emails
No upsell through personalization
Batch point-of sale reports
Real-time inventory
Real-time recommendations
Real-time sales reporting

47. 47
Serverless usage models
Portable
No servers Pay only for usage
Event-driven
developer perspective

48. 48
I want to create a data driven organization.
event

49. 49
Knative serving
Benefits
● Seamlessly scale up and down
● Built-in traffic splitting between
revisions
● Integrates networking and
service mesh automatically
● Easy to reason about object
model
Pluggable
● Connect to your own logging
and monitoring platform, or
use the built-in system
● Auto-scaler can be tuned or
swapped out for custom code

50. Anthos is our
platform for
application
modernization
Google Cloud On-prem Other Clouds

51. Anthos gives you
freedom to
modernize without
being locked in
Infrastructure Operator
Kubernetes Anthos GKE
Service Operator / SRE
Istio Anthos Service Mesh
Developer
Knative Cloud Run

52. 5252
Anthos lets you….

53. 5353
Confluent lets you deploy on any k8s Platform
including Anthos
Build-your-own
Kubernetes
Enterprise
Distributions
Cloud
Services
Kubernetes Engine

54. 5454
Event streams connect different environments
together
Private Cloud
● Deploy on premises with Confluent
Platform
● Deploy on Kubernetes with Operator
Public Cloud / Multi- Cloud
● Run self-managed with Confluent
Platform
● Leverage a fully managed service with
Confluent Cloud
Hybrid Cloud
● Deploy a consistent platform across
on-prem and cloud
● Build a persistent bridge between
datacenter and cloud with Replicator

55. 5555
Two Ways to Deploy Confluent
Confluent Platform
The Enterprise Distribution of
Apache Kafka
Confluent Cloud
Apache Kafka Re-engineered
for the Cloud
Self-Managed Software Fully-Managed Service
VM
Deploy on any platform, on-prem or cloud Available on GCP Marketplace