Companies who use Kafka today struggle with monitoring and managing Kafka clusters. Kafka is a key backbone of IoT streaming analytics applications. The challenge is understanding what is going on overall in the Kafka cluster including performance, issues and message flows. No open source tool caters to the needs of different users that work with Kafka: DevOps/developers, platform team, and security/governance teams. See how the new Hortonworks Streams Messaging Manager enables users to visualize their entire Kafka environment end-to-end and simplifies Kafka operations. In this session learn how SMM visualizes the intricate details of how Apache Kafka functions in real time while simultaneously surfacing every nuance of tuning, optimizing, and measuring input and output. SMM will assist users to quickly understand and operate Kafka while providing the much-needed transparency that sophisticated and experienced users need to avoid all the pitfalls of running a Kafka cluster. Speaker: Andrew Psaltis, Principal Solution Engineer, Hortonworks

1. 1 © Hortonworks Inc. 2011–2018. All rights reserved
Curing the Kafka blindness—
Streams Messaging Manager
Andrew Psaltis

2. 2 © Hortonworks Inc. 2011–2018. All rights reserved
Why Kafka
Source
System
Source
System
Source
System
Source
System
Kafka
Hadoop Security
Systems
Real-Time
Monitoring
Data
Warehouse
Producers
Brokers
Consumers
Kafka decouples data pipelines

3. 3 © Hortonworks Inc. 2011–2018. All rights reserved
Streaming Analytics Reference Architecture
Data Flow Apps
Powered by NiFi
Subcribing
Streaming Analytics
Apps
Kafka is Everywhere - Critical Component of Streaming Architectures
Kafka Producers Kafka Topics Kafka TopicsKafka Consumers & Producers Kafka Consumers
Data Syndication
Services Powered by
Kafka
syndicate-
speed
Kafka Topic
syndicate-
geo
Kafka Topic
syndicate-
transmission
Kafka Topic
syndicate-
temp
Kafka Topic
syndicate-
oil
Kafka Topic
syndicate-
breaks
Kafka Topic
syndicate-
battery
Kafka Topic
syndicate-
start/stop
Kafka Topic
syndicate-
acceleration
Kafka Topic
syndicate-
idle
Kafka Topic
Data Collection
at the Edge
US West Fleet
Truck Sensors C++
Agent
US Central Fleet
Truck Sensors C++
Agent
US East Fleet
Truck Sensors C++
Agent
Analytics App 1
Analytics App 2
Analytics App 5
Analytics App 3
Analytics App 4
gateway-west-
raw-sensors
Kafka Topic
IOT Ingest Gateway
Powered by Kafka
gateway-central-
raw-sensors
Kafka Topic
gateway-east-
raw-sensors
Kafka Topic

4. 4 © Hortonworks Inc. 2011–2018. All rights reserved
Kafka’s Omnipresence Has Led to the Onset of “Kafka
Blindness”
• Who is Affected?
• Platform Operation Teams
• Developers / DevOps Teams
• Security / Governance Teams
• What is “Kafka Blindness”?
• Customers who use Kafka today struggle with monitoring /
“seeing”/troubleshooting what is happening in their clusters
• What are the Symptoms?
• Difficulty seeing who is producing and consuming data
• Difficulty understanding the flow of data from producers -> topics à consumers
• Difficulty troubleshooting/monitoring.

5. 5 © Hortonworks Inc. 2011–2018. All rights reserved
Kafka Personas and Challenges

6. 6 © Hortonworks Inc. 2011–2018. All rights reserved
Distinct Needs of 3 Personas/Teams using Kafka
Concerned with monitoring
the overall health of the
cluster and the infrastructure
it runs on
Concerned with monitoring
the Kafka entities associated
with their apps
Concerned with audit,
compliance, access control &
chain of custody requirements

7. 7 © Hortonworks Inc. 2011–2018. All rights reserved
SMM Platform
Operations Use Cases

8. 8 © Hortonworks Inc. 2011–2018. All rights reserved
SMM for the Platform Ops Team
Are any of my brokers down?
How many total active producers/
consumers is there currently?
Which producers are
generating the most data
right now?
Which producers are generating
the most data right now?
Do I have any offline topic partitions?
Which producers are generating
the most data right now?
Which consumer group us
falling behind the most?
Are there any skewed
partitions for a broker?
What is the throughput in/out
for a given partition on that
broker?
How many total active
producers and consumers
exist now?
Are any of my brokers
running out of disk space?
What hosts are my brokers
located on?
How of the cluster is being used,
how much capacity do I have
available per broker / cluster?
Are any of my brokers running
hot? Which broker has the highest
throughput in and and rates?
Which partitions are
located on each broker?
How many total topics does
my Kafka cluster have?
Are all my replicas in my
topic in-synch?
Which of my topics has
produced/consumers the most
messages over the last N
minutes/hours?

9. 9 © Hortonworks Inc. 2011–2018. All rights reserved
How would you find the most active
producer in the cluster ?
Open Source Monitoring Tools
Grafana / Graphite
Command Line / Script
X
X
ü

10. 10 © Hortonworks Inc. 2011–2018. All rights reserved
Find the Most Active Producer in my Cluster
A Kafka producer called
minfi-eu-i1 is the most
active producer sending
39K messages in the last
30 mins
Click on Messages to
sort on messages sent
by all producers in the
last 30 mins

11. 11 © Hortonworks Inc. 2011–2018. All rights reserved
How would you find the consumer
that has fallen behind the most?
Open Source Monitoring Tools
Grafana / Graphite
Command Line / Script
X
X
ü

12. 12 © Hortonworks Inc. 2011–2018. All rights reserved
Find the consumer that has fallen behind the most
Click on LAG to sort on consumer lag
across all consumers in the last 30 mins
Consumer group named route-micro-
service has significantly more lag (97K)
than any another consumer in the
cluster.

13. 13 © Hortonworks Inc. 2011–2018. All rights reserved
How would you find which broker has
the highest throughput?
Open Source Monitoring Tools
Grafana / Graphite
Command Line / Script
ü
ü
ü

14. 14 © Hortonworks Inc. 2011–2018. All rights reserved
Which broker has the highest throughput?
Step 1
Click on the Brokers tab to
see a broker centric view of
the Dashboard
Step 2
Click on Throughput to sort
on data in across all brokers
For Grafana Metrics
Click on Ambari icon to view Broker
host level metrics

15. 15 © Hortonworks Inc. 2011–2018. All rights reserved
How would you find the partitions on
a given broker and understand the
flow of data from producers to
consumers?
Open Source Monitoring Tools
Grafana / Graphite
Command Line / Script
X
X
X

16. 16 © Hortonworks Inc. 2011–2018. All rights reserved
Understand flow of data flow from Producer to Consumer
Step 1
Click on Panel expand to get more
details on the broker like all partitions
that are stored on the broker
Step 2
Click on the partition to see the producers and
consumers sending/consuming from that partition.

17. 17 © Hortonworks Inc. 2011–2018. All rights reserved
SMM DevOps/App Dev
Use Cases

18. 18 © Hortonworks Inc. 2011–2018. All rights reserved
SMM for the DevOps/AppDev Teams
Find all entities (producer,
consumers, topics) associated with
my app.
What is the retention
rate for my app topics? What is the replication
factor for my app topics?
What brokers holds the
partitions for my app
topics?
Who are all the producers
and consumers connected
to my app topics?
What type of events are in my
application topics? What does the
event look like?
What is the total number of
messages into my topic over
the last N minutes/hours?
Did a consumer rebalance
occur for a given topic?
Are any of my consumers/
consumer-groups that are
over-consuming?
Are there consumers in a
consumer-group for a given
topic slow/falling behind?
How many active consumers
instances are in a given
consumer group?
Are any of my consumers/consumer-
groups that are under-consuming?What topic(s) are the consumer
group consuming messages from?

19. 19 © Hortonworks Inc. 2011–2018. All rights reserved
How would you filter all of the
producers, consumers, and topics that
are related to your application?
Open Source Monitoring Tools
Grafana / Graphite
Command Line / Script
X
X
X

20. 20 © Hortonworks Inc. 2011–2018. All rights reserved
Filtering producers, consumers, and topics
Use the Filter to filter on topics
and select all the IOT gateway
topics

21. 21 © Hortonworks Inc. 2011–2018. All rights reserved
How would you find the partitions on
a given broker and understand the
flow of data from producers to
consumers?
Open Source Monitoring Tools
Grafana / Graphite
Command Line / Script
X
X
X

22. 22 © Hortonworks Inc. 2011–2018. All rights reserved
Are any consumers /consumer-groups under-consuming?
Step 1
Expand topic panel
to see more details
Step 2
Click on the Topic
to see all producers
Analysis
Note that for each
partition there is no
data going out (0B)
and no data going to
any consumer groups.

23. 23 © Hortonworks Inc. 2011–2018. All rights reserved
How does Data Flow between Producers to Topics to Consumers?
Step 1
Expand details of a topic
that has consumers
Step 2
Click on the topic to see
producers and consumers

24. 24 © Hortonworks Inc. 2011–2018. All rights reserved
How would search for and view the
data in a topic of interest?
Open Source Monitoring Tools
Grafana / Graphite
Command Line / Script
X
X
ü

25. 25 © Hortonworks Inc. 2011–2018. All rights reserved
Explore/Search Messages in the Kafka Topic
Click on the explorer icon to search for
events in the Kafka Topic

26. 26 © Hortonworks Inc. 2011–2018. All rights reserved
SMM for the Security and Governance Team
How does data flow across
multiple kafka hops?
When was a topic created?
When was the topic configuration
last modified?
Which producers have
sent data to the topic?
Which consumers have
consumed from a topic?
What is the schema for a given topic?
How has the schema evolved for
a given topic?
When were additional
brokers added to the
topic?Which users/groups/service
accounts have read from a given
topic?
Which users/groups/service
accounts have sent data to a topic?
What are the ACL policies for a
given topic?
Who has edited the ACL
policies of a given topic?
What is the linage of a
kafka topic?

27. 27 © Hortonworks Inc. 2011–2018. All rights reserved
How would you trace the data lineage
across multiple Kafka topics?
Open Source Monitoring Tools
Grafana / Graphite
Command Line / Script
X
X
X

28. 28 © Hortonworks Inc. 2011–2018. All rights reserved
Step 1
Click on Atlas Icon to see
lineage of the the topic
gateway-west-raw-sensors
Analysis
NiFi App consumes from the gateway-
west-raw-sensors topic and publishes
events to downstream Kafka topic
called syndicate-geo-event-avro
The topic has one active consumer which is a
NiFi consumer. Which Kafka topic if any is this
NiFi Flow consumer publishing events to?
Step 2
Search for the Kafka topic
that the the nifi-truck-
sensor-west consumer was
publishing events to
Analysis
We see that that this topic has 4
downstream consumers. We just tracked
a flow across multiple Kafka topics

29. 29 © Hortonworks Inc. 2011–2018. All rights reserved
How would integrate all of the metrics
with an APM, alerting or ticketing
system ?
Open Source Monitoring Tools
Grafana / Graphite
Command Line / Script
X
X
X

30. 30 © Hortonworks Inc. 2011–2018. All rights reserved
SMM REST Admin Server
What is the SMM REST Server?
à SMM UI powered by first class REST
Services via the SMM REST Admin Server
à Monitoring Rest Endpoints can be used to
integrate with APM/Alerting/Ticketing
solutions
à Powered by Apache Knox
à Installed via an Ambari Management Pack
on a target HDP/HDF cluster where Kafka
Service is running
à Supports both HDP & HDF Platforms

31. 31 © Hortonworks Inc. 2011–2018. All rights reserved
HORTONWORKS
DATA PLATFORM (HDP®)
DATA-AT-REST
PRODUCERS CONSUMERS
TOPICS
BROKERS
SMM REST Server
HORTONWORKS
DATAFLOW (HDF)
DATA-IN-MOTION
Optimize your
Kafka Clusters
Troubleshoot
Kafka Streams
Trace end-to-
end Kafka flows
Streams Messaging Manager (SMM)

32. 32 © Hortonworks Inc. 2011–2018. All rights reserved
Questions?

33. 33 © Hortonworks Inc. 2011–2018. All rights reserved
Thank you