Real-time streaming apps are increasingly driving business growth in verticals such as retail, IoT, telecommunications, financial services and SaaS. Ensuring the scalability and reliability of the data pipelines powering such apps in the face of ever-growing data volumes has traditionally required significant development and operations expertise. The combination of Kafka as the streaming platform, KSOL as the stream processing layer and YugaByte DB as the ultra-resilient, Cassandra-compatible database make these tasks easier than ever before. In this talk, we will review the key architectural patterns involved in the context of a sample IoT Fleet Management application built on Confluent Kafka, KSQL, YugaByte DB and Spring Boot.

1. 1© 2018 All rights reserved.
Kafka, KSQL & YugaByte DB: Building Scalable & Reliable
Streaming Apps The Easy Way
Bogdan Matican & Bharat Baddepudi
November 2018

2. 2© 2018 All rights reserved.
Agenda
o About Us
o Why Reliability & Scalability in Streaming Apps is Hard?
o Solution – (Confluent) Kafka + KSQL + YugaByte DB
o Why YugaByte DB for Streaming Apps?
o Real-World Example & Demo
o Q & A

3. 3© 2018 All rights reserved.
About Us
Bogdan Matican,
Founding Engineer, YugaByte
• Facebook
• HBase
• Blobstore
• Stream Processing
• UCar
• Monitoring & Data Analytics
• YugaByte
• Core DB
• Middleware & Cloud Integrations
• Tooling & Automation
Bharat Baddepudi,
Founding Engineer, YugaByte
• Oracle
• Core DB - resource & memory mgmt.
• Exadata Flash Cache
• MapR
• HBase compatible MapR DB
• Hadoop-scale Document DB
• YugaByte
• Core DB
• Enterprise-grade Manageability

4. 4© 2018 All rights reserved.
RELIABILITY & SCALABILITY IN
STREAMING APPS

5. 5© 2018 All rights reserved.
Top 3 Challenges in Building Streaming Apps
1. Ingestion of a data firehose fast and without loss
– Reliable, high performance streaming platform
2. Aggregations to power real-time insights
– Ad-hoc analytics over a moving time window
3. Database for streaming apps
– Storage, management & serving of ever-growing data

6. 6© 2018 All rights reserved.
Closer Look at Database Requirements
1. Strong consistency
2. Elastic scaling
3. High fault tolerance
4. Superior performance
5. High data density
6. Automatic data expiry (TTL)

7. 7© 2018 All rights reserved.
Streaming Events to Real-Time Insights … The Easy Way
Devices
Sensors
Apps
Streaming Platform
Streaming Events
Event
Processing
Read the derived tables
Alerts/Notifications
Real-Time
DashboardsReliable, Elastic,
Cassandra
Compatible DB
Store raw events
Write the derived table
Real-Time
Analytics
Data
ModelingRead the stream

8. 8© 2018 All rights reserved.
YUGABYTE DB
FOR
STREAMING APPS

9. 9© 2018 All rights reserved.
Cloud NativeNoSQL + SQL
A transactional, planet-scale database
for building high-performance cloud services.

10. 10© 2018 All rights reserved.
TRANSACTIONAL PLANET-SCALEHIGH PERFORMANCE
Single Shard & Distributed ACID
Transactions
Document-Based, Strongly Consistent
Low Latency, Tunable Reads
High Throughput
CLOUD NATIVE OPEN SOURCE
Apache 2.0
Popular APIs Extended
Apache Cassandra, Redis and PostgreSQL (BETA)
Built For The Container Era
Self-Healing, Fault-Tolerant
Auto Sharding & Rebalancing
Global Data Distribution
Design Principles

11. 11© 2018 All rights reserved.
Under The Hood - 3 Node Cluster
DocDB Storage Engine
Purpose-built for ever-growing data, extended from RocksDB
yb-master1
yb-master3
yb-master2
YB-Master
Manage shard metadata &
coordinate cluster-wide ops
node1
node3
node2
Distributed Transaction Manager
Tracks ACID txns across multi-row ops, incl. clock skew mgmt.
Raft Consensus Replication
Highly resilient, used for both data replication & leader election
tablet 1’
tablet 1’
yb-tserver1 yb-tserver2
yb-tserver3
tablet 1’
tablet2-leader
tablet3-leader
tablet1-leaderYB-TServer
Stores/serves data in/from
tablets (shards)
tablet1-follower
tablet1-follower
tablet3-follower
tablet2-follower
tablet3-follower
tablet2-follower
…
…
…

12. 12© 2018 All rights reserved.
Strong Consistency
tablet 1’
tablet 1’
yb-tserver1 yb-tserver2
yb-tserver3
tablet 1’
tablet2-leader
tablet3-leader
tablet1-leader
master-follower
master-leader
master-follower
YB-Master
Manage shard metadata &
coordinate system-wide ops
YB-TServer
Host & serve user data
tablet1-follower
tablet1-follower
tablet3-follower
tablet2-follower
tablet3-follower
tablet2-follower
…
…
…
Zero Data Loss Persistence
yb-master1
yb-master3
yb-master2
1 Update record in tablet3
2 Get latest tablet-leader locations and send to client for future caching
3Redirect to current tablet3-leader
44Sync update follower replicas via Raft Sync update follower replicas via Raft
5
Wait for majority replicas to commit
Commit to local DocDB
Ack client after local commit

13. 13© 2018 All rights reserved.
…
…
Elastic Scaling
tablet 1’
tablet 1’
yb-tserver1 yb-tserver2
yb-tserver3
tablet 1’
tablet2-leader
tablet3-leader
tablet1-leader
master-follower
master-leader
master-follower
YB-Master
Manage shard metadata &
coordinate system-wide ops
YB-TServer
Host & serve user data
tablet1-follower
tablet1-follower
tablet3-follower
tablet2-follower
tablet3-follower
tablet2-follower
Add & Remove Nodes at Will
yb-master1
yb-master3
yb-master2
tablet4-leader
tablet4-follower
tablet4-follower
…
n1
n3
n2
2 a leader moves off n1
3 followers move off
n2 and n3
tablet 1’
yb-tserver4
n4
NEW NODE
ADDED
1

14. 14© 2018 All rights reserved.
…
…
Elastic Scaling
tablet 1’
tablet 1’
yb-tserver1 yb-tserver2
yb-tserver3
tablet 1’
tablet2-leader
tablet3-leader
tablet1-leader
master-follower
master-leader
master-follower
YB-Master
Manage shard metadata &
coordinate system-wide ops
YB-TServer
Host & serve user data
tablet1-follower
tablet2-follower
tablet3-follower
tablet2-follower
Add & Remove Nodes at Will
yb-master1
yb-master3
yb-master2
tablet 1’
yb-tserver4
tablet4-follower
tablet4-follower
…
tablet4-leader
tablet1-follower
tablet3-follower
…
n1
n3
n2
n4
NEW NODE
BALANCED WITH ALL
OTHER NODES
4

15. 15© 2018 All rights reserved.
High Fault Tolerance
tablet 1’
tablet 1’
yb-tserver1 yb-tserver2
yb-tserver3
tablet 1’
tablet3-leader
tablet1-leader
master-follower
master-leader
master-follower
YB-Master
Manage shard metadata &
coordinate system-wide ops
YB-TServer
Host & serve user data
tablet1-follower
tablet2-follower
tablet3-follower
tablet2-leader
…
…
…
Resilient & Self-Healing
yb-master1
yb-master3
yb-master2
2
Update record in
tablet2, see node
unreachable
3 Get latest tablet-leader locations and cache for future
4
Update via current tablet2-leader
5
Sync update follower replica via Raft
6
Wait for majority replicas to commit
Commit to local DocDB
Ack client after local commit
New tablet leaders elected in seconds1

16. 16© 2018 All rights reserved.
Superior Read Performance
3x Throughput of Traditional Quorum-based NoSQL
Low, Predictable p99 Latencies
tablet 1’
tablet 1’
yb-tserver1 yb-tserver2
yb-tserver3
tablet 1’
tablet2-leader
tablet3-leader
tablet1-leader
master-follower
master-leader
master-follower
YB-Master
Manage shard metadata &
coordinate system-wide ops
YB-TServer
Host & serve user data
tablet1-follower
tablet1-follower
tablet3-follower
tablet2-follower
tablet3-follower
tablet2-follower
…
…
…
yb-master1
yb-master3
yb-master2
1 Read record in tablet3 (default: STRONG read consistency level)
1 Read record in tablet3
(BOUNDED STALENESS read
consistency level)

17. 17© 2018 All rights reserved.
Benchmarks
3x Faster Than Apache Cassandra (per YCSB) 10x Data Density Per Node than Apache Cassandra
• 26 TB over 4 YugaByte DB nodes compared to 30
nodes for Apache Cassandra
• 385K reads/sec (0.25 ms) & 6.5K writes/sec on
"Recent Data” Workload
• Expand to 5 nodes (complete in 8 hours), data
available in 5 minutes
• Cluster rebalanced in 2.2 hours after a node failure

18. 18© 2018 All rights reserved.
REAL-WORLD EXAMPLE

19. 19© 2018 All rights reserved.
IoT Fleet Management App
A Streaming App on the CKY stack
https://github.com/YugaByte/yb-iot-fleet-management

20. 20© 2018 All rights reserved.
Architecture
Kafka Connect Sink for YugaByte DB
(For Raw Events & KSQL Aggregates)
Bootstrap.js
Chart.js
jQuery.js
Sockjs.js
Fleet w/ Location Tracking
Enabled Using IoT Sensors

21. 21© 2018 All rights reserved.
Running the IoT Fleet Management App
1. Install & Start Confluent Open Source Kafka Cluster
– https://www.confluent.io/download/
2. Install & Start YugaByte DB Cluster
– https://docs.yugabyte.com/latest/quick-start/install/
3. Install Kafka Connect Sink for YugaByte DB
– https://github.com/YugaByte/yb-kafka-connector
4. Create KSQL Stream & Tables as well as YugaByte DB Tables
– DDL statements available in the iot repo
5. Connect KSQL Processed Topics to sink to YugaByte DB Tables
– Command available in the iot repo
6. Start Load Generator
– Send raw events into Confluent Kafka and see aggregate results on the UI Dashboard
https://github.com/YugaByte/yb-iot-fleet-management

22. 22© 2018 All rights reserved.
DEMO

23. 23© 2018 All rights reserved.
Raw Events: Kafka Producer  YugaByte DB
{
"vehicleId":"0bf45cac-d1b8-4364-a906-980e1c2bdbcb",
"vehicleType":"Taxi",
"routeId":"Route-37",
"longitude":"-95.255615",
"latitude":"33.49808",
"timestamp":"2017-10-16 12:31:03",
"speed":49.0,
"fuelLevel":38.0
}
iot-data-event
topic
Sample Event in 'iot-data-event’ topic
Producer publishes to
Kafka
YugaByte DB Table for Storing Raw Events
CREATE TABLE TrafficKeySpace.Origin_Table (
vehicleId text,
routeId text,
vehicleType text,
longitude text,
latitude text,
timeStamp timestamp,
speed double,
fuelLevel double,
PRIMARY KEY ((vehicleId), timeStamp))
WITH default_time_to_live = 3600;
Kafka Connect Sink writes to
YugaByte DB
* Not yet used in IoT App UI

24. 24© 2018 All rights reserved.
Total Traffic: KSQL  YugaByte DB
3. Sink to YugaByte DB Table for Total Traffic
CREATE TABLE TrafficKeySpace.Total_Traffic (
routeId text,
vehicleType text,
totalCount bigint,
timeStamp timestamp,
recordDate text,
PRIMARY KEY (routeId, recordDate, vehicleType));
1. KSQL Stream on Raw Events Topic
CREATE STREAM traffic_stream (
vehicleId varchar,
vehicleType varchar,
routeId varchar,
timeStamp varchar,
latitude varchar,
longitude varchar)
WITH (
KAFKA_TOPIC='iot-data-event',
VALUE_FORMAT='json',
TIMESTAMP='timeStamp',
TIMESTAMP_FORMAT='yyyy-MM-dd HH:mm:ss');
CREATE TABLE total_traffic
WITH ( PARTITIONS=1,
KAFKA_TOPIC='total_traffic',
TIMESTAMP='timeStamp',
TIMESTAMP_FORMAT='yyyy-MM-dd HH:mm:ss') AS
SELECT routeId,
vehicleType,
count(vehicleId) AS totalCount,
max(rowtime) AS timeStamp,
TIMESTAMPTOSTRING(max(rowtime), 'yyyy-MM-dd') AS recordDate
FROM traffic_stream
GROUP BY routeId, vehicleType;
total_traffic
topic2. KSQL Total Traffic Aggregate on the Stream

25. 25© 2018 All rights reserved.
More Aggregations
Corresponding YugaByte DB TablesReuse traffic stream: additional KSQL Aggregates
poi_traffic topic
CREATE TABLE TrafficKeySpace.Poi_Traffic (
vehicleid text,
vehicletype text,
distance bigint,
timeStamp timestamp,
PRIMARY KEY (vehicleid));
CREATE STREAM poi_traffic
WITH ( PARTITIONS=1,
KAFKA_TOPIC='poi_traffic',
TIMESTAMP='timeStamp',
TIMESTAMP_FORMAT='yyyy-MM-dd HH:mm:ss') AS
SELECT vehicleId,
vehicleType,
cast(GEO_DISTANCE(cast(latitude AS double),cast(longitude AS
double),33.877495,-95.50238,'KM') AS bigint) AS distance,
timeStamp
FROM traffic_stream
WHERE GEO_DISTANCE(cast(latitude AS double),cast(longitude AS
double),33.877495,-95.50238,'KM') < 30;
CREATE TABLE TrafficKeySpace.Window_Traffic (
routeId text,
vehicleType text,
totalCount bigint,
timeStamp timestamp,
recordDate text,
PRIMARY KEY (routeId, recordDate, vehicleType));
window_traffic
topic
CREATE TABLE window_traffic
WITH ( TIMESTAMP='timeStamp',
KAFKA_TOPIC='window_traffic',
TIMESTAMP_FORMAT='yyyy-MM-dd HH:mm:ss',
PARTITIONS=1) AS
SELECT routeId,
vehicleType,
count(vehicleId) AS totalCount,
max(rowtime) AS timeStamp,
TIMESTAMPTOSTRING(max(rowtime), 'yyyy-MM-dd') AS recordDate
FROM traffic_stream
WINDOW HOPPING (SIZE 30 SECONDS, ADVANCE BY 10 SECONDS)
GROUP BY routeId, vehicleType;

26. 26© 2018 All rights reserved.
Try on your laptop
docs.yugabyte.com/latest/quick-start
github.com/YugaByte/yb-iot-fleet-management
github.com/YugaByte/yb-kafka-connector

27. 27© 2018 All rights reserved.
gitter.im/YugaByte
@YugaByte
Questions?