This document discusses Apache Pulsar, an open-source distributed messaging and streaming platform. It provides concise summaries of Pulsar's key capabilities: 1) Pulsar provides messaging and streaming capabilities with a unified model and supports high throughput, low latency, availability, and durability. 2) It uses Apache BookKeeper for durable log storage and supports features like geo-replication, multi-tenancy, and lightweight compute functions. 3) Pulsar has been widely adopted with over 100 customers and has a large open-source community around it.

1. © 2019 SPLUNK INC.
The Next Generation
Messaging and Queuing
System

2. © 2019 SPLUNK INC.
Intro
Senior Principal Engineer - Splunk
Co-creator Apache Pulsar
Matteo Merli
Senior Director of Engineering - Splunk
Karthik Ramasamy

3. © 2019 SPLUNK INC.
Messaging and Streaming

4. © 2019 SPLUNK INC.
Messaging
Message passing between
components, application,
services

5. © 2019 SPLUNK INC.
Streaming
Analyze events that just
happened

6. © 2019 SPLUNK INC.
Messaging vs Streaming
2 worlds, 1 infra

7. © 2019 SPLUNK INC.
Use cases
● OLTP, Integration
● Main challenges:
○ Latency
○ Availability
○ Data durability
○ High level features
■ Routing, DLQ, delays, individual acks
● Real-time analytics
● Main challenges:
○ Throughput
○ Ordering
○ Stateful processing
○ Batch + Real-Time
Messaging Streaming

8. © 2019 SPLUNK INC.
Storage
Messaging
Compute

9. © 2019 SPLUNK INC.
Apache Pulsar
Data replicated
and synced to
disk
Durability
Low publish
latency of 5ms at
99pct
Low
Latency
Can reach 1.8 M
messages/s in a
single partition
High
Throughput
System is
available if any 2
nodes are up
High
Availability
Take advantage
of dynamic
cluster scaling in
cloud
environments
Cloud
Native
Flexible Pub-Sub and Compute backed by durable log storage

10. © 2019 SPLUNK INC.
Apache Pulsar
Support both
Topic & Queue
semantic in a
single model
Unified
messaging
model
Can support
millions of topics
Highly
Scalable
Lightweight
compute
framework based
on functions
Native
Compute
Supports multiple
users and
workloads in a
single cluster
Multi
Tenant
Out of box
support for
geographically
distributed
applications
Geo
Replication
Flexible Pub-Sub and Compute backed by durable log storage

11. © 2019 SPLUNK INC.
Apache Pulsar project in numbers
192
Contributors
30
Committers
100s
Adopters
4.6K
Github Stars

12. © 2019 SPLUNK INC.
Sample of Pulsar users and contributors

13. © 2019 SPLUNK INC.
Messaging Model

14. © 2019 SPLUNK INC.
Pulsar Client libraries
● Java — C++ — C — Python — Go — NodeJS — WebSocket APIs
● Partitioned topics
● Apache Kafka compatibility wrapper API
● Transparent batching and compression
● TLS encryption and authentication
● End-to-end encryption

15. © 2019 SPLUNK INC.
Architectural view
Separate layers between
brokers bookies
● Broker and bookies can
be added independently
● Traffic can be shifted very
quickly across brokers
● New bookies will ramp up
on traffic quickly

16. © 2019 SPLUNK INC.
Apache BookKeeper
● Low-latency durable writes
● Simple repeatable read
consistency
● Highly available
● Store many logs per node
● I/O Isolation
Replicated log storage

17. © 2019 SPLUNK INC.
Inside
BookKeeper
Storage optimized for
sequential & immutable data
● IO isolation between write and read
operations
● Does not rely on OS page cache
● Slow consumers won’t impact latency
● Very effective IO patterns:
○ Journal — append only and no reads
○ Storage device — bulk write and
sequential reads
● Number of files is independent from number
of topics

18. © 2019 SPLUNK INC.
Segment
Centric
Storage
In addition to partitioning, messages are stored
in segments (based on time and size)
Segments are independent from each others and
spread across all storage nodes

19. © 2019 SPLUNK INC.
Segments vs Partitions

20. © 2019 SPLUNK INC.
Tiered
Storage
Unlimited topic storage capacity
Achieves the true “stream-storage”: keep the raw
data forever in stream form
Extremely cost effective

21. © 2019 SPLUNK INC.
Schema Registry
Store information on the data structure — Stored in BookKeeper
Enforce data types on topic
Allow for compatible schema evolutions

22. © 2019 SPLUNK INC.
Schema Registry
● Integrated schema in API
● End-to-end type safety — Enforced in Pulsar broker
Producer<MyClass> producer = client
.newProducer(Schema.JSON(MyClass.class))
.topic("my-topic")
.create();
producer.send(new MyClass(1, 2));
Consumer<MyClass> consumer = client
.newConsumer(Schema.JSON(MyClass.class))
.topic("my-topic")
.subscriptionName("my-subscription")
.subscribe();
Message<MyClass> msg = consumer.receive();
Type Safe API

23. © 2019 SPLUNK INC.
Geo
Replication
Scalable asynchronous replication
Integrated in the broker message flow
Simple configuration to add/remove regions

24. © 2019 SPLUNK INC.
Replicated Subscriptions
● Consumption will restart close to where a consumer left off - Small amount of dups
● Implementation
○ Use markers injected into the data flow
○ Create a consistent snapshot of message ids across cluster
○ Establish a relationship: If consumed MA-1 in Cluster-A it must have consumed
MB-2 in Cluster-B
Migrate subscriptions across geo-replicated clusters

25. © 2019 SPLUNK INC.
Multi-Tenancy
● Authentication / Authorization / Namespaces / Admin APIs
● I/O Isolations between writes and reads
○ Provided by BookKeeper
○ Ensure readers draining backlog won’t affect publishers
● Soft isolation
○ Storage quotas — flow-control — back-pressure — rate limiting
● Hardware isolation
○ Constrain some tenants on a subset of brokers or bookies
A single Pulsar cluster supports multiple users and mixed workloads

26. © 2019 SPLUNK INC.
Lightweight Compute with
Pulsar Functions

27. © 2019 SPLUNK INC.
Pulsar Functions

28. © 2019 SPLUNK INC.
Pulsar Functions
● User supplied compute against a
consumed message
○ ETL, data enrichment, filtering, routing
● Simplest possible API
○ Use language specific “function” notation
○ No SDK required
○ SDK available for more advanced
features (state, metrics, logging, …)
● Language agnostic
○ Java, Python and Go
○ Easy to support more languages
● Pluggable runtime
○ Managed or manual deployment
○ Run as threads, processes or containers
in Kubernetes

29. © 2019 SPLUNK INC.
Pulsar Functions
def process(input):
return input + '!'
import java.util.function.Function;
public class ExclamationFunction
implements Function<String, String> {
@Override
public String apply(String input) {
return input + "!";
}
}
Python Java
Examples

30. © 2019 SPLUNK INC.
Pulsar Functions
● Functions can store state in stream storage
● State is global and replicated
● Multiple instances of the same function can access the same state
● Functions framework provides simple abstraction over state
State management

31. © 2019 SPLUNK INC.
Pulsar Functions
● Implemented on top of Apache BookKeeper “Table Service”
● BookKeeper provides a sharded key/value store based on:
○ Log & Snapshot - Stored as BookKeeper ledgers
○ Warm replicas that can be quickly promoted to leader
● In case of leader failure there is no downtime or huge log to replay
State management

32. © 2019 SPLUNK INC.
Pulsar Functions
State example
import org.apache.pulsar.functions.api.Context;
import org.apache.pulsar.functions.api.PulsarFunction;
public class CounterFunction
implements PulsarFunction<String, Void> {
@Override
public Void process(String input, Context context) {
for (String word : input.split(".")) {
context.incrCounter(word, 1);
}
return null;
}
}

33. © 2019 SPLUNK INC.
Pulsar IO
Connectors Framework based on Pulsar Functions

34. © 2019 SPLUNK INC.
Built-in Pulsar IO connectors

35. © 2019 SPLUNK INC.
Querying data stored
in Pulsar

36. © 2019 SPLUNK INC.
Pulsar SQL
● Uses Presto for interactive SQL
queries over data stored in Pulsar
● Query historic and real-time data
● Integrated with schema registry
● Can join with data from other
sources

37. © 2019 SPLUNK INC.
Pulsar SQL
● Read data directly from BookKeeper into Presto — bypass Pulsar Broker
● Many-to-many data reads
○ Data is split even on a single partition — multiple workers can read data in parallel from single
Pulsar partition
● Time based indexing — Use “publishTime” in predicates to reduce data being read
from disk

38. © 2019 SPLUNK INC.
Pulsar Storage API
● Work in progress to allow direct access to data stored in Pulsar
● Generalization of the work done for Presto connector
● Most efficient way to retrieve and process data from “batch” execution engines

39. Thank You
© 2019 SPLUNK INC.