This talk describes Klaviyo’s internal messaging system, an asynchronous application framework built around Pulsar that provides a set of high-quality tools for building business-critical asynchronous data flows in unreliable environments. This framework includes: a pulsar ORM and schema migrator for topic configuration; a retry/replay system; a versioned schema registry; a consumer framework oriented around preventing message loss and in hostile environments while maximizing observability; an experimental “online schema change” for topics; and more. Development of this system was informed by lessons learned during heavy use of datastores like RabbitMQ and Kafka, and frameworks like Celery, Spark, and Flink. In addition to the capabilities of this system, this talk will also cover (sometimes painful) lessons learned about the process of converting a heterogenous async-computing environment onto Pulsar and a unified model.
12. 02 What We Built
1. Platform Services: a team
2. Pulsar: a broker deployment
3. StreamNative: a support relationship
4. Chariot: an asynchronous application framework
13. ORM for Pulsar Interactions
for tenant in Tenant.search(name="standalone"):
if tenant.allowed_clusters == ["standalone"]:
ns = Namespace(
tenant=tenant,
name="mynamespace",
acknowledged_quota=AcknowledgedMessageQuota(age=timedelta(minutes=10)),
)
ns.create()
topic = Topic(
namespace=ns,
name="mytopic"
)
topic.create()
subscription = Subscription(
topic=topic,
name="mysubscription",
type=SubscriptionType.KeyShared,
)
subscription.create()
assert Subscription.get(name="mysubscription") == subscription
consumer = subscription.consumer(name="myconsumer").connect()
while True:
message = consumer.receive()
consumer.acknowledge(message)
23. 03 Challenges
● Distribution as a library/framework
rather than an application
● Python/C++ Pulsar client maturity
● Combining advanced broker features
surfaced bugs
● Forking consumer daemons +
threaded clients + async/await style
is a costly combination
● Expectation management
● The “gap ledger”
● Management API quality
24. 04 What Worked Well
Process:
● Support from above
● Managed rollout speed
● Solving 2025’s problems, not 2022’s
● “Steel-thread” style focus on specific
use-cases
● Willingness to commit to bring work
in-house and start fresh where it
made sense
Technology:
● Declarative schemas for messages
and dataflows
● Schema registry as code rather than
a SPOF
● Managed Pulsar allows us to learn
with less pain
● Isolating user code from consumer
code improves reliability
25. 05 What’s Next?
Near Term:
● Manage internal adoption
● Scale to meet annual shopping
holidays’ needs
● Start work on a “publish gateway” for
connection pooling, circuit breaking,
etc.
Long Term:
● Online schema changes
● Key-local state
● Complex workflow support
● Make our work available to the
community