This talk describes Klaviyo’s internal messaging system, an asynchronous application framework built around Pulsar that provides a set of high-quality tools for building business-critical asynchronous data flows in unreliable environments. This framework includes: a pulsar ORM and schema migrator for topic configuration; a retry/replay system; a versioned schema registry; a consumer framework oriented around preventing message loss and in hostile environments while maximizing observability; an experimental “online schema change” for topics; and more. Development of this system was informed by lessons learned during heavy use of datastores like RabbitMQ and Kafka, and frameworks like Celery, Spark, and Flink. In addition to the capabilities of this system, this talk will also cover (sometimes painful) lessons learned about the process of converting a heterogenous async-computing environment onto Pulsar and a unified model.

1. Building an
Asynchronous
Application Framework
with Python and Pulsar
FEBRUARY 9 2022 Pulsar Summit
Zac Bentley
Lead Site Reliability Engineer
Boston, MA

2. 2
2022 © Klaviyo Confidential
The Problem
What We Built
Challenges
What Worked Well
What’s Next?
01
02
03
04
05

3. 3
2022 © Klaviyo Confidential
Segmentation
Reviews
Retail POS
Social
Surveys
Referrals
Logistics
Shipping
Customer service
Loyalty
On site
personalization
Forms
Ecommerce
Order confirmation
SMS
Email

4. Existing Architecture

5. 5
2022 © Klaviyo Confidential
Problems
Reliability Scalability Ownership/Process Architectural

6. 6
2022 © Klaviyo Confidential
Problems
Reliability
RabbitMQ has reliability issues when
pushed too hard.
“Backpressure will find you”
Deep queues behave poorly.
Lots of outages and firefighting.
Scalability Ownership/Process Architectural

7. 7
2022 © Klaviyo Confidential
Problems
Reliability Scalability
Scaling RabbitMQ is intrusive:
application code has to be aware of
topology changes at every level.
Geometry changes are painful.
Scale-out doesn’t bring
reliability/redundancy benefits.
Ownership/Process Architectural

8. 8
2022 © Klaviyo Confidential
Problems
Reliability Scalability Ownership/Process
Individual team ownership is
expensive in:
- Roadmap time.
- Hiring/onboarding capacity.
- Coordination.
Per-team ownership creates
redundant expertise.
Architectural

9. 9
2022 © Klaviyo Confidential
Problems
Reliability Scalability Ownership/Process Architectural
Celery is pretty hostile to SOA.
Ordered consuming: not possible.
Processing more >1 message at a time:
not possible.
Pub/sub: difficult.
Replay/introspection: not possible.

10. Existing API: Producers
from app.tasks import mytask
# Synchronous call:
mytask("arg1", "arg2", kwarg1=SomeObject())
# Asynchronous call:
mytask.apply_async(args=("arg1", "arg2"), kwargs={"kwarg1": SomeObject()})
@celery.task(acks_late=True)
def mytask(arg1, arg2, kwarg1=None):
...
@celery.task(acks_late=True)
def mytask2(*args, **kwargs):
...
Existing API: Consumer Workload Declaration

11. 11
2022 © Klaviyo Confidential
Problems
Reliability Scalability Ownership/Process Architectural

12. 02 What We Built
1. Platform Services: a team
2. Pulsar: a broker deployment
3. StreamNative: a support relationship
4. Chariot: an asynchronous application framework

13. ORM for Pulsar Interactions
for tenant in Tenant.search(name="standalone"):
if tenant.allowed_clusters == ["standalone"]:
ns = Namespace(
tenant=tenant,
name="mynamespace",
acknowledged_quota=AcknowledgedMessageQuota(age=timedelta(minutes=10)),
)
ns.create()
topic = Topic(
namespace=ns,
name="mytopic"
)
topic.create()
subscription = Subscription(
topic=topic,
name="mysubscription",
type=SubscriptionType.KeyShared,
)
subscription.create()
assert Subscription.get(name="mysubscription") == subscription
consumer = subscription.consumer(name="myconsumer").connect()
while True:
message = consumer.receive()
consumer.acknowledge(message)

14. Declarative API for Schema Management & Migrations
from klaviyo_schema.registry.teamname.data.payload_pb2 import PayloadProto
my_topic = ChariotTopic(
name="demo",
durability=Durability.DURABILITY_REDUNDANT,
max_message_size="1kb",
max_producers=100,
max_consumers=10,
publish_rate_limits=(
RateLimit(
messages=1000,
period="1m",
actions=[RateLimitAction.RATE_LIMIT_ACTION_BLOCK],
),
),
thresholds=(
Threshold(
kind=ThresholdKind.THRESHOLD_KIND_UNACKNOWLEDGED,
size="200mb",
actions=[ThresholdAction.THRESHOLD_FAIL_PUBLISH],
),
),
consumer_groups=(ConsumerGroup(name="demo-consumer-group", type=SubscriptionType.KeyShared),),
payload=RegisteredPayloadFromClass(payload_class=PayloadProto),
)

15. Existing API: Producers
from app.tasks import mytask
# Synchronous call:
mytask("arg1", "arg2", kwarg1=SomeObject())
# Asynchronous call:
mytask.apply_async(args=("arg1", "arg2"), kwargs={"kwarg1": SomeObject()})
@celery.task(acks_late=True)
def mytask(arg1, arg2, kwarg1=None):
...
@celery.task(acks_late=True)
def mytask2(*args, **kwargs):
...
Existing API: Consumer Workload Declaration

16. New API: Producers
class DemoExecutor(AsynchronousExecutor):
@lifecycle_method(timeout=timedelta(seconds=10))
async def on_executor_shutdown_requested(self): ...
@lifecycle_method(timeout=timedelta(seconds=10))
async def on_executor_shutdown(self): ...
@lifecycle_method(timeout=timedelta(seconds=10))
async def on_executor_startup(self): ...
@lifecycle_method(timeout=timedelta(seconds=10))
async def on_message_batch(self, messages: Sequence[PayloadProto]):
for idx, msg in enumerate(messages):
if idx % 2 == 0:
await self.chariot_ack(msg)
else:
await self.chariot_reject(msg)
from klaviyo_schema.registry.teamname.data.payload_pb2 import PayloadProto
from klaviyo_schema.registry.teamname.topics import my_topic
await my_topic.send(PayloadProto(...))
New API: Consumer Workload Declaration

17. Back-Of-Queue Retries
class DemoExecutor(AsynchronousExecutor):
@lifecycle_method(timeout=timedelta(seconds=10))
@requeue_retry(
batch_predicate=retry_on_exception_type(
RetryException, retry_log_level=logging.INFO
),
message_predicate=retry_until_approximate_attempt_count(10),
delay=wait_exponential(max=timedelta(seconds=5)) + timedelta(seconds=1),
)
async def on_message_batch(self, messages: Sequence[PayloadProto]):
raise RetryException("Expected retry")
~> chariot worker start --topic demo --consumer-group democg --parallel 10
--start-executors-lazily --executor-class app.executors.demo:DemoExecutor
--message-batch-assignment-behavior AnyKeyToAnyExecutor
~> chariot worker start --topic demo --consumer-group democg --parallel 10
--start-executors-lazily --executor-class app.executors.demo:DemoExecutor
--message-batch-assignment-behavior NoOverlapBestEffortKeyExecutorAffinity
--message-batch-flush-after-items 1000 --message-batch-flush-after-time 10sec
Custom Batching and “Steering” for Parallel Execution without Reordering

18. 18
2022 © Klaviyo Confidential
Problems Solutions
Reliability Scalability Ownership/Process Architectural

19. 19
2022 © Klaviyo Confidential
Solutions
Reliability
To become a user is to express the
enforced maximum workload you’ll
run.
Pulsar’s redundancy helps weather
outages.
Deep backlogs are usable because
reads aren’t always writes.
Scalability Ownership/Process Architectural

20. 20
2022 © Klaviyo Confidential
Solutions
Reliability Scalability
The “CEO” (Central Expert Owner)
can scale out pulsar to respond to
demand.
Teams express scalability need in the
form of elevated rate limits or partition
counts.
Consultation with the community and
StreamNative is invaluable.
Ownership/Process Architectural

21. 21
2022 © Klaviyo Confidential
Solutions
Reliability Scalability Ownership/Process
Teams own producers/consumers.
Teams submit their contracts, in the
form of schema PRs, to the broker
owners.
Schema changes and backwards
compatibility aren’t simple but they
are now predictable.
Architectural

22. 22
2022 © Klaviyo Confidential
Solutions
Reliability Scalability Ownership/Process Architectural
Many new patterns are now on the table:
- Pub-sub
- Ordered consume
- Batched consumption +
out-of-order acks
- Deduplication/debouncing
Reading topics at rest improves visibility.
Async interaction with the same stream
from multiple codebases is now possible.

23. 03 Challenges
● Distribution as a library/framework
rather than an application
● Python/C++ Pulsar client maturity
● Combining advanced broker features
surfaced bugs
● Forking consumer daemons +
threaded clients + async/await style
is a costly combination
● Expectation management
● The “gap ledger”
● Management API quality

24. 04 What Worked Well
Process:
● Support from above
● Managed rollout speed
● Solving 2025’s problems, not 2022’s
● “Steel-thread” style focus on specific
use-cases
● Willingness to commit to bring work
in-house and start fresh where it
made sense
Technology:
● Declarative schemas for messages
and dataflows
● Schema registry as code rather than
a SPOF
● Managed Pulsar allows us to learn
with less pain
● Isolating user code from consumer
code improves reliability

25. 05 What’s Next?
Near Term:
● Manage internal adoption
● Scale to meet annual shopping
holidays’ needs
● Start work on a “publish gateway” for
connection pooling, circuit breaking,
etc.
Long Term:
● Online schema changes
● Key-local state
● Complex workflow support
● Make our work available to the
community

26. klaviyo.com/careers
zac@klaviyo.com