When we set out to launch our Conversations Platform at Expedia Group our goals were simple. Enable millions of travelers to have natural language conversations with an automated agent via text, Facebook, or their channel of choice. Let them book trips, make changes or cancellations, and ask questions -- “How long is my layover?” “Does my hotel have a pool?” How much will I get charged if I want to bring my golf clubs?”. Then take all that we know about that customer across all of our brands and apply machine learning models to give customers what they are looking for immediately and automatically, whether it be a straightforward answer or a complex new itinerary. And the final goal: go from zero to production in four months. Such a platform is no place for batch jobs, back-end processing, or offline APIs. To quickly make decisions that incorporate contextual information, the platform needs data in near real-time and it needs it from a wide range of services and systems. Meeting these needs meant architecting the Conversations Platform around a central nervous system based on Confluent Cloud and Apache Kafka. Kafka made it possible to orchestrate data from loosely coupled systems, enrich data as it flows between them so that by the time it reaches its destination it is ready to be acted upon, and surface aggregated data for analytics and reporting. Confluent Cloud made it possible for us to meet our tight launch deadline with limited resources. With event streaming as a managed service, we had no costly new hires to maintain our clusters and no worries about 24x7 reliability. When we built the platform, we did not foresee the worldwide pandemic and the profound effect it would have on the travel industry. Companies were hit with a tidal wave of customer questions, cancellations, and rebookings. Throughout this once-in-a-lifetime event, the Conversations Platform proved up to the challenge, auto-scaling as necessary and taking much of the load off of live agents. In this session, we’ll share how we built and deployed the Conversations Platform in just four months, the lessons we learned along the way, key points to consider for anyone architecting a platform with similar requirements, and how it handled the unprecedented demands placed upon it by the pandemic. We’ll also show a demo of the platform that includes high-level insights obtained from analytics and a visualization of the low-level events that make up a conversation.

1. Topic
Governance
Ravi Vankamamidi
Director Technology,
Expedia Group
1

2. Challenges @ Scale
• Managing Topic lifecycle is hard
• Low latency operations need Planning
2

3. Topic Governance
Management layer on top of Kafka Topics
3

4. What is it?
4
Topic
Governance
Topic
explosion &
event
ordering
Latency issues
Topic
Discovery
Auditability
Unstructured
data
Service
choking Partition
Calculator
Standardized Naming
Convention
Store ownership
metadata
Schema contract
ACL
Management
Event Group

5. Create Topic - How does it work?
5
Request Topic
• Sizing
information
(partition)
• Justification
Topic Onboarding
Service
Metadata
DB
Schema
Registry
Topic 1 Topic 2
Topic 3
Topic 6
Topic 4
Topic 5
Approved
Approver
Rejected

6. Request Service Access - How does it work?
6
Topic Onboarding
Service
Metadata
DB
Schema
Registry
Topic 1 Topic 2
Topic 3
Topic 6
Topic 4
Topic 5
Approved
Approver
Rejected
Request ACL
• Service
information
• Justification
Topic 5
ACL
Service 1
SDK

7. Demo
7

8. Conclusion
• Topic Governance in Kafka can improve manageability of Kafka
8
Partition Calculator ACL ManagementEvent Group

9. Q&A
9