(Pradeep Chintam, American Express Global Business Travel) Kafka Summit SF 2018 When a new project, Global Trip Record was launched at American Express GBT and we were looking for a robust, scalable and fault-tolerant middleware to handle all the orchestration and connectivity needs of the project. The existing solution was monolithic, and we wanted to convert that to a microservices framework, but the biggest challenge was managing the increasing number of external applications that are connected to the platform. Any slow external application or partner system connected to the platform was slowing down the entire platform. There is always a need for partner systems to go offline or a need to resend the entire day’s data, especially with a system like our data lake where the data volumes are huge. After evaluating multiple solutions, we settled on Apache Kafka, and started with a simple implementation of around 100,000 messages to just decouple one partner system and the core platform. Today, we are running our microservices (Docker) running in OpenShift (Kubernetes) processing Kafka Streams, running real-time anomaly detection using Kafka Streams, powering our data lake through Kafka, feeding our distributed caching layer (Apache Ignite) and connecting all internal and external systems using Kafka. With a total of more than 10 million messages per day, i.e., 1.5TB of data with just a small three-node cluster, we are one happy platform for over a year now. With the kind of stability, flexibility and success in our project, a lot of other teams started and will soon be in production with Kafka Steams. The powerful combination of Kafka and OpenShift has proven to be an easily scalable model with great elasticity to the entire platform.

1. CREATING AN ELASTIC PLATFORM USING
KAFKA &
MICROSERVICES ON KUBERNETES.
PRADEEP CHINTAM
SENIOR ENGINEER – AMERICAN EXPRESS GLOBAL BUSINESS TRAVEL
2

2. WHY WE NEED AN ELASTIC PLATFORM.
• American Express Global Business Travel (GBT) is the world’s leading business partner for managed
travel. We help companies and employees prosper by making sure travellers are present where and
when it matters. We keep global business moving with the powerful backing of 16,000 travel
professionals in more than 140 countries. Companies of all sizes, and in all places, rely on GBT to
provide travel management services, organize meetings and events, and deliver business travel
consulting.
• Different applications for different channels OR a unified channel, a platform, which can be consumer
agnostic (almost) ?
American Express Global Business Travel
3

3. WHAT ARE THE NUMBERS.
• Middleware is the backbone of our platform.
• 2.5 million messages in a day.
• Average message size is 100 KB
• Each message is shared with at least 5 partners.
• Sharing information with 35 + partners.
• Every partner has a different version of our processed messages.
• So, what does this mean for the platform?
American Express Global Business Travel
4

4. SOME STATS ON TECHNOLOGY AND KEY FEATURES
• Break the legacy application to functional units.
• Abstract the functionality.
• 40 + Micro services for application.
• 35 + Nano services (why Nano?) for our partners.
• Make the platform not only flexible for volume, but also for technology changes.
American Express Global Business Travel
5

5. WHY MICRO/NANO SERVICES.
• The world was happy with simple Java/J2EE based applications.
• Applications were made lightweight by removing EJB components
• Tomcat helped to drive the revolution
• SOAP still has a special place.
• RESTful services? Are they really different from Servlets?
• How about creating new instances on demand with in few seconds?? Traditional applications fail at this
step.
American Express Global Business Travel
6

6. NEED FOR HIGH AVAILABILITY.
• Everything is created with high availability in mind.
• Database using No SQL
• Micro services distributed across nodes on Open shift.
• Middleware Orchestration layer distributed across nodes.
• How about high availability between other applications talking to our application and our application.
American Express Global Business Travel
7

7. KAFKA + KUBERNETES
• Great advantage of partitions.
• How Kubernetes (Open shift) helped.
• How auto scaling can rescue applications from spikes.
• Portable Micro/Nano services.
• Cloud ready.
• Cluster of 3 nodes, handling 1.5 TB of data per day.
American Express Global Business Travel
8

8. List<ConsumerRecord<String, String>> partitionRecords =
consumerRecords.records(partition);
ForkJoinPool customThreadPool = new ForkJoinPool(recordsThreadPoolSize);
try {
customThreadPool.submit(() -> {
partitionRecords.parallelStream().forEach(record -> {
// Process the record here…
}
}
American Express Global Business Travel
9

9. DEPLOYMENT MODEL
American Express Global Business Travel
10

10. HOW WE MONITORED AND MAINTAINED.
• Kafka Manager to maintain the cluster.
• Kafka-monitor for monitoring the broker health
• Offset monitor for monitoring the consumer lags.
• Custom application/library for schema registry.
• Custom library for data transfer between Kafka to DB or to another cluster.
• Surprising, but I didn't know about Confluent.
American Express Global Business Travel
11

11. USING KAFKA FOR IN MEMORY COMPUTING
APPLICATIONS.
American Express Global Business Travel
12

12. STREAMS (KTABLES) VS RDBMS
• Industry is moving away from Store everything to Process events.
• What about reports then? Use a datalake.
• Lets say we contacted a traveler who needs help because the traveler’s flight is cancelled due to bad
weather.
• How many such interactions are pending?
• How many responded yes?
• How many declined after responding Yes? What are the primary reasons? How did we do on a given
day?
American Express Global Business Travel
13

13. USING KAFKA STREAMS TO IDENTIFY RED FLAGS.
American Express Global Business Travel
14

14. FUTURE USE CASES.
• Ordering for Block Chain.
• What are the key issues in implementing blockchain?
• Why is it important to maintain ordering for transactions?
• How Kafka can be used to keep all partner on the network in sync for ordering transactions?
• Create universal cluster instead of individual clusters.
• Use KSQL for alerts and metrics.
• Monetize usage.
American Express Global Business Travel
15

15. QUESTIONS ??
American Express Global Business Travel
16