(Dmitry Milman + Ankur Kaneria, Express Scripts) Kafka Summit SF 2018 Building cloud-based microservices can be a challenge when the system of record is a relational database residing on an on-premise mainframe. The challenge lies in the ability to efficiently and cost-effectively access the ever-increasing amount of data. Express Scripts is reimagining its data architecture to bring best-in-class user experience and provide the foundation of next-generation applications. This talk will showcase how Kafka plays a key role within Express Scripts’ transformation from mainframe to a microservice-based ecosystem, ensuring data integrity between two worlds. It will discuss how change data capture (CDC) is leveraged to stream data changes to Kafka, allowing us to build a low-latency data sync pipeline. We will describe how we achieve transactional consistency by collapsing all events that belong together onto a single topic, yet have the ability to scale out to meet the real time SLAs and low-latency requirements through means of partitions. We will share our Kafka Streams configuration to handle the data transformation workload. We will discuss our overall Kafka cluster footprint, configuration and security measures. Express Scripts Holding Company is an American Fortune 100 company. As of 2018, the company is the 25th largest in the U.S. as well as one of the largest pharmacy benefit management organizations in the U.S. Customers rely on 24/7 access to our services, and need the ability to interact with our systems in real time via various channels such as web and mobile. Sharing our mainframe t0 microservices migration journey, our experiences and lessons learned would be beneficial to other companies venturing on a similar path.

1. Digital Transformation in Healthcare
Building a low latency data pipeline
October 17, 2018
Dmitry Milman & Ankur Kaneria – Express Scripts

2. 2
© 2018 Express Scripts. All Rights Reserved.
Agenda
• Who we are
• Existing Architecture
• Entity Architecture
• Data Sync Platform
• Current Pipeline Footprint
• Achieving Transactional Consistency
• Data Pipeline Health
• Initial Load of Data
• What’s Next

3. 3
© 2018 Express Scripts. All Rights Reserved.
What does Express Scripts do ?
• Fortune 25. Ticker: ESRX
• Just like your medical plan covers visits to your doctor, your
Express Scripts prescription plan covers the medicine your
doctor prescribes
• We’re the largest independent manager of pharmacy benefits in
the United States and one of the country’s largest pharmacies
• Serve more than 85 million people. Every 4th American is our
member
• Claims Volume
• Daily ~6 million
• Annual ~2 Billion

4. 4
© 2018 Express Scripts. All Rights Reserved.
Executive Summary
• In the past, we rebuilt the same functionality for every big project
• Entity Platform strategy focuses on building once and reusing
• This increases up-front costs, but decreases capital spend over time & increases
speed of delivery
W H Y W E ’ R E M O V I N G TO E N TI TI E S
Pharmacy
As A
Platform
Rebates
20/20
Support
Clients
Manage
Products
Orders
Member
Prescription
Orders
Claim
Benefit
Contract
Claim
Benefit
Contract
Member
Benefit
Pharmacy
As A
Platform
Rebates
20/20
Support
Clients
Manage
Products
Orders Prescription Benefit
Member Claim Contract
Data
Sync

5. 5
© 2018 Express Scripts. All Rights Reserved.
Walking the Interaction and Business Tier Stack
User Interaction Channels
(Applications)
Business Platforms
User Interface
Logic
Human Workflow
UI Frameworks
Business platforms
consists of entities
with REST-based
APIs that
completely control
access to a data
domain.
They are first class
citizens.
RequiredRequired
Web Sites
Mobile Apps
…and more...
Entity
Platform
Entity
Platform
Entity
Platform
Data Data Data
Person
Membership
Referral
Order
Shipment
Drug
Price Quote
and more...
Interaction APIs
Aggregations
UX-Specific Views
Optional
User-specific order
status
Cross-entity items, not aligned with a UI
that don’t map to an entity.
Optional
Business Process & Logic

6. 6
© 2018 Express Scripts. All Rights Reserved.
Claims History Lookup
Member Site
(Applications)
Business Platforms
Member Website
Used by our members to
view claims history,
orders etc.
RequiredRequired
User
Accounts
Person
Entity
Claims
Entity
Data Data Data
1. Member login in. Authenticated
through User Accounts
2. A resource id that uniquely identifies
the member is retrieved
3. Using the unique member key,
appropriate member specific claims
history data is accessed and
displayed to the member
1 2 3

7. 7
© 2018 Express Scripts. All Rights Reserved.
Target Sync
Platform
Data Sync Platform
Source
Service
Platform
Source
DB
Source
Service
Target
Service
Platform
Target
Fail Topic
Retry
Stream
Processor
Target
DB
Target
Service
Source
Change Topic
Target
Change Topic
CDC
Stream
Processor
Update
Stream
Processor
on error
• Change Data Capture
• Kafka
• Stream Processors
• Entity store

8. 8
© 2018 Express Scripts. All Rights Reserved.
Pipeline Footprint at a Glance
Attunity Kafka
• Replicate
• Version 6.0
• 4 VMs
• Each 8 Core, 32 GB RAM, Linux
• Active/Standby
• Brokers
• 8 node cluster
• 8 x 24Cores, 256GB RAM, 22TB HD
• 4TB SSD for OS
• Enterprise Manager (AEM)
• Version 6.1
• 1 VM
• 8 Core, 32 GB RAM, Windows
• ZooKeeper
• 5 nodes ensemble
• 5 x 36Cores, 256GB RAM, 7TB HD
• 4TB SSD for OS and (500 GB for ZK
data)
• 1 Schema Registry, 1 Control Center

9. 9
© 2018 Express Scripts. All Rights Reserved.
Topic.T1
Topic.T2
Topic.T3
CDC
Stream Join
Table1
Table2
Table3
Tran1 Tran2 Tran3 Tran4 Tran5 Tran6
Transactional Consistency and CDC
Merged
Stream

10. 10
© 2018 Express Scripts. All Rights Reserved.
Transactional Consistency and CDC – Contd…
• Transactions can impact multiple tables
• Attunity bulk writes all events making up a transaction
• Natural order of events guaranteed only at partition level
• Kafka streams applications need to aggregate events
• Aggregated events need to be applied as a single unit

11. 11
© 2018 Express Scripts. All Rights Reserved.
Grouping Related Events
• Aggregate events in the same window
• Event emitted for each aggregation
• How to know all events arrived?
https://kafka.apache.org/11/documentation/streams/
developer-guide/dsl-api.html#session-windows

12. 12
© 2018 Express Scripts. All Rights Reserved.
Possible Solution
• Related events should be together
• https://www.confluent.io/blog/put-several-event-types-kafka-topic/
• Handle deserializing different event types
• Confluent Schema Registry supports multiple Avro types on same topic
vs.

13. 13
© 2018 Express Scripts. All Rights Reserved.
Partitioning Strategies
• Transaction
• Potential out of order events
partition 0
partition 1
partition 0
partition 1
• Entity key

14. 14
© 2018 Express Scripts. All Rights Reserved.
CDC Stream Processor
Stream
Time (ms)
0 50 100 150 200 250 300 350 400 450 500 550 600 650
UpdateTopic
• Leverages Kafka Processor API and state store
• Punctuation handles inactivity gap
State Store

15. 15
© 2018 Express Scripts. All Rights Reserved.
Update Processor
• Receives aggregated events from CDC processor
• Merges new state with current
• Create
• Update
• Integrity checks
• Error handling

16. 16
© 2018 Express Scripts. All Rights Reserved.
Pipeline Data Integrity
• Error processor
• One way flow of data
1
2
3
Source
DB
Retry Topic
CDC Topic
Entity
Update Topic
Target Sync
Platform
Retry
Stream
Processor
CDC
Stream
Processor
Update
Stream
Processor
Data
Services
REST API

17. 17
© 2018 Express Scripts. All Rights Reserved.
Reconciliation
• High level existence check
• Data Compare
Source
vs.
Target
Missing on Target
Missing on Source
+

18. 18
© 2018 Express Scripts. All Rights Reserved.
Performance of the Pipeline
• Latency monitoring
• Throughput monitoring

19. 19
© 2018 Express Scripts. All Rights Reserved.
Prerequisites
• Initial load is needed for stateful data
• Initial load strategies
• CDC provided functionality
• Sqoop/Spark
• Microbatched extraction

20. 20
© 2018 Express Scripts. All Rights Reserved.
What’s Next
• Data Lake for analytics
• Incorporate Elasticsearch for improved searching
• Microservices to act on data events in real time