"Comcast has made a concerted effort to transform itself from a cable/ISP company to a technology company. Data-driven decision making is at the heart of this transformation, and we use data to understand how customers interact with our products, and we see data as the most truthful representation of the voice of our customer. My team, Product Analytics & behavior science (PABS) team plays the role as interpreter, transforming data into consumable insights. The X1 entertainment operating system, is one of the largest video streaming platforms in the world, and our customers consume more than a billion hours of content a week on X1. Our team consumes X1 telemetry at a rate of more than 25TBs of data per day and uses this data to inform our product teams members about the performance of and engagement with the platform. We also use this data to research customer behaviors to help better inform our product team members about areas of opportunity in our products, which range from fixing bugs to creating new features. To power these insights, we need to have a reliable real-time data pipelines to deliver these insights, and we need our data scientists and data engineers to be able to quickly and efficiently be able to develop and commit new code to ensure we can measure new features the product teams are developing. To do this in an environment at this scale, we have been using Databricks, and Databricks delta to gain operational efficiencies, optimization and cost savings. Some of the features from delta that we took advantage of to achieve the desired levels of efficiencies, optimization and cost savings are: · Distributed writes to s3 (essentially eliminating 500 errors) · s3 log with fast reads and ACID transactions (massive increases in s3 scans/reads, and enabling consistent views of the bucket/table) · Vacuum · Pptimize (which has allowed us to reduce a 640 node job to 40, and massively increase efficiencies of our clusters as well as our DS/DE’s)"

1. Building Sessionization
Pipeline at Scale with
Databricks Delta
April 24th, 2019

2. Comcast
-Xfinity X1
-Xfinity Internet and xFi
-Xfinity Home
-Xfinity Mobile

3. How do we improve our products?
Data captures our
customers feedback
at scale…
We decipher and
extract insights…
to enhance customer
experience
to empower data-
informed decisions…
We collect, store, and use all data in accordance with our privacy disclosures to users and applicable laws.

4. Data Scale
Billions of Events Petabytes of Stored Data Millions TPS

5. What is sessionization?

6. Challenges/Goals
1. Scalability
2. Reliability/ Robust
3. Performance

7. Value Gains
Before After
Batch process Stream process
84 jobs 3 jobs
~14 hours data delay ~7 hours data delay
Min. late data and failure support Checkpointing

8. Initial Design
Data Parse & Assign Sessionize Enrich
Scalability
Reliability
Performance

9. ?
Manually Partition Key to Enable
Scaling
Data Parse & Assign
Scalability
Reliability
Performance
Key 1
Key 2
Sessionize EnrichKey 32
s3://mybucket/key=<key>/type=<type>/date=<yyyy-mm-dd>/hour=<hh>/…

10. From Batch to Streaming
Data Parse & Assign Sessionize Enrich
Scalability
Reliability
Performance
Data

11. Delta
Data Parse & Assign Sessionize Enrich
Scalability
Reliability
Performance
Delta Delta Delta
Data
Optimize Optimize

12. ?
Random Prefixes
Data Parse & Assign Sessionize Enrich
Scalability
Reliability
Performance
Delta Delta Delta
Data
Optimize Optimize
random prefix random prefix random prefix

13. Auto Optimize and More
Data Parse & Assign Sessionize Enrich
Scalability
Reliability
Performance
Delta Delta Delta
DataData
Upsert
Delta
random prefix random prefix random prefix

14. Result
Reduced a 84 jobs process to 3 jobs
Deliver enriched data 2x faster
Increase operation friendliness
Delta
random prefix
Delta
random prefix
Delta
random prefix

15. Outcome
Scalable data pipeline that provides consumable insights to
our teams near real-time reliably.
Data Parse & Assign Sessionize Enrich
Delta Delta Delta
OKRs Experience
Enhancements
Product
Research
Feature
Developments