Overview of Building and Scaling Data Lineage at Netflix and our vision of covering the diverse landscape

1. Building and Scaling
Data Lineage at Netflix
Strata SF 2019
Di Lin, Girish Lingappa, Jitender
Aswani (@jaswani)

2. CTA: Intentionally Purposeful to
Improve Diversity in Tech

3. People, Security,
& Infra Data Eng
(DEI)
Data Lineage Asset Inventory Anomaly Detection
Leverage data to
improve cloud infra

4. Show of hands

5. in thinking / planning stages?

6. currently
building the
system?

7. done
building?

8. Imagination Trip
(User Stories)

9. #1 Data Scientist
validate
source of the
metrics?

10. #2 Software Engineer
who
downstream to
you service is
impacted?

11. #3 Reliability Engineer
identify and
alert on jobs /
table at risk of
missing SLA?

12. Detection and
Data Cleansing
Use Cases Inspired Lineage
Retention and
Data Efficiency
Cost Attribution Platform Reliability
Data Integrity

13. intentionality
behind complex
data landscape

14. Freedom & Responsibility
distributed decision
making =>
● move fast,
● scale complexity,
● Judicious about
what’s best for Netflix

15. Data Landscape

16. Top Challenges

17. Challenge # 1
Diverse Data Landscape
First time expanding beyond traditional data
warehouse realm

18. Challenge # 2
Platform Evolution
New Platforms Upgrades

19. Challenge # 3
Data Conformance
Variance in granularity of the meta data

20. Design Principles

21. Design Principle # 1
Ensure data coverage

22. Design Principle # 2
Enable seamless integration

23. Design Principle # 3
Envision a flexible data model

24. Data Coverage

25. Data Coverage - data gets pushed
● Platform tools publish events (preferred approach)
○ Compute engines
○ Data movement tools
○ Reporting tools

26. Data Coverage - we pull & parse data
● Ingestion scripts parse logs and metadata
○ Enumerate jobs and scripts
○ Parse plan info
○ Parse queries

27. Lineage Data Flow

29. Use Cases

30. #1: Platform Reliability
● SLA Recommendation
● Predict SLA miss
● Instructive notifications
● Anomaly detection

31. #2 : Data visibility
● Reinstate key tables
● Deprecate tables
● Schema changes
● Contract no longer held

32. #3: Efficiency
● Cost evaluation for entities
○ Storage
○ Compute
○ Usage
● Retention
● Disaster recovery

33. Please visit us at
jobs.netflix.com
Continue your journey...
Lineage blog
Infra DE team blog

34. Netflix @ Strata
● Wednesday
○ Self-service data platform
● Thursday
○ App performance measurement
○ Security detection platform
○ Personalization enhancement 4:40 PM @ 2002
11:50 AM @ 2002
11:50 AM @ 2006
2:40 PM @ 2024

35. ???

36. Thank you.

37. “Provide a complete and accurate
data lineage system enabling
decision-makers to win moments
of truth”
Lineage Mission