Presented at Hadoop Summit'2016

1. The Evolution of Big Data Pipelines At Intuit
June 30, 2016
#hadoopsummit #HS16SJ
@rmlokesh
@rekhajoshm

2. Your Speakers
Lokesh Rajaram
Senior Software Engineer, Intuit
likes Photography
Rekha Joshi
Principal Software Engineer, Intuit
Currently likes Chopped

3. • Evolution!
• The Need
• The Recipe
• The Stages
• Q&A
The Plan

4. Evolution Is Change, And Is Everywhere!
Unicellular
Amoeba
Multicellular
Humans

5. Cannot Evolve? Disappear..
Gone!

6. Evolution of Big Data

7. Our Mission
To improve our customers’ financial lives so profoundly … they
can’t imagine going back to the old way!

8. Consumers Small Businesses Accounting Professionals
Who we serve

9. 42M 2.3M 7MFile their own taxes with
TurboTax
Run their small businesses
with QuickBooks
Manage their personal finances
with Mint
The Numbers Are Growing
65+ Applications, 25% of US GDP

10. Era of Windows Era of Web Era of the CloudEra of DOS
Intuit - An Evolution Case Study
Compliant
data
Mobile First
1980s 1990s 2000s
• Employees: 150
• Customers: 1.3M customers
• Revenue: $33M
• Employees: 4,500
• Customers: 5.6M
• Revenue: $1.04B
• Employees: 7,700
• Customers: 37M
• Revenue: $4.2B
20162010
Regulatory data Transactional data Batch data Real time data Complex, secure data

11. Data Is The Decision Maker

12. Evolution of Big Data Pipelines – The Need
Secure Cloud
Environment
Single Cohesive
Data Pipeline
AB Testing
Personalization
Streaming
Profile Store
Fraud Detection
Support Varied
Use Cases
and more..

13. Evolution of Big Data Pipelines
Thin Slices - Minimal Viable Product

14. Evolution of Big Data Pipelines – The Recipe
Taking the
Data In
Transforming
Data
Handling The
Indigestion
With Scale

15. Evolution of Big Data Pipelines – The Recipe
No
Snowflakes
Solutions
Getting Vested
Stakeholders
Agreements
Establishing
The
Standards

16. Evolution of Big Data Pipelines – The Recipe
Breaking The
Silos
Moving Organization
In
One Direction

17. Evolution of Big Data Pipelines – The Recipe
● Making The Configuration Knobs Work
● At Scale Performance
● Latency
● Consumer Facing: Few ms
● Analytic Reports: Few secs/mins
● Throughput Requirements – Varied on business units, time
● Schema, PII, Metadata, Changes, Audit, Governance
● Controlled Access←→ Innovation
● Lag, Ordering, Duplication, Processing
● Error Monitoring and Failover
● Cluster Deployment

18. Organization Evolution ßà Data Evolution

19. SDK
User-entered
data
Apache Kafka
Collector: User-entered and
clickstream data
Real-time
processing
Personalization
Engine
Profile Store
Big Data Pipeline Slice View

20. Big Data Pipeline Components

21. Monitoring The Pipeline
AWS resource alarms
Custom App MetricsJVM and App Metrics
Custom process alerts
Logging and alert

22. Evolution In Stages

23. Evolution - Stage 0: Disparate And Chaotic
Disparate Databases

24. Data Pipeline ( an example)
Collect event stream data into one location
Handle ~ 200k events / sec
Payload ~ 3-5KB
Enrich message and load it into Hive in defined SLA

25. Evolution - Stage 1
Event Stream
Oozie
Sqoop
Netezza
Loader
Hive QL
operations
Storm
Samza
Flume

26. Evolution - Stage 2
Event
Stream
{ ReST }

27. Evolution - Stage 3 (HA & DR)
SDK
{ ReST }
SDK
{ ReST }
Mirroring

28. Challenges & opportunities

31. Set of Changes
• Network upgrades
• Increase pipe
• Broker
• Mirrormaker
• Host TCP

33. Evolution - Stage 4 (Streaming + Batch)
SDK
{ ReST }
SDK
{ ReST }
Mirroring

34. Evolution - Stage 5 (Cloud only - Future state)
SDK
{ ReST }
Kafka Connectors

35. Evolution - Stage 5 (Cloud only - Future state)

36. Pipeline Essentials
SDK { ReST }
SDK { ReST }

37. Traffic Rate Monitoring

38. Pipeline Characteristics And Testing
Event > 300 MB/s
Pipeline latency: Few ms
Pipeline next sequence flow which involves batch processes to hive: few secs
Tests:
●Data Loss
●Data Parity
●Pipeline Validation
●Pipeline Load Capacity

39. Data Pipeline Testing (Sample Logs)
ScanWorkerHDFS.java:86:ERROR:[pool-3-thread-2] (post.latitude, 44.9800, post.latitude, 44.9729)
ScanWorkerHDFS.java:86:ERROR:[pool-3-thread-2] (post.longitude, -93.2638, post.longitude, -
93.2873)
ScanWorkerHDFS.java:86:ERROR:[pool-3-thread-2] (post.manufacturer, Samsung, post.(missing), )
ScanWorkerHDFS.java:86:ERROR:[pool-3-thread-2] (post.(missing), , post.metroCode, 613)
(post.netspeed, high, post.netspeed, Cable/DSL)

40. Evolution is Continuous..

41. Interested in Joining?
goo.gl/BLPfyR

42. Thank You!!