Flurry Analytic Backend - Processing Terabytes of Data in Real-time
â˘
9 gefällt mirâ˘5,158 views
Empfohlen
Empfohlen
Weitere ähnliche Inhalte
Was ist angesagt?
Was ist angesagt? (20)
Andere mochten auch
Andere mochten auch (13)
Ăhnlich wie Flurry Analytic Backend - Processing Terabytes of Data in Real-time
Ăhnlich wie Flurry Analytic Backend - Processing Terabytes of Data in Real-time (20)
Mehr von Trieu Nguyen
Mehr von Trieu Nguyen (20)
KĂźrzlich hochgeladen
KĂźrzlich hochgeladen (20)
Flurry Analytic Backend - Processing Terabytes of Data in Real-time
- 1. www.flurry.com November 14, 2013 Anthony Watkins, Senior Director of Developer Relations Processing Terabytes of Data in Real- Time @flurrymobile @antwatkins
- 3. Flurry is a leading mobile advertising and analytics provider Publisher Advertiser Audience AppCircle Applications: 10,000+ Devices/month: 300M Conversions/month: 120M AppSpot Applications: 2,500+ Devices/month: 250M Impressions/month: 7.5B Analytics Applications: 400,000 Devices/month: 1.2B Data points/month: 1.9T
- 4. ⢠Why Flurry Switched from a MapReduce Framework to pipeline processing ⢠How Flurry uses Kafka in data processing ⢠Tuning of Kafka to work in Flurryâs environment ⢠Flurry Monitoring and error handling of streams Topics The Path to Real-Time Processing www.flurry.com 4
- 6. Past Processing Model www.flurry.com 6 Device Reports NoSQL DataStore Batch Collectors MapReduce (jobs) External Action
- 7. Flurry Analytics MapReduce Architecture www.flurry.com 7 Agent Portal Data Log Processor Developer Portal Metrics Computer HDFS HBase HBase Hadoop/Hbase Jetty Jetty HTTP Binary Encoded Data Raw Data Log Archive Metrics Table (Cube) Normalized Data Storage User Profile Data MySQL Hadoop Map/Reduce Hadoop Map/Reduce Web Layer Metrics Processing
- 8. Data Collection and Processing in MR Pros www.flurry.com 8 MapReduce (jobs)
- 9. Data Collection and Processing in MR Cons www.flurry.com 9 Device Reports MapReduce (jobs) Job Time Startup Time
- 10. Flurry Kafka The Move to Kafka www.flurry.com 10
- 11. About Kafka Origin www.flurry.com 11 November 2010 June 2011 November 2012
- 12. About Kafka www.flurry.com 12 Producer ProducerProducer Kakfa Broker Consumer Consumer Consumer
- 13. About Kafka www.flurry.com 13 Kafka Broker * * Partition image courtesy of http://kafka.apache.org/images/log_anatomy.png
- 14. About Kafka www.flurry.com 14 Producer 1 Producer NProducer 2 Kafka Cluster Broker 1 P0 P2 Broker 2 P1 P3 Consumer Group C1 C2 C3
- 15. Why Kafka for Flurry www.flurry.com 15 Device Reports MapReduce (jobs) Kafka Startup Time
- 16. Introducing the Data Log Consumer (DLC) www.flurry.com 16 Agent Portal Data Log Consumer Developer Portal Metrics Computer HDFS HBase HBase Hadoop/Hbase Jetty Jetty HTTP Binary Encoded Data Metrics Table (Cube) Normalized Data Storage User Profile Data MySQL Kafka Hadoop Map/Reduce Web Layer Metrics Processing
- 17. ⢠Zookeeper timeouts ⢠Completely async service ⢠Default fsync interval ⢠Commit threshold from local environments Tuning Kafka for Flurry Challenges www.flurry.com 17
- 18. How Flurry Uses Kafka Infrastructure and Setup www.flurry.com 18 Consumer Group C1 C2 C⌠C325 Kafka Cluster B1 B2 B3 Broker P1 P2 P⌠P400 Topic
- 19. Flurry Monitoring / Error Handling Monitoring www.flurry.com 19 ⢠Alerts ⢠Consumer Failure ⢠Broker Failure Error Handling
- 20. Next Steps: 0.8 www.flurry.com 20 Data Log Consumer HDFS Kafka Data Log Consumer Kafka Kafka Cluster Broker 1 P0 P2 Broker 2 P1 P3 P1â P3â P0â P2â
- 21. Next Steps: Extended Pipeline www.flurry.com 21 Input Data NoSQL DataStore Real-Time Batch Collectors Consumer/ Producer Systems MapReduce (jobs) External Action External Action
- 22. Next Steps: Topics and Consumer Groups Infrastructure and Setup www.flurry.com 22 Consumer Group 2 C1â C2â C⌠CNâ Topic 1 Consumer Group 1 C1 C2 C⌠CN Consumer Group N C1ââ C2ââ C⌠CNââ Topic 2