20131011 - Los Gatos - Netflix - Big Data Design Patterns
âą
10 gefĂ€llt mirâą2,653 views
![Design Patterns for Big Data
Architecture: Best Strategies for
Streamlined [Simple, Powerful]
Design
Allen Day, PhD
Data Scientist, MapR Technologies
October 2013
©MapR Technologies - Confidential](https://image.slidesharecdn.com/2013-131124213350-phpapp01/85/20131011-Los-Gatos-Netflix-Big-Data-Design-Patterns-1-320.jpg)
Empfohlen
Empfohlen
Weitere Àhnliche Inhalte
Was ist angesagt?
Was ist angesagt? (20)
Andere mochten auch
Andere mochten auch (20)
Ăhnlich wie 20131011 - Los Gatos - Netflix - Big Data Design Patterns
Ăhnlich wie 20131011 - Los Gatos - Netflix - Big Data Design Patterns (20)
Mehr von Allen Day, PhD
Mehr von Allen Day, PhD (20)
KĂŒrzlich hochgeladen
KĂŒrzlich hochgeladen (20)
20131011 - Los Gatos - Netflix - Big Data Design Patterns
- 1. Design Patterns for Big Data Architecture: Best Strategies for Streamlined [Simple, Powerful] Design Allen Day, PhD Data Scientist, MapR Technologies October 2013 ©MapR Technologies - Confidential
- 2. Me, Us âą Allen Day, Principal Data Scientist, MapR R contributor (10 yr), Hadoop (6 yr) Human Genetics (UCLA Medicine), Machine Learning âą MapR Distributes open source components for Hadoop Adds major enhancements for performance, high-availability, and ease-of-use âą See Also â âallendayâ most places (twitter, github, etc.) â aday@maprtech.com, allenday@allenday.com â @mapR ©MapR Technologies - Confidential
- 3. Three Business Use Cases Personalized Search ©MapR Technologies - Confidential Personalized Medicine Market Segmentation
- 4. Three Business Use Cases Personalized Search Personalized Medicine ⹠Public web index + personal search history ⹠Custom ranking of results ⹠Patient medical history ⹠Genomic info. ⹠Match against database of therapies ©MapR Technologies - Confidential Market Segmentation ⹠Group similar customers ⹠Target with cross-sell / upsell campaign
- 5. Three Business Use Cases Personalized Search Personalized Medicine ⹠Public web index + personal search history ⹠Custom ranking of results ⹠Patient medical history ⹠Genomic info. ⹠Match against database of therapies Personal data Personal data Which ones are similar? ©MapR Technologies - Confidential Market Segmentation ⹠Group similar customers ⹠Target with cross-sell / upsell campaign Marketing
- 6. Three Business Use Cases Personalized Search Personalized Medicine ⹠Public web index + personal search history ⹠Custom ranking of results ⹠Patient medical history ⹠Genomic info. ⹠Match against database of therapies Personal data Personal data Which ones are similar? ©MapR Technologies - Confidential Market Segmentation ⹠Group similar customers ⹠Target with cross-sell / upsell campaign Marketing
- 7. Three Business Use Cases Personalized Search Personalized Medicine ⹠Public web index + personal search history ⹠Custom ranking of results ⹠Patient medical history ⹠Genomic info. ⹠Match against database of therapies Personal data How can you tell? Personal data ©MapR Technologies - Confidential Market Segmentation ⹠Group similar customers ⹠Target with cross-sell / upsell campaign Marketing
- 8. But First⊠WHAT IS A DESIGN PATTERN? ©MapR Technologies - Confidential
- 9. âa design pattern is a general reusable solution to a commonly occurring problem within a given context in software design. A design pattern is not a finished design that can be transformed directly into source or machine code. It is a description or template for how to solve a problem that can be used in many different situationsâ http://en.wikipedia.org/wiki/Software_design_pattern ©MapR Technologies - Confidential
- 10. History of Design Pattern Ideation 1977 Architecture & Civil Engineering ©MapR Technologies - Confidential 1994 OO Software Architecture 2012 Parallelization Software ? Application Parallelization
- 11. Not Just Software http://en.wikipedia.org/wiki/A-line ©MapR Technologies - Confidential
- 12. Big Data Application Shapes 1. How big is your input record? 2. How big is the data that is relevant to processing the input record? 3. How big is the total data that could be relevant to processing the input? 4. How fast do inputs flow in? 5. How fast do outputs need to flow out? 6. How complex (unstructured) are 1-5? 7. How predictable are 1-6? (spikiness, variance) 8. Is accuracy more important than speed? 9. Does the processing contain cycles (feedback loops)? ©MapR Technologies - Confidential
- 13. Big Data Application Shapes 1. How big is your input record? 2. How big is the data that is relevant to processing the input record? 3. How big is the total data that could be relevant to processing the input? 4. How fast do inputs flow in? 5. How fast do outputs need to flow out? 6. How complex (unstructured) are 1-5? 7. How predictable are 1-6? (spikiness, variance) 8. Is accuracy more important than speed? 9. Does the processing contain cycles (feedback loops)? Volume Velocity Variety ©MapR Technologies - Confidential
- 14. Big Data Application Shapes 1. How big is your input record? 2. How big is the data that is relevant to processing the input record? 3. How big is the total data that could be relevant to processing the input? 4. How fast do inputs flow in? 5. How fast do outputs need to flow out? 6. How complex (unstructured) are 1-5? 7. How predictable are 1-6? (spikiness, variance) 8. Is accuracy more important than speed? 9. Does the processing contain cycles (feedback loops)? ©MapR Technologies - Confidential
- 15. Big Data Application Shapes 1. How big is your input record? 2. How big is the data that is relevant to processing the input record? 3. How big is the total data that could be relevant to processing the input? 4. How fast do inputs flow in? 5. How fast do outputs need to flow out? 6. How complex (unstructured) are 1-5? 7. How predictable are 1-6? (spikiness, variance) 8. Is accuracy more important than speed? 9. Does the processing contain cycles (feedback loops)? ©MapR Technologies - Confidential
- 16. Choose a Pattern: Volume & Velocity 1. How big is your target data? <10 GB mid ? ? A Single element at a time >200 GB 2. How big is your query data? One pass over 100% B C Big storage Streaming Multiple passes over big chunks 3. How fast do you need a result? Throughput > response D ©MapR Technologies - Confidential Nearline Analytics < 100s (human scale) E Exploratory Analysis
- 17. Twitter Zeitgeist as a Composite of Design Patterns Live data source e.g. Twitter Firehose B C Big storage Streaming D ©MapR Technologies - Confidential Nearline Analytics Downstream applications
- 18. Big Data Application Shapes 1. How big is your input record? 2. How big is the data that is relevant to processing the input record? 3. How big is the total data that could be relevant to processing the input? 4. How fast do inputs flow in? 5. How fast do outputs need to flow out? 6. How complex (unstructured) are 1-5? 7. How predictable are 1-6? (spikiness, variance) 8. Is accuracy more important than speed? 9. Does the processing contain cycles (feedback loops)? Volume Velocity Variety ©MapR Technologies - Confidential
- 19. Big Data Application Shapes 1. How big is your input record? 2. How big is the data that is relevant to processing the input record? 3. How big is the total data that could be relevant to processing the input? 4. How fast do inputs flow in? 5. How fast do outputs need to flow out? 6. How complex (unstructured) are 1-5? 7. How predictable are 1-6? (spikiness, variance) 8. Is accuracy more important than speed? 9. Does the processing contain cycles (feedback loops)? Volume Velocity Variety Intents & Methods ©MapR Technologies - Confidential
- 20. Application characteristic Personalized Search Personalized Medicine Market Segmenting Input record size Co-processed data size Archive size Small Large Large Large Large Small Small Large Large Input rate Output rate Process complexity Input/process spikiness Speed or accuracy? Cycles? Fast Fast High Low Speed Yes Fast Slow High Low Accuracy No Fast Fast Low High Speed Yes ©MapR Technologies - Confidential
- 21. Percolation in Classic Form Real-time data source Real-time insertion Data store OfïŹine percolation of recent data Large-scale Incremental Processing Using Distributed Transactions and NotiïŹcations http://research.google.com/pubs/pub36726.html ©MapR Technologies - Confidential
- 22. Percolation in Classic Form Real-time data source Data store OfïŹine percolation of recent data Queue Data store Real-time insertion Queued data are unavailable for action â not percolation ©MapR Technologies - Confidential Real-time insertion Delayed insertion
- 23. Percolation in Classic Form Real-time data source Real-time insertion ©MapR Technologies - Confidential Data store OfïŹine percolation of recent data
- 24. Percolation of a Composite Store Real-time data source Real-time insertion Data store OfïŹine percolation Index Both parts visible ©MapR Technologies - Confidential
- 25. Market Segmentation âą Divide customers into subsets with common needs âą Design specific strategies for each subset âą Major emphasis on âfreshâ data ©MapR Technologies - Confidential
- 26. Market Segmentation Feature Extraction Real-time transactions Customer history Assign Segment (search) db Market Segments What does this have to do with percolation? ©MapR Technologies - Confidential query Clustering
- 27. Percolator 1 Feature Extraction Real-time transactions Customer history ©MapR Technologies - Confidential Feature extraction is percolation because it is triggered by the arrival of a new record and because it updates that new record.
- 28. Percolator 2 Real-time transactions Customer history Market segment assignment is percolation because it is triggered by the arrival of a new record and because only that record's segment is updated. What about the clustering step? ©MapR Technologies - Confidential Assign Segment (search) db Market Segments query
- 29. Scheduled Update - Not Percolation Customer history Clustering The clustering loop is not percolation since it runs at ïŹxed intervals instead of incrementally as updates are received. It also doesn't update just a single customer record. ©MapR Technologies - Confidential Market Segments
- 30. Personalized Search âą Observe web usersâ activity over an extended period âą Understand individual user interests âą Customize search results for each user âą âŠas fast as possible ©MapR Technologies - Confidential
- 31. Personal Search History and Web Index Search Persona Activity db query Persona update Histories trigger query Search Web Crawl feature extraction Doc Store ©MapR Technologies - Confidential db update trigger Doc Index Persona Index
- 32. Percolator 1 Expensive feature extraction does not block document ingest Web Crawl feature extraction Doc Store ©MapR Technologies - Confidential
- 33. Percolators 2 and 3 Persona Activity Persona update Histories Web Crawl Doc Store ©MapR Technologies - Confidential update Doc Index Persona Index
- 34. Percolator 4 Updates to personas trigger updates in related personas Search Persona Activity db query Persona update Histories ©MapR Technologies - Confidential Persona Index
- 35. Percolator 5? Persona Index Persona Histories trigger query Search db trigger Doc Index ©MapR Technologies - Confidential Persona and doc index updates trigger a personalization refresh
- 36. Pattern Context Persona Activity Web Crawl ©MapR Technologies - Confidential Encapsulated Process
- 37. Cyclic Dependency Graph ©MapR Technologies - Confidential
- 38. Percolator Thoughts âą M7 tables are great as the first persistence point in percolation âą In-memory flag column family works great for triggering updates â Efficient - eliminates need for queuing â Fast triggering with row & column Bloom filters âą Percolation is best supported by dedicated column families â Percolators I/O characteristics differ â M7 works especially well because it supports lots of column families ©MapR Technologies - Confidential
- 39. Cyclic Dependency Graph, M7 Schema ©MapR Technologies - Confidential
- 40. Personalized Medicine 5. Interpretation & Follow-up 4. Reporting 1. Select Tests 2. Draw Biosample 3. Genome Sequencing & Analysis ©MapR Technologies - Confidential
- 41. Personalized Medicine Applications âą Pre-conception screening âą Clinical research & trials â Drug re-targeting âą Therapeutics â Companion diagnostics â Therapy selection ©MapR Technologies - Confidential
- 42. Personalized Medicine Patient history (EHR) EHR archive Insert (eventually) db Sequence extraction Patient health context query Search Ranked therapies Genome Sample Here we do not see real-time data pushed to a persistence layer and processed offline. This pattern does not fit with percolation⊠©MapR Technologies - Confidential
- 43. Personalized Medicine Patient history (EHR) EHR archive Insert (eventually) db Sequence extraction Genome Sample ©MapR Technologies - Confidential Patient health context query Search User-based recommendation pattern Ranked therapies
- 44. Recommendation in Classic Form Queue History Archive db Recent history ©MapR Technologies - Confidential query User Search Ranked similar histories
- 45. Item-Based Recommendation in Classic Form Queue History archive Cooccurrence analysis Off-line analysis Recent history query Item linkage db Search ©MapR Technologies - Confidential Interactive recommendation Ranked items
- 46. Recommendation Thoughts âą Item-based recommendation is for efficiency â expensive step in computing co-occurrence can be done offline and cached prior to a user query âą User-based recommendation is for accuracy â user comparisons are done online to find the current best recommendation âą MapR is great for recommendation â M7 tables are high I/O performance, can eliminate queues â Faster archive updates with optimized MapReduce â High-availability for mission LIFE critical applications ©MapR Technologies - Confidential
- 47. Business Use Cases & Design Patterns Recommender â Personalized Medicine Pattern X â Health data Percolator â Personalized Search Percolator â Other Industry Percolator â Personalized Medicine Pattern X â Other Industry ©MapR Technologies - Confidential
- 48. Summary: Best Practices âą Look at the big picture â Find recurring patterns âą Design systems at a high-level â Solve problems once and reuse components â Increase R&D productivity â Decrease operational and maintenance overhead ©MapR Technologies - Confidential
- 49. Thank You! Allen Day, PhD Principal Data Scientist, MapR Technologies aday@maprtech.com, allenday@allenday.com @allenday, @mapr ©MapR Technologies - Confidential
Hinweis der Redaktion
- Shapes too big; overwhelmI would describe three projects by short name; then add three distinct shapes, making two hearts since both healthcare; start with all line drawings; two distracting to be color
- Shapes too big; overwhelmI would describe three projects by short name; then add three distinct shapes, making two hearts since both healthcare; start with all line drawings; two distracting to be color
- Shapes too big; overwhelmI would describe three projects by short name; then add three distinct shapes, making two hearts since both healthcare; start with all line drawings; two distracting to be color
- Shapes too big; overwhelmI would describe three projects by short name; then add three distinct shapes, making two hearts since both healthcare; start with all line drawings; two distracting to be color
- Shapes too big; overwhelmI would describe three projects by short name; then add three distinct shapes, making two hearts since both healthcare; start with all line drawings; two distracting to be color
- Talk track: Both genotyping and market segmentation solutions have a useful design component known as percolation. The key idea is that there is a fast push to store data and an offline processing step that modifies data. The modified data could go back to the same data store orâŠ.Speaker: you might note that we show real-time steps in red; and non-real time steps in black.
- Talk track: Both genotyping and market segmentation solutions have a useful design component known as percolation. The key idea is that there is a fast push to store data and an offline processing step that modifies data. The modified data could go back to the same data store orâŠ.Speaker: you might note that we show real-time steps in red; and non-real time steps in black.
- Talk track: Both genotyping and market segmentation solutions have a useful design component known as percolation. The key idea is that there is a fast push to store data and an offline processing step that modifies data. The modified data could go back to the same data store orâŠ.Speaker: you might note that we show real-time steps in red; and non-real time steps in black.
- Talk track: Both genotyping and market segmentation solutions have a useful design component known as percolation. The key idea is that there is a fast push to store data and an offline processing step that modifies data. The modified data could go back to the same data store orâŠ.Speaker: you might note that we show real-time steps in red; and non-real time steps in black.
- Talk track: In market segmentation, you want to identify useful segments of your customer base to target for a market campaign, for retention, for specific product offerings, etc. What makes âgoodâ segments depends on what you want to do and how the environment changes. You may not know ahead of time what categories make useful segments. One way to find this is to capture customer histories and do a clustering step for discovery and definition of the market segments.This market segment db is then queried and updated in response to new real-time data insertion or new rounds of clustering. Specific feature extraction may also be a useful step from the customer history persistence layer.
- Talk track: the feature extraction step could be triggered by real-time data insertionâŠ
- Talk track: a second percolator processes new customer histories relative to the market segments.
- Talk track: the clustering step is not triggered by the real-time insertion; it is a scheduled step and thus not an example of percolation.What about the other use case we said was similar, the Genotyping?
- Here, we trigger updates to the persona index based on EITHERUpdates to persona history, ORUpdates to the document indexThe idea here being that if enough docs have changed or personas are finding âunusualâ stuff, the persona is stale and we should recompute it
- Talk track: MapR advantages include the smooth use of HBase on a MapR cluster for the persistence layer at the insertion point, or even better, the use of MapR M7 tables instead. There are two specific advantages to M7 (besides the all-important reliability):a)Less risk of delays/ IO storms etc that can happen with HBase. This is VERY important when pushing real-time data to a data store.b) Strategic advantage of using in-memory flags on column families â very efficient in M7 where you can have lots of column families as opposed to only a few in HBase, operationally speaking.
- Best practice: use one column family per percolator to manage their independent i/o characteristicsPrevent i/o storms
- Talk track: Now letâs consider the other health data example, genome sequencing for personalized medicine. This is an approach that can be used to get the particular genomic characteristics of a cancerous tumor and compare to known patient histories in order to select the best option for a customized therapy.
- Talk track: While percolation is not used in this example, it does represent a specialized form of recommendation: user-based recommendation.In this genome sequencing/ personalized medicine example, A very high bar is set for the accuracy of the recommendation. Here a user-based pattern is best. Letâs look at the generalized formâŠ
- Talk track: here is the basic pattern for user-based recommendation, as used in the real use case of personalized medicine. In contrast, In consumer recommendation for shopping or movie or music recommendation, rapid response is key and accuracy is slightly less important. There item-based recommendation is generally best, because the expensive step in computing co-occurrence can be done offline prior to a user query.
- Talk track: MapR advantages include the smooth use of HBase on a MapR cluster for the persistence layer at the insertion point, or even better, the use of MapR M7 tables instead. There are two specific advantages to M7 (besides the all-important reliability):a)Less risk of delays/ IO storms etc that can happen with HBase. This is VERY important when pushing real-time data to a data store.b) Strategic advantage of using in-memory flags on column families â very efficient in M7 where you can have lots of column families as opposed to only a few in HBase, operationally speaking.