Big Data Architecture Systems Open Source
•Als PPT, PDF herunterladen•
0 gefällt mir•675 views
Empfohlen
Empfohlen
Weitere ähnliche Inhalte
Was ist angesagt?
Was ist angesagt? (20)
Ähnlich wie Big Data Architecture Systems Open Source
Ähnlich wie Big Data Architecture Systems Open Source (20)
Kürzlich hochgeladen
Kürzlich hochgeladen (20)
Big Data Architecture Systems Open Source
- 1. Big Data and Me Bhupesh Bansal Feb 3, 2012
- 2. Relational Model Architecture Reference : http:// www.slideshare.net / adorepump / voldemort-nosql
- 3. Linkedin 2006 Reference : http://www.slideshare.net/linkedin/linked-in-javaone-2008-tech-session-comm
- 5. Specialized Systems Architecture Reference : http:// www.slideshare.net / adorepump / voldemort-nosql
- 6. Linkedin 2007 Reference : http://www.slideshare.net/linkedin/linked-in-javaone-2008-tech-session-comm
- 8. Batch Driven Architecture Reference : http:// www.slideshare.net / bhupeshbansal /hadoop-user-group-jan2010
- 9. Motivation I : Big Data 02/06/12 Reference : algo2.iti.kit.edu/.../fopraext/index.html
- 10. Motivation II: Data Driven Features
- 11. Motivation III: Makes Money 02/06/12 Proprietary & Confidential
- 12. Motivation IV: Big Data is cool 02/06/12
- 13. Reference : http:// www.slideshare.net / BenSiscovick /the-business-of-big-data-ia-ventures-8577588
- 28. How Voldemort was born ? Reference : 1) http://www.slideshare.net/bhupeshbansal/hadoop-user-group-jan2010 2) http://www.slideshare.net/adorepump/voldemort-nosql
- 40. What do we use Hadoop/Voldemort for ? Proprietary & Confidential 02/06/12
- 41. Batch Driven Architecture Reference : http:// www.slideshare.net / bhupeshbansal /hadoop-user-group-jan2010
- 42. Data Flow Driven Architecture Reference : http:// sna-projects.com /blog/2011/08/ kafka /
- 43. Questions
Hinweis der Redaktion
- Example: member data--does not make sense to repeatedly join positions, emails, groups, etc. Explain about joins How to better model in java? Json like data model
- Example: member data--does not make sense to repeatedly join positions, emails, groups, etc. Explain about joins How to better model in java? Json like data model
- Statistical learning as the ultimate agile development tool (Peter Norvig), “business logic” through data rather than code
- No Joins Across data domains due to APIs Within data domains due to performance Natural operation: getAll(id…) Latency: if you want to call 30 services on your main pages, they better be quick (30 * 20ms = 600ms)
- - Strong Consistency: all clients see the same view, even in presence of updates - High Availability: all clients can find some replica of the data, even in the presence of failures Partition-tolerance: the system properties hold even when the system is partitioned high availability : Mantra for websites Better to deal with inconsistencies, because their primary need is to scale well to allow for a smooth user experience.
- Hashing .. Why do we need it ?? Basic problem : Clients need to know which data is where ?? Many ways of solving it Central configuration Hashing Linear hashing works : issue is when cluster is dynamic ?? KeyHash –node IDmapping change for a lot of entries When you add new slots Consistent hashing : preserves key –Node mapping for most of the keys and only change the minimal amount needed How to do it ?? Number of partitions ---------------------------- Arbitrary , each node is allocated many partitions (better load balancing and fault tolerance) Few hundreds to few thousands .. Key partition mapping is fixed and only ownership of partitions can change
- Give example of read and writes with vector clocks Pros and cons vs paxos and 2pc User can supply strategy for handling cases where v1 and v2 are not comparable.
- Fancy way of doing Optimistic locking
- Very simple APIS NO Range Scans .. . No iterator on KeySet / Entry SET : Very hard to fix performance Have plans to provide such an iterator
- Explain about partitions Make things fast by removing slow things, not by tuning HTTP client not performant Separate caching layer
- Transfer time: 30 minutes Can max out a gb network, so be careful
- Example: member data--does not make sense to repeatedly join positions, emails, groups, etc. Explain about joins How to better model in java? Json like data model
- Questions, comments, etc