Hadoop Transforms Data and Business
•
4 gefällt mir•881 views
The document discusses the growth and impact of Hadoop over the past 4 years. It notes that Hadoop is already widely used, with over 500PB of data and thousands of implementations. It also discusses how Hadoop enables new types of analytics and insights through its ability to handle large and diverse datasets. The document outlines the growing ecosystem around Hadoop and predicts continued growth and innovation in both the technology and its applications in industries.
Empfohlen
Empfohlen
Weitere ähnliche Inhalte
Was ist angesagt?
Was ist angesagt? (19)
Ähnlich wie Hadoop Transforms Data and Business
Ähnlich wie Hadoop Transforms Data and Business (20)
Mehr von Cloudera, Inc.
Mehr von Cloudera, Inc. (20)
Kürzlich hochgeladen
Kürzlich hochgeladen (20)
Hadoop Transforms Data and Business
- 1. Hadoop: What’s Next? Mike Olson
- 2. Reflections On You 12+ months using on average 114.5TB average size 66 average nodes in Use 500+ certified on Hadoop in 1 year 60+PB Total Data from pre-conference survey
- 4. Immutable Law of Data RDBMS Hadoop Volume, Variety, Velocity increase
- 5. Immutable Law of Data RDBMS Hadoop Volume, Variety, Velocity increase Geopbytes Brontobytes Yottabytes Zettabytes Exabytes Terabytes
- 6. Linked Complex Unstructured Pre-relational Raw Detailed Heterogeneous Dirty Graphs Large Schemaless
- 7. Hadoop Was Built for Data.
- 9. Room to Grow
- 11. Hadoop: The Core of a Platform
- 12. A Platform Built by You Hue Hue SDK OozieOozie HBaseFlume, Sqoop Zookeeper / Avro Hive Pig/ Hive
- 14. A Platform Enabling Applications… Query & Reporting Complex ETL Trade Compliance POS Analysis Search Quality Click Stream Analysis Machine Learning Graph Analysis And More… Fraud Detection Archive Scientific Security
- 15. Solving Critical Business Problems • Modeling true risk • Customer churn analysis • Recommendation engine • Ad targeting • PoS transaction analysis • Analyzing network data to predict failure • Threat analysis • Trade surveillance • Search quality • Data “sandbox”
- 16. • Capture critical IT data • Monitoring usage • Driving bottom line value
- 17. • Risk analysis • Customer insight • Drive growth
- 18. • Customer intimacy • Precision targeting • Driving top line growth
- 19. So Much To See Today! • Optimizing search • Advanced analytics in the Army • Using Flume &Hive for log data • Analyzing VOIP data with R
- 20. What’s Next? Market • Adoption • Agility • Flexibility Technology • Accelerated innovation from community • More tools e.g., monitoring • More automation • More stability • More interfaces
- 21. • At the core of the open source platform for data • Four years old and going strong!
- 23. Organizational Impact • More knobs and dials • Fine grain control • Achieve previously impossible / impractical • Save money • Save time • Greater flexibility with data Copyright 2010 Cloudera Inc. All rights reserved
- 24. Hadoop World Keynote (NOTES) • Themes – Hadoop is already a big deal • Keep in mind the why • Solving real problems now – It is about the platform with Hadoop at the core • Why • Helps you profit • More accessible now than ever, real people with enterprise ops and enterprise skills, no longer the exclusive demand of the PhDs – What’s on the Horizon for Hadoop Copyright 2010 Cloudera Inc. All rights reserved
- 25. Hadoop is Having a Transformative Impact (notes) • Continued growth and excitement • Transformative to your career, your enterprise, your market – Star maker – Get ready for Hadoop being a big deal for your companies – Your market – hyper personalization – Use data to interact in a more customized fashion – “It’s hard not to have a TB of data” – Mike – Operability and SLAs for a critical enterprise platform – Education and training – A new stack for analytics (CEP (flume) CDH (Sqoop) dbms/BI) • Future is now – Use cases now and impact it is having and where it will be, look at Facebook, Yahoo, eBay etc. Copyright 2010 Cloudera Inc. All rights reserved
- 26. What is on the Horizon for Hadoop (notes) • Continued growth and excitement • Transformative to your career, your enterprise, your market – Star maker – • good for your career, help make critical changes in the way customers are supported, major new business opportunities etc. • Pull cloudera certification #’s – Get ready for Hadoop being a big deal for your companies • Enterprise will be more agile and able capture and analyze more data to better target ads, find fraud, etc. • Agility – impacts the things that matter to you • What’s happened before the transaction – Your market – hyper personalization • 100s’s of vertical apps to be created (developers are you listening?) • Trend that crosses? Any other trend we can compare to? DBMS growth? Improvements in operations, • How detailed sources have changed • Devices, understanding how people interact with your business – retail, online entertainment, fin serv, government – Use data to interact in a more customized fashion – “It’s hard not to have a TB of data” – Mike – Operability and SLAs for a critical enterprise platform – Education and training – A new stack for analytics (CEP (flume) CDH (sqoop) dbms/BI) • Future is now – Use cases now and impact it is having and where it will be, look at Facebook, Yahoo, eBay etc. Copyright 2010 Cloudera Inc. All rights reserved
- 27. Emerging Importance of Data Scientist • Able to impact business at many levels • New conference focused data and data related roles — O’Reilly Strata Conference Copyright 2010 Cloudera Inc. All rights reserved
- 28. Unprecedented Data Volume, Velocity and Variety Data Growth Out Pacing Processing Power Organizations Swamped and Turning to Hadoop 61% CAGR 42% CAGR Data Transistors Copyright 2010 Cloudera Inc. All rights reserved
- 29. Transforming Analytic Requirements • Insight into this data needs more than simple tabular analysis – More is needed for meaningful answers • You can and will do deeper and more introspective analysis – Machine learning, natural language processing, clustering, sophisticated statistical analysis, modeling and back testing • Looking for patterns – You can see patterns in lots of data that are invisible in less data. You need pattern discovery tools Copyright 2010 Cloudera Inc. All rights reserved
- 30. Hadoop: Already a Big Deal!! Massive Adoption Vibrant & Growing Community 100’s of PB Under Management 1000’s of Implementations
- 31. Benefitting From a Dynamic OS Community • Community around Hadoop is proliferating and expanding • > ½ Hadoop sub-projects promoted to TLPs • Dozens of related projects • 100’s of developers & growing Copyright 2010 Cloudera Inc. All rights reserved
- 32. Interest in Hadoop Has Exploded More are looking for it Leading analysts report significant growth in inquiries Major increase in coverage Copyright 2010 Cloudera Inc. All rights reserved
- 33. A Data Management Platform Applications Copyright 2010 Cloudera Inc. All rights reserved
- 34. Market Impact • Hyper personalization • Extreme targeting • Expand competitive advantages • Better retention of customers • Improved risk analysis
Hinweis der Redaktion
- ** 180 public, self-declared powered by Apache Hadoop and related projects (apache powered by wiki)** that’s a small fraction of users** There are 900+ people hereAttendees right now: 915Responses: 151Avg usage in months: > 12mo (for non 0 responses), 8.76mo overallAverage storage size: 114.5TB (for non 0 responses) 78TB overallTotal data under management: 11.79PB for responses, 104.7PB extrapolatedAverage # nodes: 66.25 (for non 0 responses) 46 overallTotal notes under management: 6957 for responses, 60625 extrapolatedLongest time spent on Hadoop: 36 monthsMost data: 2PBLargest cluster: 1300 nodesPercentage over 10 nodes: 55%Percentage over 10TB: 38.5%** 40% are here to start learning about Hadoop
- Last year, we told you that Hadoop was going to be a big deal. It is.You’re here, so you know that. You’re the reason why.Shout outs: Doug, Tom (new book!); Yahoo! for its early and continued contribution (security). Global community.Three things driving success.
- The big story is data. There’s a lot of it. The increase is fastest in varieties of data poorly suited to RDBMS’ tabular storage. 2.5 zettaytes (x1000 exabytes) of new information in 2012 Digital universe grew by 62% last year to 800 zettabytes and will grow to 1.2 yottabytes this year
- In case you want to know what comes next.Keynotes will sound increasingly ridiculous.
- Corollary:With volume, variety and velocity, data is messy.In the 80s and 90s, RDBMS was the answer.Now, we see row stores and column stores (Vertica), emerging class of distributed hash-style systems (Membase, Hbase, Cassandra), Hadoop. Major shift in the data center!
- Hadoop was built for data. Cheap storage lets you deal with volume, but also: variety and velocity.This technology was invented by the biggest names on the Web – Google, Facebook, Yahoo! – to solve their business problems. But those are also your business problems. Those companies weren’t different than you; they were just a little bit in the future. Hadoop is the high-value analytics engine for today’s businesses.
- It is famous for getting big – no “up to”.
- Variety and velocity sometimes come before volume.RECALL: First reason for Hadoop’s success: data.
- Second reason: Open source.Don’t hand Oracle a loaded gun.Community is thrashing proprietary. Bill Joy: All the smartest people work somewhere else.Linux, Postgres, MySQL, JBoss, Xensource: Commoditiized existing markets.Not so, Hadoop! No existing commercial product does large-scale, powerful analytics over a mixture of complex and structured data. First truly innovative OSS platform I know.
- Third reason: No longer just HDFS and MapReduce.Talking to Kellan Elliott-McRae, VP Eng Etsy. “Toolchain is mature.” Talks like an engineer, but he’s right.Last year: Hadoop. This year: A mature, flexible, powerful data management platform ready for enterprise-grade use.Cloudera has been in the market for two years, solving real problems. We’ve badly needed this platform to emerge from the pieces.We believe deeply in the open source ethos. Not political: Gives us a huge unfair advantage in the market.We spent two years listening to you.
- Today, we are announcing the newest release of CDHv3.Integrated the critical pieces of the platform from the global open source community.Incorported Yahoo!’s excellent work on security – see Todd Lipcon’s talk today.Deeply committed to OSS. This is and will always be a 100% pure Apache-licensed platform. No one can turn off your access to your data, and no one can take away your critical business analytics.
- Hadoop: Innovative, powerful, platform, but just one piece of the puzzle in your data center.Needs to play nicely with all the other systems and tools you run.Established vendors, with Cloudera, have announced integration with Hadoop. You’ll see this list grow quickly over the next year. “Right tool for the job.”
- This platform is ripe for creation of new vertical apps and analytical tools. Expect to see an explosion in the number and variety of solutions built on it.
- So what are people doing?You’re surrounded by more than 900 other people today. They’ll tell you how they’re using Hadoop to solve real business problems. Not science projects; not a toy. The platform is driving costs down and letting companies extract profit from data in ways that were impossible just a few years ago.You’ll hear about real use cases from smart people.
- You'll hear General Electric talk about how they're able to capture and monitor data on their mission-critical IT systems. They're able to track load and usage, monitor what users are doing and watch for impending failures. GE runs a large internal cloud; with the Hadoop platform, they're squeezing costs out of that cloud, and driving value straight to the bottom line.
- You'll hear Bank of America talk about its very fast-growing Hadoop cluster. BofA uses the platform to better understand its customers, and to more precisely score risk in its individual user accounts and in its portfolio as a whole. These days, the player in the market who is the most right about risk wins, and wins big. Banks can reduce their reserve capital requirements, freeing up money to invest elsewhere, and can avoid loss and drive gains by choosing market positions with the best information available.
- You'll hear eBay describe how they're using this platform to get much closer to customers. They're able to understand much better the preferences and behavior of individual users. That means they can deliver precisely-targeted products in auctions, tailored exactly to the individual. That precise targeting turns into more purchases, and more purchases drives eBay's top line.
- Forty-two sessions across a very long day.Unlike last year, we’ve scheduled lots of breaks for networking and hallway conversations – always the real reason to attend a conference!Hilary Mason from bit.ly, Flip Kromer of Infochimps, Drew Conway, Terry Jones and Clay Johnson are the young Turks transforming the industry, defining a whole new profession of "data scientist" and creating new frameworks and tools for understanding data and creating insight and profit out of information. Hit the coffee breaks and work the hallways hard.
- We are, all together, in the middle of a dramatic change in how the world works. Data is central to government, business, security and leisure. That's more true today than it was when we saw you here last year. It will be still more true next year. We have all, together, created this amazing, powerful, innovative new open source software platform.Over the next twelve months, you’ll see the market realize this even more broadly. You’ll see the platform continue to evolve and improve. You’ll see more companies take advantage of the power of Hadoop to unlock information from data and drive value and profit.
- Look how far Hadoop has come!Data, open source and the emergence of the platform have made the software mainstream. It’s enterprise-ready, and works with enterprise tools. Look at the progress since Doug created the project back in 2006. Imagine the transformation we’ll see over the next four years.And I love how, even with all that progress, the little yellow elephant is still pushing!Go out and enjoy the day. Hear some great talks and make new friends. Get a book signed by Tom. We’ll see you back here this afternoon – and don’t forget the reception afterward, one floor down.
- Tim is the founder and CEO of O’Reilly Media. Its goal: to be a catalyst for technology change by capturing and transmitting the knowledge of "alpha geeks" and other innovators.Privileged to know him for about fourteen years.Learned a great deal about open source, community and technology from conversations, his speeches and his writings.Created a number of very important industry conferences – OSCON, Etech and others. New in February – Strata, aimed squarely at data scientists.Thank Tim for coming.
- Let's also try to drive home the message that we expect hundreds of vertical apps to be built on top of CDH over the next year or two. So it's not just about the users, it's about the developers (, developers, developers!!!).
- Opportunities for individuals to find and solve interesting, professionally rewarding problems.help make critical changes in the way customers are supported and interacted withContribute to the bottom linemajor new business opportunities etc.
- New kinds of data introduced daily – mashups, sensors, log files, videos etc.Data created at machine speedI think it's worth highlighting that this platform acknowledges the three biggest trends in enterprise data management today: growth in data volumes, growth in unstructured data, and clouds/commodity hardware. Analytics and apps are compelling – see the stories at this conference
- Creation of new leadersSeparate competitors