Impala Unlocks Interactive BI on Hadoop
•Download as PPTX, PDF•
14 likes•21,609 views
The Cloudera Impala project is pioneering the next generation of Hadoop capabilities: the convergence of interactive SQL queries with the capacity, scalability, and flexibility of a Hadoop cluster. In this webinar, join Cloudera and MicroStrategy to learn how Impala works, how it is uniquely architected to provide an interactive SQL experience native to Hadoop, and how you can leverage the power of MicroStrategy 9.3.1 to easily tap into more data and make new discoveries.
Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Similar to Impala Unlocks Interactive BI on Hadoop
Similar to Impala Unlocks Interactive BI on Hadoop (20)
More from Cloudera, Inc.
More from Cloudera, Inc. (20)
Recently uploaded
Recently uploaded (20)
Impala Unlocks Interactive BI on Hadoop
- 1. Impala unlocks Interactive BI on Hadoop with MicroStrategy Justin Erickson | Cloudera | Senior Product Manager Jochen Demuth | MicroStrategy | Director, Partner Engineering May 2013
- 2. Agenda • Why Impala? • Architectural Overview • Real-World Use Cases • Interactive Analytics with MicroStrategy • Taking Big Data Out of Isolation ©2013 Cloudera, Inc. All Rights Reserved. 2
- 3. Why Hadoop? • Scalability • Simply scales just by adding nodes • Local processing to avoid network bottlenecks • Flexibility • All kinds of data (blobs, documents, records, etc) • In all forms (structured, semi-structured, unstructured) • Store anything then later analyze what you need • Efficiency • Cost efficiency (<$1k/TB) on commodity hardware • Unified storage, metadata, security (no duplication or synchronization) ©2013 Cloudera, Inc. All Rights Reserved. 3
- 4. What’s Impala? • Interactive SQL • Typically 5-65x faster than Hive (observed up to 100x faster) • Responses in seconds instead of minutes (sometimes sub-second) • Nearly ANSI-92 standard SQL queries with Hive SQL • Compatible SQL interface for existing Hadoop/CDH applications • Based on industry standard SQL • Natively on Hadoop/HBase storage and metadata • Flexibility, scale, and cost advantages of Hadoop • No duplication/synchronization of data and metadata • Local processing to avoid network bottlenecks • Separate runtime from MapReduce • MapReduce is designed and great for batch • Impala is purpose-built for low-latency SQL queries on Hadoop ©2013 Cloudera, Inc. All Rights Reserved. 4
- 5. Benefits of Impala 5 More & Faster Value from “Big Data” BI tools impractical on Hadoop before Impala Move from 10s of Hadoop users per cluster to 100s of SQL users No delays from data migration Flexibility Query across existing data Select best-fit file formats (Parquet, Avro, etc.) Run multiple frameworks on the same data at the same time Cost Efficiency Reduce movement, duplicate storage & compute 10% to 1% the cost of analytic DBMS Full Fidelity Analysis No loss from aggregations or fixed schemas ©2013 Cloudera, Inc. All Rights Reserved.
- 6. Impala and Hive 6 Shares Everything Client-Facing Metadata (table definitions) ODBC/JDBC drivers SQL syntax (Hive SQL) Flexible file formats Machine pool Hue GUI But Built for Different Purposes Hive: runs on MapReduce and ideal for batch processing Impala: native MPP query engine ideal for interactive SQL Storage Integration Resource Management Metadata HDFS HBase TEXT, RCFILE, PARQUET, AVRO, ETC. RECORDS Hive SQL Syntax Impala SQL Syntax + Compute FrameworkMapReduce Compute Framework Batch Processing Interactive SQL ©2013 Cloudera, Inc. All Rights Reserved.
- 7. Not All SQL on Hadoop is Created Equal 7 Batch MapReduce Make MapReduce faster Slow, still batch Remote Query Pull data from HDFS over the network to the DW compute layer Slow, expensive Siloed DBMS Load data into a proprietary database file Rigid, siloed data, slow ETL Impala Native MPP query engine that’s integrated into Hadoop Fast, flexible, cost-effective $ ©2013 Cloudera, Inc. All Rights Reserved.
- 8. Our Design Strategy 8 Storage Integration Resource Management Metadata Batch Processing MAPREDUCE, HIVE & PIG … Interactive SQL IMPALA Machine Learning MAHOUT, DATAFU HDFS HBase TEXT, RCFILE, PARQUET, AVRO, ETC. RECORDS Engines One pool of data One metadata model One security framework One set of system resources An Integrated Part of the Hadoop System ©2013 Cloudera, Inc. All Rights Reserved.
- 9. Impala Use Cases 9 Interactive BI/analytics on more data Asking new questions Query-able archive w/ full fidelity Data processing with tight SLAs Cost-effective, ad hoc query environment that offloads the data warehouse for: ©2013 Cloudera, Inc. All Rights Reserved.
- 10. Global Financial Services Company 10 Saving 90% on incremental EDW spend & improving performance by 5x Offload data warehouse for query-able archive Store decades of data cost-effectively Process & analyze on the same system Improve capabilities through interactive query on more data ©2013 Cloudera, Inc. All Rights Reserved.
- 11. Six3 Systems 11 Boosting performance by 20X for mission- critical, real-time cyber security Analyze unstructured data with flexibility & real-time response Integrate with existing desktop & BI tools Deploy in minutes with Cloudera Manager ©2013 Cloudera, Inc. All Rights Reserved.
- 12. Expedia 12 Implementing self-service BI on big data, reducing data latency by 50% Offload data warehouse for archiving, ETL & analytics Unify IT environment Continuously ingest & analyze at scale Drive greater usability & adoption of big data stack ©2013 Cloudera, Inc. All Rights Reserved.
- 13. CONFIDENTIALThe Information Contained In This Presentation Is Confidential And Proprietary To MicroStrategy. The Recipient Of This Document Agrees That They Will Not Disclose Its Contents To Any Third Party Or Otherwise Use This Presentation For Any Purpose Other Than An Evaluation Of MicroStrategy's Business Or Its Offerings. Reproduction or Distribution Is Prohibited.14 About MicroStrategy Innovator and Leader In Interactive BI Company • Top independent analytics software platform vendor • 20+ years old, publicly traded • Approximately $600M revenue in 2012. No debt, $200M+ cash in the bank • Global presence with operations in 23 countries Technology • Long-time market leader and innovator in analytics • Unique unitary architecture, known for high performance and scalability • Revolutionary Cloud-based analytics services • Innovations in mobile commerce and identity Analysts • Leader for six consecutive years in Gartner’s BI Magic Quadrant • Leader in Forrester BI Self Service Wave • #1 Ranked Mobile BI Vendor by Gartner & Dresner Advisory • Top ranking BI vendor in the BI Scorecard Customers • Millions of business users • Thousands of mission-critical applications • Nearly 4,000 customer institutions globally across all industries and government • Customers range from Global 500 giants like Chevron and Carrefour to cutting edge technology innovators like eBay and LinkedIn
- 14. CONFIDENTIALThe Information Contained In This Presentation Is Confidential And Proprietary To MicroStrategy. The Recipient Of This Document Agrees That They Will Not Disclose Its Contents To Any Third Party Or Otherwise Use This Presentation For Any Purpose Other Than An Evaluation Of MicroStrategy's Business Or Its Offerings. Reproduction or Distribution Is Prohibited.15 Retail Financial Services Communications Other Major Companies Innovators and Leaders Worldwide 4 of the Top 5 Global Retailers Manufacturing 5 of the Top 10 Automotive Companies 8 of the Top 10 Communications & Media Companies Pharmaceuticals 7 of the Top 10 Healthcare & Life Sciences Companies 6 of the Top 10 Financial Service Companies Government Federal, State, and Local Government Institutions Consumer Packaged Goods Leading Consumer Packaged Goods Companies Our Customers Are Leaders In All Industries Supporting the Most Demanding, Mission Critical BI Applications
- 15. CONFIDENTIALThe Information Contained In This Presentation Is Confidential And Proprietary To MicroStrategy. The Recipient Of This Document Agrees That They Will Not Disclose Its Contents To Any Third Party Or Otherwise Use This Presentation For Any Purpose Other Than An Evaluation Of MicroStrategy's Business Or Its Offerings. Reproduction or Distribution Is Prohibited.16 Intuitive Interface • Interactive dashboards • Visual Analytics • Build once, deploy anywhere Data Federation • Virtual model spanning multiple data sources High Performance • Push-down analytics • In-memory cube acceleration Flexible, Reliable, and Easy to Manage • High-efficiency object reuse • Powerful SDK • Comprehensive admin tools MicroStrategy Analytics Platform Comprehensive Analytics Suite for Big Data Web | Mobile | Portals | Office™ Data from Across the Enterprise Dashboards Statements Visual Discovery MicroStrategy Analytics Platform Reports Data Marts Relational Databases Cloudera Impala for Hadoop Multi Dimensional Sources
- 16. CONFIDENTIALThe Information Contained In This Presentation Is Confidential And Proprietary To MicroStrategy. The Recipient Of This Document Agrees That They Will Not Disclose Its Contents To Any Third Party Or Otherwise Use This Presentation For Any Purpose Other Than An Evaluation Of MicroStrategy's Business Or Its Offerings. Reproduction or Distribution Is Prohibited.17 MicroStrategy Visual Insight • Stunning visualizations • On-screen filtering • Speed-of-thought in memory database Common Use Cases • Interactive data exploration and root cause analysis • Dashboard creation • Self-service BI MicroStrategy Visual Insight Interactive Analysis, Drag-and-Drop to Build Intuitive Dashboards in Minutes Data Marts Relational Databases Multi Dimensional Sources Data from Across the Enterprise Cloudera Impala for Hadoop
- 17. CONFIDENTIALThe Information Contained In This Presentation Is Confidential And Proprietary To MicroStrategy. The Recipient Of This Document Agrees That They Will Not Disclose Its Contents To Any Third Party Or Otherwise Use This Presentation For Any Purpose Other Than An Evaluation Of MicroStrategy's Business Or Its Offerings. Reproduction or Distribution Is Prohibited.18 Combine Data from Multiple Federated Sources Take Big Data Out of Isolation Put Big Data analysis in context with information from federated data sources into one single dashboard User / Departmental Data Data Warehouse Appliances Hadoop Databases Relational Databases Multidimensional Databases Columnar Databases 1 1 2 2 3 2 & 3 Bring All Relevant Data to Decision Makers, No Matter Where It Resides
- 18. CONFIDENTIALThe Information Contained In This Presentation Is Confidential And Proprietary To MicroStrategy. The Recipient Of This Document Agrees That They Will Not Disclose Its Contents To Any Third Party Or Otherwise Use This Presentation For Any Purpose Other Than An Evaluation Of MicroStrategy's Business Or Its Offerings. Reproduction or Distribution Is Prohibited.19 Browsers Portals Enterprise Applications Web Email Email PDF Office DocumentsMobile AndroidiOSBlackBerry Build Once, Deploy Anywhere Makes Big Data Accessible to a Wider Business Audience Build once Deploy via any media 1 2
- 19. CONFIDENTIALThe Information Contained In This Presentation Is Confidential And Proprietary To MicroStrategy. The Recipient Of This Document Agrees That They Will Not Disclose Its Contents To Any Third Party Or Otherwise Use This Presentation For Any Purpose Other Than An Evaluation Of MicroStrategy's Business Or Its Offerings. Reproduction or Distribution Is Prohibited.20 From Big Data to Business Value MicroStrategy Delivers Insights on Big Data Faster Any and All Data World’s Most Intuitive Interface Benchmarking Projections Trend Analysis Data Summarization Relationship Analysis Relational Multidimensional Hadoop-based Structured Semi-Structured Unstructured Comprehensive Analytics Cloudera Impala for Hadoop Shortened time-to-value for data scientist Enables self-service for the business user
- 20. • Submit questions in the Q&A panel • Watch this webinar on-demand at http://cloudera.com • Follow Cloudera at @Cloudera • Follow MicroStrategy at @microstrategy • Thank you for attending! Learn more about the Cloudera MicroStrategy partnership http://cloudera.com/Microstrategy.htm l Download Impala http://cloudera.com/downloads Learn more about Impala at http://cloudera.com/impala
Editor's Notes
- More & Faster Value from Big DataProvides an interactive BI/Analytics experience on HadoopPreviously BI/Analytics was impractical due to the batch orientation of MapReduceEnables more users to gain value from organizational data assets (SQL/BI users)Makes more data available for analysis (raw data, multi-structured data, historical data)Removes delays from data migrationInto specialized analytical DBMSsInto proprietary file formats that happen to be stored in HDFSInto transient in-memory storesFlexibilityQuery across existing data in HadoopHDFS and HBaseAccess data immediately and directly in its native formatSelect best-fit file formatsUse raw data formats when unsure of access patterns (text files, RCFiles, LZO)Increase performance with optimized file formats when access patterns are known (Parquet, Avro)Run multiple frameworks on the same data at the same timeAll file formats are compatible across the entire Hadoop ecosystem – i.e. MapReduce, Pig, Hive, Impala, etc. on the same data at the same timeRun multiple frameworks on the same data at the same timeAll file formats are compatible across the entire Hadoop ecosystem – i.e. MapReduce, Pig, Hive, Impala, etc.Cost EfficiencyReduce movement, duplicate storage & computeData movement: no time or resource penalty for migrating data into specialized systems or formatsDuplicate storage: no need to duplicate data across systems or within the same system in different file formatsCompute: use the same compute resources as the rest of the Hadoop system – You don’t need a separate set of nodes to run interactive query vs. batch processing (MapReduce)You don’t need to overprovision your hardware to enable memory-intensive, on-the-fly format conversions10% to 1% the cost of analytic DMBSLess than $1,000/TBFull Fidelity AnalysisNo loss of fidelity from aggregations or conforming to fixed schemasIf the attribute exists in the raw data, you can query against it
- Our design strategy is to tightly integrate and couple Impala within the Hadoop system. Impala (and interactive SQL) is just another application that you bring to your data. It’s integrated with Hadoop’s existing security and resource management frameworks and is completely interoperable with existing data formats and processing engines.One pool of dataStorage platforms (HDFS & HBase)Open data formats (files & records)Shared across multiple processing frameworksOne metadata modelNo synchronization of metadata between 2 different systems (analytical DBMS and Hadoop)Same metadata used by other components within Hadoop itself (Hive, Pig, Impala, etc.)One security frameworkSingle model for all of HadoopDoesn’t require “turning off” any portion of native Hadoop securityOne set of system resourcesOne set of nodes – storage, CPU, memoryOne management consoleIntegrated resource managementScale linearly as capacity or performance needs grow
- Interactive BI/Analytics on more dataRaw, full fidelity data – nothing lost through aggregation or ETL/LTNew sources & types – structured/unstructuredHistorical dataAsking new questionsExploration and data discovery for analytics and machine learning – need to find a data set for a model, which requires lots of simple queries to summarize, count, and validate.Hypothesis testing – avoid having to subset and fit the data to a warehouse just to ask a single questionData processing with tight SLAsCost-effective platformMinimize data movementReduce strain on data warehouseQuery-able storageReplace production data warehouse for DR/active archiveStore decades of data cost effectively (for better modeling or data retention mandates) without sacrificing the capability to analyze
- Query-able archive w/full fidelityReplace production data warehouse for DR/active archiveStore decades of data cost effectively (for better modeling or data retention mandates) without sacrificing the capability to analyzeExample: Global financial services companyOffloaded DB25x performance improvement over HiveSaved 90% on incremental EDW spend
- Agile analytics is the hottest area in BI right now and the reason for that is that fundamentally, it is a revolutionary, new approach that enables anybody to very quickly find new insight, share that insight with everybody, and do it without the involvement of IT. That’s really a game changer considering that business intelligence has traditionally has required high levels of IT involvement in order to create dashboards, create reports, and then distribute those results out to large numbers of people. MicroStrategy Visual Insight is a pioneering new technology in agile analytics. It combines stunning visualizations with on-screen filtering and at the back end, there’s a very high speed in-memory database that powers everything for speed of thought interactivity between user and data. There are really two common use cases for this technology. The first is just pure visual data discovery, the idea of using visualizations rapidly shifting between different types of interactive visualizations to find information that you’re looking for, to isolate that information, to find trends, to look for root causes, to find the nuggets of insight into your data. The second major use case is dashboard creation. That’s the idea of bringing together multiple visualizations in an interactive dashboard that can then be published and shared with a variety of other people.