Hadoop World 2011: Integrating Hadoop with Enterprise RDBMS Using Apache Sqoop and Other Tools - Guy Harrison, Quest Software & Arvind Prabhakar, Cloudera
Emixa Mendix Meetup 11 April 2024 about Mendix Native development
Hadoop World 2011: Integrating Hadoop with Enterprise RDBMS Using Apache Sqoop and Other Tools - Guy Harrison, Quest Software & Arvind Prabhakar, Cloudera
1. November 2011 Apache Sqoop (Incubating) Integrating Hadoop with Enterprise RDBS – Part I Arvind Prabhakar (arvind at apache dot org) Apache Sqoop Committer and Software Engineer at Cloudera
20. Sqoop Connectors Oracle – Developed by Quest Software Couchbase – Developed by Couchbase Netezza – Developed by Cloudera Teradata – Developed by Cloudera SQL Server – Developed by Microsoft Microsoft PDW – Developed by Microsoft Volt DB – Developed by Volt DB 19
21. Sqoop Roadmap SQOOP-365: Proposal for Sqoop 2.0 https://issues.apache.org/jira/browse/SQOOP-365 Highlights Sqoop as a Service Connections as First Class Objects Role based Security 20
36. SQOOP extensibility SQOOP implements a generic approach to RDBMS/Hadoop data transfer But database optimization is highly platform specific Each RDBMS has distinct optimizations strategies For Oracle, optimization requires: Bypassing Oracle caching layers Avoiding Oracle optimizer meddling Exploiting Oracle metadata to balance mapper load
37. Reading from Oracle – default SQOOP ID > MAX/2 ID > 0 and ID < MAX/2 MAPPER MAPPER CACHE ORACLE SESSSION ORACLE SESSION RANGE SCAN RANGE SCAN Index block Index block Index block Index block Index block Index block ORACLE TABLE
41. Design goals Partition data based on physical storage By-pass Oracle buffering By-pass Oracle parallelism Do not require or use indexes Never read the same data block more than once Support Oracle datatypes
46. Working with the SQOOP framework SQOOP lets you concentrate on the RDBMS logic, not the Hadoop plumbing: Extend ManagerFactory (what to handle) Extend ConnManager (DB connection and metadata) For imports: Extend DataDrivenDBInputFormat (gets the data) Data allocation (getSplits()) Split serialization (“io.serializations” property) Data access logic (createDBRecordReader(), getSelectQuery()) Implement progress (nextKeyValue(), getProgress()) Similar procedure for extending exports
47. Extensions to native SQOOP MERGE functionality Update if exists, insert otherwise Hive connector Source defined as HQL query rather than HDFS directory Eclipse UI
48. Availability Apache licensed source available from : https://github.com/QuestSoftwareTCD/OracleSQOOPconnector Download from (Quest): http://www.quest.com/hadoop/ Download from (Cloudera): http://ccp.cloudera.com/display/SUPPORT/Downloads
49. Other SQOOP connectors Microsoft SQL Server: http://www.microsoft.com/download/en/details.aspx?id=27584 Teradata: https://ccp.cloudera.com/display/con/Cloudera+Connector+for+Teradata+User+Guide%2C+version+1.0-beta-u4 Microstrategy: https://ccp.cloudera.com/display/con/MicroStrategy+Free+Download+License+Agreement Nettezza: https://ccp.cloudera.com/display/con/Netezza+Free+Download+License+Agreement VoltDB: http://voltdb.com/company/blog/sqoop-voltdb-export-and-hadoop-integration
54. Hadapt Formally HadoopDB – Hadoop/Postgres hybrid Postgres servers on data nodes allow for accelerated (indexed) HIVE queries Extensions to the Hive optimizer http://www.hadapt.com/
55. Greenplum SQL based access to HDFS data via in-DB MapReduce http://www.greenplum.com/sites/default/files/EMC_Greenplum_Hadoop_DB_TB_0.pdf
56. Toad for Cloud Databases Federated SQL queries across Hive, Hbase, NoSQL, RDBMS
57. Conclusions RDBMS-Hadoop interoperability is key to Enterprise Hadoop adoption SQOOP provides a good general purpose framework for transferring data between any JDBC database and Hadoop We’d like to see it become a standard Each RDBMS offers distinct tuning opportunities, so optimized SQOOP extensions offer real value Hadoop-RDBMS integration projects are proliferating rapidly