This session explores how one organization built an integrated analytics platform by implementing Gluent to offload its Oracle enterprise data warehouse (EDW) data to Hadoop, and to transparently present native Hadoop data back to its EDW. As a result of its efforts, the company is now able to support operational reporting, OLAP, data discovery, predictive analytics, and machine learning from a single scalable platform that combines the benefits of an enterprise data warehouse with those of a data lake. This session includes a brief overview of the platform and use cases to demonstrate how the company has utilized the solution to provide business value.

1. Offload, Transform, and Present - the New World of Data Integration
Michael Rainey

2. • Michael Rainey - Technical Advisor
• Spreading the good word about gluent
products with the world
• Data Integration expertise
• Oracle ACE Director
• @mRainey
2
Introduction
we liberate enterprise data

3. 3
About Gluent
Globally distributed team with a deep
background in building high performance
enterprise applications and systems …
Now on a mission to
liberate enterprise data.
Gluent Data Platform
Advisor
Open Data Formats
Security
Orchestration
Access Offload Present

4. 3
About Gluent
Globally distributed team with a deep
background in building high performance
enterprise applications and systems …
2nd place in Strata+HadoopWorld
Startup Showcase 2017.
(Demo video at gluent.com)
Now on a mission to
liberate enterprise data.
Recognized by Gartner in
their Cool Vendors in Data
Management, 2017 report!
Download (no registration needed):
https://gluent.com/cool-vendor-2017/
Listed as one of the
“Top 10 Coolest Big Data
Startups of 2017” by CRN
gluent.com/gluent-recognized-in-
top-10-coolest-startups-of-2017-on-crn

5. 4
Gluent is running one-hour webinars each Wednesday in July beginning tomorrow, July 12.
See details below and register for your free spot today!
Apache Impala Internals
Speaker: Tanel Poder, Gluent
Wednesday, July 19 @ 12 PM CDT
gluent.com/event/gluent-webinar-apache-impala-internals-with-tanel-poder
Building an Analytics Platform with Oracle & Hadoop
Speakers: Gerry Moore & Suresh Irukulapati, Vistra Energy
Wednesday, July 26 @ 9 AM CDT
gluent.com/event/gluent-webinar-building-an-integrated-analytics-platform-with-oracle-and-hadoop
Gluent Webinars - JULY 2017

6. 5
Extract Transform Load (ETL)
Source 1
Source 2
Source X
EDW
Transform
Extract
Load

7. 6
ETL Examples - employee data
EDW
HR
Network
Facilities
Security
Training

8. 6
ETL Examples - employee data
EDW
HR
Network
Facilities
Security
Training

9. 6
ETL Examples - employee data
EDW
HR
Network
Facilities
Security
Training

10. 6
ETL Examples - employee data
EDW
HR
Network
Facilities
Security
Training
Employee

11. 6
ETL Examples - employee data
EDW
HR
Network
Facilities
Security
Training
Employee

12. 7
ETL Examples - IoT from set-top boxes
IoT

13. 7
ETL Examples - IoT from set-top boxes
IoT
EDW
CRM
CRM

14. 7
ETL Examples - IoT from set-top boxes
IoT
EDW
CRM
CRM
Enhanced
customer
data

15. 7
ETL Examples - IoT from set-top boxes
IoT
EDW
CRM
CRM
Enhanced
customer
data

16. 8
ETL Examples - data scientist pipelines

17. 8
ETL Examples - data scientist pipelines

18. 8
ETL Examples - data scientist pipelines

19. 8
ETL Examples - data scientist pipelines

20. 9
Typical ETL development - too much time spent on “E” and “L”
reader = new TransformingReader(reader)
.add(new SetCalculatedField("AvailableCredit",
"parseDouble(CreditLimit) - parseDouble(Balance)"))
.add(new ExcludeFields("CreditLimit",
"Balance"));
DataWriter writer = new
JdbcWriter(getJdbcConnection(), "dp_credit_balance")
.setAutoCloseConnection(true);
JobTemplate.DEFAULT.transfer(reader, writer);

21. 9
Typical ETL development - too much time spent on “E” and “L”
ETL Developer / Data Engineer
should spend their time on
Transformations!
reader = new TransformingReader(reader)
.add(new SetCalculatedField("AvailableCredit",
"parseDouble(CreditLimit) - parseDouble(Balance)"))
.add(new ExcludeFields("CreditLimit",
"Balance"));
DataWriter writer = new
JdbcWriter(getJdbcConnection(), "dp_credit_balance")
.setAutoCloseConnection(true);
JobTemplate.DEFAULT.transfer(reader, writer);

22. 10
We still need to move data!

23. • Data lake - enterprise data stored in raw form
• Data hub - centralized data store with standards, governance, and quality
11
Data lake or data hub?

24. • Sharing data without moving data
• Metadata is stored about each “source” database
• Queries are passed through a metadata
layer, which provides info about where the
data lives and how to translate the query
12
What about data federation?
Source 1
Source 2
Source X
Source 1
Source 2
Source X
App 1
App 2
App 1
App 2

25. • Sharing data without moving data
• Metadata is stored about each “source” database
• Queries are passed through a metadata
layer, which provides info about where the
data lives and how to translate the query
• But…
12
What about data federation?
Source 1
Source 2
Source X
Source 1
Source 2
Source X
App 1
App 2
App 1
App 2

26. • Sharing data without moving data
• Metadata is stored about each “source” database
• Queries are passed through a metadata
layer, which provides info about where the
data lives and how to translate the query
• But…
• We must recode applications!
• Data remains in RDBMS silos and are
limited to CPU/storage constraints
12
What about data federation?
Source 1
Source 2
Source X
Source 1
Source 2
Source X
App 1
App 2
App 1
App 2

27. 13
Three things we need the ability to do:
1. Offload data from Oracle to Hadoop
2. Load data from Hadoop to Oracle
3. Query Hadoop data in Oracle

28. • “Offload” - copy or move data from the relational database to Hadoop
• Why offload?
• Store data in a centralized location for access and sharing
• Use the distributed, parallel processing power of Hadoop for transformations
• Enable the use of “new world” technologies (Spark, Impala, etc)
• Why Hadoop?
• We can now afford to keep a copy of all enterprise data for data sharing reasons!
14
Offload, not Extract

29. 15
How to offload? There are many options
Tool Offload Data
Sqoop Yes
Oracle Loader for Hadoop
Oracle SQL Connector for HDFS
ODBC Gateway
Big Data SQL
Gluent Data Platform Yes

30. • Command line client used to bulk copy data from a relational database to
HDFS over JDBC connection
• Also works in reverse, HDFS to RDBMS
• Sqoop generates MapReduce jobs for the work
16
Bulk data load with Sqoop
sqoop import --connect jdbc:oracle:thin:@myserver:1521/MYDB1
--username myuser
--null-string ''
--null-non-string ''
--target-dir=/user/hive/warehouse/ssh/sales
--append -m1 --fetch-size=5000
--fields-terminated-by '','' --lines-terminated-by ''n''
--optionally-enclosed-by ''"'' --escaped-by ''"''
--split-by TIME_ID
--query ""SELECT * FROM SSH.SALES WHERE TIME_ID < DATE'1998-01-01'""

31. • Gluent Advisor helps determine which tables
and/or partitions are candidates for offload
• Offload options:
• Move or copy 100% of data
• Move “cold” partitions only
• Data can be kept in-sync with an incremental
offload
• Example: when a partition goes inactive (cold), it
can be offloaded to Hadoop
• Data can be updated from RDBMS to Hadoop
17
Offload data with Gluent Data Platform

32. • Gluent Advisor helps determine which tables
and/or partitions are candidates for offload
• Offload options:
• Move or copy 100% of data
• Move “cold” partitions only
• Data can be kept in-sync with an incremental
offload
• Example: when a partition goes inactive (cold), it
can be offloaded to Hadoop
• Data can be updated from RDBMS to Hadoop
17
Offload data with Gluent Data Platform
gluent.
80% - 20%
offload -x -t EDW.BALANCE_DETAIL_MONTHLY --less-than-value=2017-05-01

33. • Gluent Advisor helps determine which tables
and/or partitions are candidates for offload
• Offload options:
• Move or copy 100% of data
• Move “cold” partitions only
• Data can be kept in-sync with an incremental
offload
• Example: when a partition goes inactive (cold), it
can be offloaded to Hadoop
• Data can be updated from RDBMS to Hadoop
17
Offload data with Gluent Data Platform
gluent.
80% - 20%
offload -x -t EDW.BALANCE_DETAIL_MONTHLY --less-than-value=2017-05-01

34. 18
Three things we need the ability to do:
1. Offload data from Oracle to Hadoop
2. Load data from Hadoop to Oracle
3. Query Hadoop data in Oracle

35. 18
Three things we need the ability to do:
1. Offload data from Oracle to Hadoop
2. Load data from Hadoop to Oracle
3. Query Hadoop data in Oracle
Present

36. • Sqoop in reverse
• Reads HDFS files, converts to JDBC arrays and insert (or update) statements
19
The “L” of ETL - Loading the data
RDBMS

37. 20
Forget the “L”oad. Present the data to the RDBMS
RDBMS

38. 21
Load the data from Hadoop to RDBMS?
Present

39. 21
Load the data from Hadoop to RDBMS?
Tool Offload Data Load Data
Sqoop Yes Yes
Oracle Loader for Hadoop Yes
Oracle SQL Connector for HDFS Yes
ODBC Gateway Yes
Big Data SQL Yes
Gluent Data Platform Yes
Present
Present

40. 21
Load the data from Hadoop to RDBMS?
Tool Offload Data Load Data
Sqoop Yes Yes
Oracle Loader for Hadoop Yes
Oracle SQL Connector for HDFS Yes
ODBC Gateway Yes
Big Data SQL Yes
Gluent Data Platform Yes
Present
Present

41. • Oracle SQL Connector for HDFS
• Uses Oracle External Table to access delimited file or Hive table in Hadoop
• When queried, all data is read from Hadoop to Oracle, then processed
• Oracle to Hadoop over database links
• Create a database link from Oracle to Hadoop using the Impala (or Hive) ODBC
gateway
• Pushes filters to Hadoop but not grouping aggregates
• Big Data SQL
• Access Hive tables via an Oracle external table
• Predicate pushdown (not aggregations & joins)
22
Present data to Oracle

42. 23
Gluent Present
gluent.

43. 23
Gluent Present
gluent.
present -t "SH.SALES" -xv --target-name="SH.SALES_DIVISION_X"

44. 23
Gluent Present
gluent.
present -t "SH.SALES" -xv --target-name="SH.SALES_DIVISION_X"
Gluent Present can
pushdown predicates,
aggregations, and joins

45. 24
Three things we need the ability to do:
1. Offload data from Oracle to Hadoop
2. Load data from Hadoop to Oracle
3. Query Hadoop data in Oracle
Present

46. 25
How to query the data? There are many options
Tool Offload Data Load Data Allow Query
Data
Offload
Query
Parallel
Execution
Sqoop Yes Yes Yes
Oracle Loader for Hadoop Yes Yes
Oracle SQL Connector for HDFS Yes Yes Yes
ODBC Gateway Yes Yes Yes
Big Data SQL Yes Yes Yes Yes
Gluent Data Platform Yes Yes Yes YesPresent

47. 25
How to query the data? There are many options
Tool Offload Data Load Data Allow Query
Data
Offload
Query
Parallel
Execution
Sqoop Yes Yes Yes
Oracle Loader for Hadoop Yes Yes
Oracle SQL Connector for HDFS Yes Yes Yes
ODBC Gateway Yes Yes Yes
Big Data SQL Yes Yes Yes Yes
Gluent Data Platform Yes Yes Yes YesPresent

48. 26
Present Data From Anywhere To Anywhere
gluent.

49. 26
Present Data From Anywhere To Anywhere
gluent.

50. 26
Present Data From Anywhere To Anywhere
gluent.

51. 26
Present Data From Anywhere To Anywhere
gluent.

52. • A new approach…a new acronym?
27
Offload Transform Present (OTP?)

53. EDW
28
ETL Examples - employee data
EDW
HR
Network
Facilities
Security
Training
Employee

54. 29
Now using OTP - employee data
EDW
Employee

55. 29
Now using OTP - employee data
EDW
Employee

56. 29
Now using OTP - employee data
EDW
Employee

57. 29
Now using OTP - employee data
EDW
Employee

58. 30
ETL Examples - IoT from set-top boxes
IoT
EDW
CRM
CRM
Enhanced
customer
data

59. EDW
CRM
CRM
IoT
Enhanced
customer
data
30
ETL Examples - IoT from set-top boxes

60. 31
ETL Examples - data scientist pipelines

61. 31
ETL Examples - data scientist pipelines

62. 31
ETL Examples - data scientist pipelines

63. • No upfront massive project cost to move data around.
• Can happen in minutes / hours, not weeks / months
• Large corporations can move as fast as startups
• Data Engineers / ETL Developers can focus on transformations!
• A new approach to data sharing across enterprise
• Offload all data for simple access
• Takes a different mindset
32
OTP simplifies the process of data movement!

64. 33
Data Consolidation
SALES
(customer A)
Customers
Products Preferences
Promotions
Prices
SALES
(customer B)
Customers
Products Preferences
Promotions
Prices
SQL

65. 33
Data Consolidation
SALES
(customer A)
Customers
Products Preferences
Promotions
Prices
SALES
(customer B)
Customers
Products Preferences
Promotions
Prices
Gluent
Gluent
Offload
Customers A
Customers B
Raw data
SQL

66. 33
Data Consolidation
SALES
(customer A)
Customers
Products Preferences
Promotions
Prices
SALES
(customer B)
Customers
Products Preferences
Promotions
Prices
Gluent
Gluent
Offload Transform (optional)
Customers A
Customers B
->
Raw data
SQL

67. 33
Data Consolidation
SALES
(customer A)
Customers
Products Preferences
Promotions
Prices
SALES
(customer B)
Customers
Products Preferences
Promotions
Prices
Gluent
Gluent
Offload Transform (optional)
Customers A
Customers B
Customers
ALL
->
Raw data Consolidated
data
Union all, SQL,
Spark
SQL

68. 33
Data Consolidation
SALES
(customer A)
Customers
Products Preferences
Promotions
Prices
ALL
ALL
SALES
(customer B)
Customers
Products Preferences
Promotions
Prices
Gluent
Gluent
Offload Transform (optional) Present
Customers A
Customers B
Products ALL
Preferences ALL
Prices
Gluent
Customers
ALL
Promotions
SALES
(ALL)
-> ->
Raw data Consolidated
data
Customers
ALL
Union all, SQL,
Spark
SQLSQL
Virtualized
tables

69. 34
we liberate enterprise data
thank you!

70. 35
Gluent is running one-hour webinars each Wednesday in July beginning tomorrow, July 12.
See details below and register for your free spot today!
Apache Impala Internals
Speaker: Tanel Poder, Gluent
Wednesday, July 19 @ 12 PM CDT
gluent.com/event/gluent-webinar-apache-impala-internals-with-tanel-poder
Building an Analytics Platform with Oracle & Hadoop
Speakers: Gerry Moore & Suresh Irukulapati, Vistra Energy
Wednesday, July 26 @ 9 AM CDT
gluent.com/event/gluent-webinar-building-an-integrated-analytics-platform-with-oracle-and-hadoop
Gluent Webinars - JULY 2017