Pentaho ETL Tools

Business Intelligence
Applications -ETL
Microsoft SSIS 2008 & 2012, Talend Open Studio & Pentaho
Data Integration
By
Sunny Okoro

2
Contents
ETL Applications............................................................................................................................................3
Applications...................................................................................................................................................5
Database Platform ........................................................................................................................................8
Pentaho Enterprise Data Integration Edition.............................................................................................11
Talend Open Studio.....................................................................................................................................25
Microsoft SQL Server Integration Services 2008 ........................................................................................35
Oracle SQL Developer Data Migration Utility .............................................................................................48
Microsoft SQL Server Integration Services 2012 ........................................................................................56

3
ETL Applications
Pentaho Enterprise Data Integration Edition
Talend Open Studio

4
Microsoft SQL Server Integration Services 2012&2008
Oracle SQL Developer Data Migration Utility

5
Applications
MySQL Workbench
1.
2.
3.
4.
Oracle SQL Developer

6
Microsoft Excel 2010
Oracle JDeveloper

7
IBM Data Studio
Microsoft Visual Studio
Microsoft Visual Studios 2010

8
Database Platform
MYSQL 5.5
IBM DB2 10

9
Microsoft SQL Server 2012
Microsoft SQL Server 2008R2

10
Sybase SQL AnyWhere12
Oracle 11g

11
Pentaho Enterprise Data Integration Edition
Database Connection

13
1. Extract data from Sybase SQL Anywhere12 Adventure Works database server and dump the
data into a XML, text and Excel File. The task will contain five components which are Table Input,
Microsoft Excel Output, Text File Output, Add XML and XML Output as illustrated below.
Table Input
The table input contains the query to retrieve the data from the database. I have renamed the step
name to Customer Table to reflect the name of the data being extracted. The connection has a drop

14
down arrow which allows me to select from lists of established database or data source. I placed a limit
to increase the processing time.
The preview options allow developers to preview the data in real time. They can specify the number of
rows returned for preview. I am new to this ETL application and is much different compared to SSIS.
Text FileOuput

15
The text file output works like the Microsoft SSIS File component. The only difference is that the user
does not used variables to specify extensions to the file name as illustrated below to distinguish each
generated files from old files. The developer has to check one of the boxes to specify their criteria for
file naming standard.

16
The fields tab allows developer to format the destination output
Add XML
Allows developers to specify their encoding format, root xml element like auto, root etc and output
structure

17
XML Ouput
This components works like the text output file. The developer has to specify the filename and other
criteria.

18
Microsoft Excel Output
This component has multiple parts to format the data according to specifications that matches to Excel
destination.

23
Textile Output Extract
DueDate;OrderDate;ShipDate;CountryRegionCode;CUSTOMERID;FREIGHT;SUBTOTAL;TAXAMT;TOTALDUE;TerritoryID;TerritoryName
07/13/2001;07/01/2001;07/08/2001;US;29825;616.10;24643.94;1971.51;27231.55;5;Southeast
07/13/2001;07/01/2001;07/08/2001;US;29825;616.10;24643.94;1971.51;27231.55;5;Southeast
07/13/2001;07/01/2001;07/08/2001;US;29825;616.10;24643.94;1971.51;27231.55;5;Southeast
07/13/2001;07/01/2001;07/08/2001;US;29825;616.10;24643.94;1971.51;27231.55;5;Southeast
07/13/2001;07/01/2001;07/08/2001;US;29825;616.10;24643.94;1971.51;27231.55;5;Southeast
07/13/2001;07/01/2001;07/08/2001;US;29825;616.10;24643.94;1971.51;27231.55;5;Southeast
07/13/2001;07/01/2001;07/08/2001;US;29825;616.10;24643.94;1971.51;27231.55;5;Southeast
07/13/2001;07/01/2001;07/08/2001;US;29825;616.10;24643.94;1971.51;27231.55;5;Southeast
07/13/2001;07/01/2001;07/08/2001;US;29825;616.10;24643.94;1971.51;27231.55;5;Southeast
07/13/2001;07/01/2001;07/08/2001;US;29825;616.10;24643.94;1971.51;27231.55;5;Southeast
07/13/2001;07/01/2001;07/08/2001;US;29825;616.10;24643.94;1971.51;27231.55;5;Southeast
07/13/2001;07/01/2001;07/08/2001;US;29825;616.10;24643.94;1971.51;27231.55;5;Southeast
07/13/2001;07/01/2001;07/08/2001;US;29672;38.83;1553.10;124.25;1716.18;5;Southeast
07/13/2001;07/01/2001;07/08/2001;US;29672;38.83;1553.10;124.25;1716.18;5;Southeast
Results Abridged
Excel Output File Extract
SalesOrderNumber OrderDate DueDate ShipDate ProductID CustomerID SubTotal TaxAmt Freight TotalDue ProductName TerritoryName CountryRegionCode TerritoryGroup TerritoryID
SO43659 07/01/2001 07/13/2001 07/08/2001 776 29825 24643.94 1971.51 616.10 27231.55 Mountain-100 Black, 42 Southeast US North America 5
SO43659 07/01/2001 07/13/2001 07/08/2001 771 29825 24643.94 1971.51 616.10 27231.55 Mountain-100 Silver, 38 Southeast US North America 5
SO43659 07/01/2001 07/13/2001 07/08/2001 714 29825 24643.94 1971.51 616.10 27231.55 Long-Sleeve Logo Jersey, M Southeast US North America 5
SO43659 07/01/2001 07/13/2001 07/08/2001 716 29825 24643.94 1971.51 616.10 27231.55
Long-Sleeve Logo Jersey,
XL Southeast US North America 5
SO43659 07/01/2001 07/13/2001 07/08/2001 709 29825 24643.94 1971.51 616.10 27231.55 Mountain Bike Socks, M Southeast US North America 5
SO43659 07/01/2001 07/13/2001 07/08/2001 712 29825 24643.94 1971.51 616.10 27231.55 AWC Logo Cap Southeast US North America 5
SO43659 07/01/2001 07/13/2001 07/08/2001 711 29825 24643.94 1971.51 616.10 27231.55 Sport-100 Helmet, Blue Southeast US North America 5
Results Abridged
XML Output File Extract
<?xml version="1.0" encoding="UTF-8"?>
<Rows>
<Row><CUSTOMERID>29825</CUSTOMERID><OrderDate>07/01/01</OrderDate><DueDate>07/13/01</DueDate><ShipDate>07/08/01</ShipDate><CountryRegionCode>US</CountryRegionCode><Sales
OrderNumber>SO43659</SalesOrderNumber><ProductID>776</ProductID><ProductName>Mountain-100 Black, 42</ProductName><FREIGHT> 616.0984</FREIGHT><TAXAMT> 1971.5149</TAXAMT><SUBTOTAL>
24643.9362</SUBTOTAL><TOTALDUE> 27231.5495</TOTALDUE><TerritoryID> 5</TerritoryID><TerritoryName>Southeast</TerritoryName><TerritoryGroup>North America</TerritoryGroup></Row>

24
OrderNumber>SO43659</SalesOrderNumber><ProductID>771</ProductID><ProductName>Mountain-100 Silver, 38</ProductName><FREIGHT> 616.0984</FREIGHT><TAXAMT> 1971.5149</TAXAMT><SUBTOTAL>
OrderNumber>SO43659</SalesOrderNumber><ProductID>714</ProductID><ProductName>Long-Sleeve Logo Jersey, M</ProductName><FREIGHT> 616.0984</FREIGHT><TAXAMT> 1971.5149</TAXAMT><SUBTOTAL>
Results Abridged
I am new to this application, I would later discover how to resolve the date mapping for XML.

25
Talend Open Studio
Database Systems: MYSQL 5.6
Task:
Extract data from the database with customer information and payment history which will include sum
and countinformation of each amount paid.
Step 1
Create view in the database.

26
Step 2
Using Talend Open Studio design the ETL job with components that will connect to the database and
extract the data
tMYSQLInput Component
Contains the database connection to MYSQL 5.6
Database Connection

27
Guess schema allows the mapping of the columns to the database source.
The SQL Builder allows the developer to view SQL Statements.

28
The connector between Tmysqlinput_1 and the aggregation component allows developers to create
new columns that would hold the aggregation data as illustrated below.

29
tAggregaateSoredRow_1 Component
Contains the aggregation details of the ETL process

30
The connector between the destinations contains the columns required to complete the mapping
process after the aggregation process. Using the Edit Schema maps the columns respectively as
illustrated below.

31
tFileOutputExcel_1 Component

32
This component allows developers to specify requirements for the Excel destination including column
mapping using edit schema as illustrated below.
Customer sales file extract

33
customer_id store_id Cust_Name address address2 phone city postal_code country amount payment_date Sum_Payment Count_Payment
2 0 $0.00 $128.73 27
4 0 $0.00 $81.78 22
1 0 $0.00 $118.68 32
3 0 $0.00 $135.74 26
2 1 JOHNSON,PATRICIA 1121 Loja Avenue 838635286649 San Bernardino 17886 United States $4.99 27-05-2005 $0.00
4 2 JONES,BARBARA 1566 Inegl Manor 705814003527 Myingyan 53561 Myanmar $4.99 15-06-2005 $0.00
1 1 SMITH,MARY 1913 Hanoi Way 28303384290 Sasebo 35200 Japan $2.99 25-05-2005 $0.00

34
3 1 WILLIAMS,LINDA 692 Joliet Street 448477190408 Athenai 83579 Greece $1.99 27-05-2005 $0.00
Result Abridged

35
Microsoft SQL Server Integration Services 2008
Database Systems: Oracle 11g, Microsoft SQL Server 2008 R2 and 2012
Task
Create view to contain customer information relating to sales history (sales order number, tax, amount
paid etc). The data will be dumped into an Oracle Database using Oracle Client for Microsoft Visual
Studio, text file and SQL Server 2012.

36
View Creation
View Verification
SSIS Package
Few technical problems with Microsoft Visual Studio 2010 forced me to switch to Microsoft Visual
Studio 2008 for Business Intelligence Development.

37
Microsoft SSIS Package
Execution Phase
Design Stage
SQL Server Database 2008 R2
This component contains the database configuration needed to extract the data

38
FlatFile Destination Component
This components contains all the configuration details for the flat file destination

39
Multicast Component
This component acts like a copy machine that directs the output data from the aggregation component
to various components.
Derived Column Component
This component allows developers to add new columns to the package. In this task, a created date
column would be added to output that would allow users to know when a particular data is loaded into
the database.

40
DataConversion Component
This component allows developers to map the Input and Output columns to a specific data type and
sizes based on the database specification. Both the Oracle data conversion were identical with exception
of the created date for SQL Server 2012.
Database Destination Components
This component contains the database destination. Oracle and SQL Server uses different connection
parameters to establish connection with exception of dbo prefix in SQL Server. That is the major

41
difference between Oracle and SQL Server Database Destination components. I have maintained the
same names for the tables in both databases.

42
Variables and File Naming Standard

43
Each time the variables passed in the expression will be used to create the file using the connection
string as illustrated. I did not add any constraints to the connector or error handling to this package
since is not a production based package
SISS Execution
SISS can be executed using Microsoft Visual Studio, SQL Agent, Command Line or Execute Page Utility.
Execute Page Utility

45
Validation
Oracle 11G
The file is created per design with the date encoded as illustrated below
Data Validation
Text File Extract
SQL Server 2012

46
Customer_ID,Cust_Name,ProductKey,SalesOrderNumber,OrderQuantity,UnitPrice,TotalProductCost,Sal
esAmount,TaxAmt,Freight
11003,Christy,Zhu,346,SO43701,1,3399.99,1912.15,3399.99,272.00,85.00
14501,Ruben,Prasad,336,SO43700,1,699.10,413.15,699.10,55.93,17.48
21768,Cole,Watson,310,SO43697,1,3578.27,2171.29,3578.27,286.26,89.46
25863,Sydney,Wright,346,SO43699,1,3399.99,1912.15,3399.99,272.00,85.00
28389,Rachael,Martinez,346,SO43698,1,3399.99,1912.15,3399.99,272.00,85.00
11005,Julio,Ruiz,351,SO43704,1,3374.99,1898.09,3374.99,270.00,84.37
11011,Curtis,Lu,344,SO43705,1,3399.99,1912.15,3399.99,272.00,85.00
16624,Albert,Alvarez,310,SO43703,1,3578.27,2171.29,3578.27,286.26,89.46
27645,Colin,Anand,311,SO43702,1,3578.27,2171.29,3578.27,286.26,89.46
16351,Martha,Xu,313,SO43709,1,3578.27,2171.29,3578.27,286.26,89.46
16517,Katrina,Raji,314,SO43710,1,3578.27,2171.29,3578.27,286.26,89.46
20042,Brad,Deng,330,SO43708,1,699.10,413.15,699.10,55.93,17.48
27616,Emma,Brown,312,SO43707,1,3578.27,2171.29,3578.27,286.26,89.46
27621,Edward,Brown,312,SO43706,1,3578.27,2171.29,3578.27,286.26,89.46
Results Abridged

47
SELECT*
FROMmaster.dbo.CustomerSales
Results Abridged
SELECT "Customer_ID",
"Cust_Name",
"ProductKey",
"SalesOrderNumber",
"OrderQuantity",
"UnitPrice",
"TotalProductCost",
"SalesAmount",
"TaxAmt",
"Freight"
FROM "CustomerSales"
Results Abridged
Createad_Date Customer_ID Cust_Name ProductKey SalesOrderNumber OrderQuantity UnitPrice TotalProductCost SalesAmount TaxAmt Freight
5/12/2013
20:24
17440 Richard,Cox 477 SO63903 1 4.99 1.87 4.99 0.4 0.12
5/12/2013
20:24
18130 Kaitlyn,Mitchell 477 SO63880 1 4.99 1.87 4.99 0.4 0.12
5/12/2013
20:24
18221 Marc,Alvarez 477 SO63887 1 4.99 1.87 4.99 0.4 0.12
5/12/2013
20:24
18933 Kelli,Huang 477 SO63904 1 4.99 1.87 4.99 0.4 0.12
5/12/2013
20:24
20645 Xavier,Stewart 477 SO63941 1 4.99 1.87 4.99 0.4 0.12
5/12/2013
20:24
21166 Tamara,Yuan 477 SO63936 1 4.99 1.87 4.99 0.4 0.12
5/12/2013
20:24
21550 Kara,Deng 477 SO63910 1 4.99 1.87 4.99 0.4 0.12
5/12/2013
20:24
11724 Jason,Carter 477 SO63965 1 4.99 1.87 4.99 0.4 0.12
5/12/2013
20:24
12599 Jillian,Fernandez 477 SO63981 1 4.99 1.87 4.99 0.4 0.12
5/12/2013
20:24
13858 Melanie,Richardson 477 SO63969 1 4.99 1.87 4.99 0.4 0.12
5/12/2013
20:24
14671 Cristina,Lal 477 SO63994 1 4.99 1.87 4.99 0.4 0.12
5/12/2013
20:24
15063 Kari,Malhotra 477 SO63955 1 4.99 1.87 4.99 0.4 0.12
5/12/2013
20:24
15369 Connor,Perez 477 SO63980 1 4.99 1.87 4.99 0.4 0.12
5/12/2013
20:24
15931 Micah,Xu 477 SO63958 1 4.99 1.87 4.99 0.4 0.12
Customer_ID Cust_Name ProductKey SalesOrderNumber OrderQuantity UnitPrice TotalProductCost SalesAmount TaxAmt Freight
11245 Ricky,Vazquez 477 SO51178 1 4.99 1.87 4.99 0.4 0.12
16313 Hailey,Russell 477 SO51180 1 4.99 1.87 4.99 0.4 0.12
11006 Janet,Alvarez 477 SO51198 1 4.99 1.87 4.99 0.4 0.12
11448 Kyle,Patterson 477 SO51197 1 4.99 1.87 4.99 0.4 0.12
12390 Denise,Martinez 477 SO51191 1 4.99 1.87 4.99 0.4 0.12
18906 Adrienne,Gutierrez 477 SO51196 1 4.99 1.87 4.99 0.4 0.12
21440 Ian,Morgan 477 SO51199 1 4.99 1.87 4.99 0.4 0.12
11117 April,Deng 477 SO51216 1 4.99 1.87 4.99 0.4 0.12

48
Oracle SQL Developer Data Migration Utility
In this task I would migrate MYSQL Database Sakila and DB2GSDB Database into Oracle database. I have
created the repository named Oracle System using Oracle System account. The actual data will be stored
in new schema named sakila default password of sakila for MYSQL migration. For IBM DB2 each schema
will be created with the schema name as the user name and password for that schema. I would connect
to the Oracle GOSALES schema for demonstration .

51
Repository explorer shows the three migration projects including the DB2 and MYSQL
Data Validation- MYSQL Migragation
To connect to the new created schema sakila I have to use the connection parameters as illustrated
below.

52
MySQL Workbench

53
Data Validation- IBM DB2 Migragation
IBM Data Studio

54
File Location
Oracle SQL Developer generates several output files which can be used to reconstruct the database as
illustrated below
MYSQL Output Migragation File

56
Microsoft SQL Server Integration Services 2012

57
Table Creation
Table verification

58
Stored Procedures Creation
Stored Procedure Verification

59
SSIS Package Design and Implementation
CustSales Package SSIS 2008 Format

60
Solution
This package will create reports associated with sales.
The state report will contain total sales across the states associated with a country code provided in the
parameter to the stored procedure. This report will contain total sales, average, maximum and
minimum results of each state
The customer report will contains aggregations associated with each customer of total sales, average,
maximum and minimum of their sales history.
The yearly report will contain aggregations associated with sales of specific year including total sales,
average, maximum and minimum of sales history.
The customer sales pivot report will contain total sales of each customer sales information in pivot
format associated with a yearly format.
Each reports generated are stored in oracle database and text files. The text files extensions include the
file name, date and country code provided in the variable to the stored procedure.
Design
The package was designed using Microsoft 2008 Business Intelligence Development and was later
converted to the 2012 SSIS format using Microsoft Visual Studio 2010 as illustrated below

61
CustSales Package SSIS 2012 Version
Design Details
The stored procedure is executed using the value received from the variable which is mapped to the
stored procedure parameter to retrieve specific data.

68
Pivot Report View
Verification

72
Execution with SQL Agent
Project Conversion Wizard
- Allows developers to store packages in SQL Server Database

73
Integration Service Deployment

74
SQL Server Agent Configuration

Pentaho ETL Tools

Empfohlen

Empfohlen

Weitere ähnliche Inhalte

Andere mochten auch

Andere mochten auch (20)

Ähnlich wie Pentaho ETL Tools

Ähnlich wie Pentaho ETL Tools (20)

Mehr von Sunny U Okoro

Mehr von Sunny U Okoro (20)

Kürzlich hochgeladen

Kürzlich hochgeladen (20)

Pentaho ETL Tools