These are the slides used by Kazuki Uehara, who is part of the team at NTT which migrated around 30 database from Oracle to PostgreSQL.

1. Copyright©2016 NTT corp. All Rights Reserved.
Migration from Oracle to PostgreSQL
- The problems and the solutions -
Kazuki Uehara
NTT OSS Center
March 19, 2016
Copyright(c)2016 NTT Corp. All Rights Reserved.

2. 2Copyright©2016 NTT corp. All Rights Reserved.
• Self Introduction
• Introduction of Today's topics
• Problems about Database migration.
• OSS Products that support the Database
migration
• Conclusion
Agenda

3. 3Copyright©2016 NTT corp. All Rights Reserved.
• Kazuki Uehara
• From Japan
• Hobby
• Travelling by bicycle
• Taking pictures
• Work
• technical support
• technical consulting
• functional verification and performance evaluation
• Products
• PostgreSQL
• pgpool-II、Slony-I
Who am I?

4. 4Copyright©2016 NTT corp. All Rights Reserved.
• Who we are?
• NTT（Nippon Telegraph and Telephone Corporation）
• National flagship carrier in Japan
• What NTT OSS Center is doing？
• Promotes the adoption of OSS by the group companies
• Total support
• support desk, Introduction support, Product maintenance
• R&D
• developing OSS and related tools with the communities
• Deals with about 60 OSS products.
About us
NTT group
subsidiary
about 900 companies
NTT
NTT OSS Center

5. 5Copyright©2016 NTT corp. All Rights Reserved.
• The world's most advanced OSS DBMS.
• Continued featuere/performance
improvement by the community.
We involve in PostgreSQL
600 systems
for the last seven years.
Number of adoptions of PostgreSQL
in the group companies
year

6. 6Copyright©2016 NTT corp. All Rights Reserved.
PostgreSQL vs Oracle
PostgreSQL 9.5 Oracle 12c
SQL ○ (ISO SQL 2011) ○ (ISO SQL 2011)
stored procedure ○ (PL/pgsql,Java,perl,...) ○ (PL/sql,Java,...)
trigger ○ ○
Online Backup ○ ○
partitionig ○ ○
Replication
○
(Synchronous/Asynchronous)
○
(Synchronous/Asynchronous)
HA Cluster ○ (pacemaker,pgpool-II,...) ○ (VCS, MSCS,...)
clustered systems with
shared disk storage
× ○ (RAC)
License BSD
Named User Plus License
/ Processor Lincense
License Fee ○ free of charge × Compensation
• No big difference.
• You can use PostgreSQL at many systems.

7. 7Copyright©2016 NTT corp. All Rights Reserved.
• Focus on Database migration technique from
Oracle to PostgreSQL.
• What is system migration?
• Migration is the process of transferring the system and
data to another environment.
• Three categories of system migration:
• rehost ・・・To replace only platform hardware
• rewrite ・・・To replace OS and programing languages
• rebuild ・・・Remake the entire system
• Database migration is needed in 'rewrite' or 'rebuild'.
Today's topics
1. Problems in the Database migration
2. How you can solve them

8. 8Copyright©2016 NTT corp. All Rights Reserved.
• Introduction
• Today's topic
• Problem about Database migration
• Why you need Database migration?
• The items to be considered for the Database migration
• Issues on Database migration
• OSS Products that support the DB migration
• Conclusion

9. 9Copyright©2016 NTT corp. All Rights Reserved.
• Your system is obsolete
• Support of HW expired.
• Maintenance costs rise due to aging of equipment.
• Your Database is expensive or poorly operating
• You want to cut the license cost
• You want to improve performance of middleware.
• You can downsize HW.
• You want to use new features.
• You found your commercial Database was over spec
• didn't use the proper functionalities of the commercial
product.
Why you need Database migration?

10. 10Copyright©2016 NTT corp. All Rights Reserved.
The items to be considered for the Database
migration
AP Server
1. Logical design of DB
• ER Diagram
2. Physical design of DB
• arrangement of the data
3. Data
• dump, restore
4. Operation procedures
• maintenance tool (backup,
batch)
5. SQL used in application
• the dedicated SQL of
commercial product.
DB Server
DB
（Oracle → PostgreSQL）
Logical
design
Physical
design
application
SQL a
SQL b
…
data
Client /
Web
DataBase
Administrator
Operationnal
procedures
entity
attribute
table
index
columnrelationship
…
…
1 2
5
3
4

11. 11Copyright©2016 NTT corp. All Rights Reserved.
The items to be considered for the Database
migration
AP Server
1. Logical design of DB
• ER Diagram
2. Physical design of DB
• arrangement of the data
3. Data
• dump, restore
4. Operation procedures
• maintenance tool (backup,
batch)
5. SQL used in application
• the dedicated SQL of
commercial product.
DB Server
DB
（Oracle → PostgreSQL）
Logical
design
Physical
design
application
SQL a
SQL b
…
data
Client /
Web
DataBase
Administrator
Operationnal
procedures
entity
attribute
table
index
columnrelationship
…
…
1 2
5
3
4

12. 12Copyright©2016 NTT corp. All Rights Reserved.
1. It will require significant cost to estimate
for the migration of SQL.
• The migration cost go up when we spend too much
time on estimation.
• You can provide the rough estimate in order to reduce
cost.
2. It will require significant cost to modify
incompatible SQL.
• You have to find incompatible SQL from a lot of
sources. In addition, you have to consider for each how
should you modify.
What is the problems?
The risk of losing profits on migration exists.

13. 13Copyright©2016 NTT corp. All Rights Reserved.
• This is sample source for Oracle.
• If you use it on PostgreSQL, what should you modify?
• There are three places in this source that require to
modify.
What kind of SQL should you modify?
import java.sql.*;
import java.util.*;
public class test02 {
String sqlString = "DELETE mytbl";
public ResultSet testMethod() throws SQLException {
ResultSet rs = stmt.execute(sqlString);
ResultSet rs2= stmt.executeQuery("SELECT sysdate FROM dual");
return rs;
}
}

14. 14Copyright©2016 NTT corp. All Rights Reserved.
• This is sample source for Oracle.
• If you use it on PostgreSQL, what should you modify?
• There are three places in this source that require to
modify.
What kind of SQL should you modify?
import java.sql.*;
import java.util.*;
public class test02 {
String sqlString = "DELETE mytbl";
public ResultSet testMethod() throws SQLException {
ResultSet rs = stmt.execute(sqlString);
ResultSet rs2= stmt.executeQuery("SELECT sysdate FROM dual");
return rs;
}
}
DELETE statement requires FROM clause in PostgreSQL.

15. 15Copyright©2016 NTT corp. All Rights Reserved.
• This is sample source for Oracle.
• If you use it on PostgreSQL, what should you modify?
• There are three places in this source that require to
modify.
What kind of SQL should you modify?
import java.sql.*;
import java.util.*;
public class test02 {
String sqlString = "DELETE mytbl";
public ResultSet testMethod() throws SQLException {
ResultSet rs = stmt.execute(sqlString);
ResultSet rs2= stmt.executeQuery("SELECT sysdate FROM dual");
return rs;
}
}
DELETE statement requires FROM clause in PostgreSQL.
PostgreSQL doesn't have sysdate.

16. 16Copyright©2016 NTT corp. All Rights Reserved.
• This is sample source for Oracle.
• If you use it on PostgreSQL, what should you modify?
• There are three places in this source that require to
modify.
What kind of SQL should you modify?
import java.sql.*;
import java.util.*;
public class test02 {
String sqlString = "DELETE mytbl";
public ResultSet testMethod() throws SQLException {
ResultSet rs = stmt.execute(sqlString);
ResultSet rs2= stmt.executeQuery("SELECT sysdate FROM dual");
return rs;
}
}
DELETE statement requires FROM clause in PostgreSQL.
PostgreSQL doesn't have sysdate.
It doesn't exist DUAL table in PostgreSQL

17. 17Copyright©2016 NTT corp. All Rights Reserved.
• Introduction
• Today's topic
• Problem about Database migration
• OSS Products that support the Database
migration
• db_syntax_diff
• orafce
• Case study
• Conclusion

18. 18Copyright©2016 NTT corp. All Rights Reserved.
• Two OSS products as solutions.
• db_syntax_diff
• This tool was made by us as migration supporting tool.
• Extracts incompatible SQL from application's source of
Oracle.
• Using this tool, anyone can be easily review source of
application.
• We can review in a relatively short time even for large
scale systems.
• orafce
• This is contrib module for PostgreSQL.
• It is an emulation tool for PostgreSQL to use
compatibility functions and operators with Oracle
RDBMS.
To solve the problem …

19. 19Copyright©2016 NTT corp. All Rights Reserved.
An overall outline of db_syntax_diff
ｓｒｃ
db_syntax_diff
XML Output file
• This file is XML format.
Outline of processing
• to do parsing using original parser.
• draw a comparison between the results of
parsing and the contents of dictionary file.
dictionary
file What is dictionary file?
• The list of incompatible 'SQL'.
• You can modify this file.
ｓｒｃｓｒｃ
Input files
• You can input single file or directory.
• C source（ProC）, Java source, JSP source, SQL file

20. 20Copyright©2016 NTT corp. All Rights Reserved.
• Install the required package using the yum.
• expand the files got from GitHub.
• set the environment variable.
How to use db_syntax_diff 1/4
# yum install perl perl-XML-SAX.noarch xalan-j2 perl-Parse-Yapp
perl-XML-NamespaceSupport.noarch perl-XML-LibXML.x86_64
$ tar -xvf db_syntax_diff.tar.gz
$ vi ~/.bash_profile
export CLASSPATH=/usr/share/java/xalan-j2.jar:$CLASSPATH
export CLASSPATH=/usr/share/java/xalan-j2-serializer.jar:$CLASSPATH
export PATH=$HOME/db_syntax_diff/src:$PATH
export PERL5LIB=$HOME/db_syntax_diff/src/lib
$ source ~/.bash_profile
※If you use xalan-java 2.7.1 later, you have to set CLASSPATH for serializer.jar.

21. 21Copyright©2016 NTT corp. All Rights Reserved.
How to use db_syntax_diff 2/4
$ db_syntax_diff.pl --help
db_syntax_diff version 2.0
The SQL analyzer for converting to PostgreSQL.
Usage:db_syntax_diff.pl [-e encodingname][-d definition-file]
[-i inputsourcedir[,suffix1[,suffix2]...] ]
[-o outfile][-f filterword][-m modename]
[-I includedir[,includedir1[,includedir2]...]][-h][-v [loglevel]]
[inputfilename]...
-e encodingname, --encoding=encodingname File encoding. The value which can be
specified is "utf8" and "shiftjis" and "eucjp". [default: eucjp]
-d definition-file, --define=definition-file Definition-file file name.
-i inputsourcedir, --input=inputsourcedir Input-file directry.
-o outfile, --output=outfile Output-file file name. [default: STDOUT]
-f filterword, --filter=filterword Pattern filterword. The value which can be specified is
"oracle8" and "oracle8i". [default: ALL]
-m modename, --mode=modename File type of source file. The value which can be
specified is "c" and "sql" and "cpp" and "java". [default: java]
-I includedir, --Include=includedir Add the directory includedir to the list of directories
to be searched for header files. [default: ./]
-h, --help Print usage and exit.
-v, --verbose Print progress of the practice to STDERR. The value which can be
specified is "1" and "3" and "5" and "7". [default: none]
inputfilename Input-file file name.

22. 22Copyright©2016 NTT corp. All Rights Reserved.
• I introduce the result as a simple example.
• target file : sample.java
• Run this command.
How to use db_syntax_diff 3/4
$ db_syntax_diff.pl -m java -e utf8 ~/tmp/sample.java -o ~/result.xml
import java.sql.*;
import java.util.*;
public class test02 {
String sqlString = "DELETE mytbl";
public ResultSet testMethod() throws SQLException {
ResultSet rs = stmt.execute(sqlString);
ResultSet rs2= stmt.executeQuery("SELECT sysdate FROM dual");
return rs;
}
}

23. 23Copyright©2016 NTT corp. All Rights Reserved.
• This is a part of result.
How to use db_syntax_diff 4/4
<?xml version="1.0" encoding="UTF-8"?>
<REPORT file_number="1" start_time="2016/2/25 19:12:06" finish_time="2016/2/25 19:12:07">
<METADATA>
<PARAMETER>-m java -e utf8 /home/uehara/tmp/sample.java -o /home/uehara/result.xml</PARAMETER>
</METADATA>
<FILE name="/home/uehara/tmp/sample.java" string_item_number="3" report_item_number="4" item_number="7">
<STRING_ITEM line="testMethod:sqlString:7">
<TARGET>DELETE mytbl</TARGET>
</STRING_ITEM>
・・・
<REPORT_ITEM id="SQL-107-008" type="SQL" level="LOW2">
<SOURCE>
<CLASS>test02</CLASS>
<METHOD>testMethod</METHOD>
<LINE>5</LINE>
<COLUMN>7</COLUMN>
<VARIABLE>sqlString</VARIABLE>
</SOURCE>
<STRUCT>!(?:[^¥w¥d_]|¥A)FROM(?:[^¥w¥d_]|¥z)</STRUCT>
<TARGET>DELETE mytbl</TARGET>
<MESSAGE>FROMの省略は未サポートです。</MESSAGE>
</REPORT_ITEM>
<REPORT_ITEM id="SQL-119-706" type="SQL" level="LOW1">
・・・
Which file?
What kind of incompatible SQL?
Simple report is output.
What type? How difficult?
What line/column?
What SQL statement?

24. 24Copyright©2016 NTT corp. All Rights Reserved.
dictionary
file
Specifications of the wrapper tool
ｓｒｃ
db_syntax_diff
Output files
• This file is XML format.
• It is aggregated in 4 patterns.
Outline of processing
• This tool operates db_syntax_diff.
• In addition, This tool makes CSV files
from XML file.
ｓｒｃｓｒｃ
Input files
• You can input single file or directory.
csv
XML
db_syntax_diff_wrapper
configuration
file

25. 25Copyright©2016 NTT corp. All Rights Reserved.
• result.csv
• file path of target file
• Number of line
• Number of columns
• ID (db_syntax_diff)
The results of wrapper tool
• The classification
• Difficulty of migration
• simple report
• target query
• Middle
• You can't simple migrate.
• High
• migration is very difficult.
• Low1
• It's only necessary to delete or replace.
• Low2
• You can't replace, but migration is easy.
Difficulty

26. 26Copyright©2016 NTT corp. All Rights Reserved.
• You can reduce the modification cost of SQL.
• The number of incompatible SQL decrease by using
orafce.
• Installation Instructions (Only 3 steps)
1. You can get a RPM file from PostgreSQL.org.
• http://yum.postgresql.org/9.5/redhat/rhel-7-
x86_64/repoview/orafce95.html
• Latest version(3.2.1) has been published.
2. Install orafce RPM.
3. After, you just run a query "CREATE EXTENSION
orafce".
How to use orafce
# rpm -ivh orafce95-3.2.1-1.rhel7.x86_64.rpm
Updating / installing...
1:orafce95-3.2.1-1.rhel7 ################################# [100%]

27. 27Copyright©2016 NTT corp. All Rights Reserved.
A simple example
-bash-4.2$ psql postgres
psql (9.5.1)
Type "help" for help.
postgres=# CREATE TABLE bar(i int,v VARCHAR2(20));
ERROR: type "varchar2" does not exist
LINE 1: CREATE TABLE bar(i int,v VARCHAR2(20));
^
postgres=#postgres=# CREATE EXTENSION orafce;
CREATE EXTENSION
postgres=#postgres=# CREATE TABLE bar(i int,v VARCHAR2(20));
CREATE TABLE
postgres=#postgres=# ¥d bar
Table "public.bar"
Column | Type | Modifiers
--------+--------------+-----------
i | integer |
v | varchar2(20) |
postgres=#

28. 28Copyright©2016 NTT corp. All Rights Reserved.
A simple example
-bash-4.2$ psql postgres
psql (9.5.1)
Type "help" for help.
postgres=# CREATE TABLE bar(i int,v VARCHAR2(20));
ERROR: type "varchar2" does not exist
LINE 1: CREATE TABLE bar(i int,v VARCHAR2(20));
^
postgres=#postgres=# CREATE EXTENSION orafce;
CREATE EXTENSION
postgres=#postgres=# CREATE TABLE bar(i int,v VARCHAR2(20));
CREATE TABLE
postgres=#postgres=# ¥d bar
Table "public.bar"
Column | Type | Modifiers
--------+--------------+-----------
i | integer |
v | varchar2(20) |
postgres=#
PostgreSQL doesn't have
data type 'VARCHAR2' .

29. 29Copyright©2016 NTT corp. All Rights Reserved.
A simple example
-bash-4.2$ psql postgres
psql (9.5.1)
Type "help" for help.
postgres=# CREATE TABLE bar(i int,v VARCHAR2(20));
ERROR: type "varchar2" does not exist
LINE 1: CREATE TABLE bar(i int,v VARCHAR2(20));
^
postgres=#postgres=# CREATE EXTENSION orafce;
CREATE EXTENSION
postgres=#postgres=# CREATE TABLE bar(i int,v VARCHAR2(20));
CREATE TABLE
postgres=#postgres=# ¥d bar
Table "public.bar"
Column | Type | Modifiers
--------+--------------+-----------
i | integer |
v | varchar2(20) |
postgres=#
PostgreSQL doesn't have
data type 'VARCHAR2' .
You can use
data type 'VARCHAR2'
in PostgreSQL.

30. 30Copyright©2016 NTT corp. All Rights Reserved.
IT infrastructureClient
SW
Web/AP/DB
Job Management/Backup
Other system
Server Enclosure
Case study 1/3
• Billing system
• A managing system of business contract information
• Migration from commercial product to RHEL / Apache /
mod_jk / JBoss EAP / PostgreSQL
intracompany
network
intracompany
network
target number of files Number of lines
SQL/DDL
about 740 files about 250KLJava
Pro*C
Scale of Database migration

31. 31Copyright©2016 NTT corp. All Rights Reserved.
• We judged that there are not big problem in
database migration by using db_syntax_diff.
• We extracted 10000 pieces of incompatible SQL.
• But difficulty of alomost the whole incompatible SQL
was Low1 or Low2.
Case study 2/3
# difficulty SQL/DDL Java Pro*C
1 Low1 4,186 1 11
2 Low2 5,361 798 94
3 Middle 20 0 0
4 High 0 0 0
total 9,567 799 105
tool's output
• Low1
• It's only necessary to
delete or replace.
• Low2
• You can't replace, but
migration is easy.
• Middle
• You can't simple migrate.
• High
• migration is very difficult.

32. 32Copyright©2016 NTT corp. All Rights Reserved.
• We can reduce the number of incompatible
SQL by orafce.
• In this case, it can reduce to 7000.
• Based on our own experience, PostgreSQL
can use 73% of Oracle's SQL by orafce
Case study 3/3

33. 33Copyright©2016 NTT corp. All Rights Reserved.
• The migration of SQL is the most difficult
process in Database migration.
1. It will require significant cost to estimate for the
migration of SQL.
2. It will require significant cost to modify incompatible
SQL.
• These issues can be solved by db_syntax_diff
and orafce.
• Try Database migration, don’t hesitate.
Conclusion
• If you have interest, please help the development of
db_syntax_diff.
• To start with translate manual into English.

34. 34Copyright©2016 NTT corp. All Rights Reserved.
Thank you!