Oracle to PostgreSQL - a Travel Guide from Practice; Roland Stirnimann - Trivadis TechEvent 2019

1. Trivadis Blog@rstirnimann_ch
Oracle to PostgreSQL
A Travel Guide from Practice
Roland Stirnimann

2. Roland
• 14 years Trivadis
• Oracle HA, migration
• DevOps with Ansible
• PostgreSQL
• Data platforms
rstirnimann_ch

3. Agenda
• Warm-Up - Some Differences to Oracle…
• PostgreSQL versus EDB Postgres
• Project Experiences
• Oracle to PostgreSQL
• Oracle to EDB PostgreSQL Advanced Server
• Conclusion

4. Warm-Up – Some Differences
to Oracle…

5. • Oracle enforces constraints per statement, PostgreSQL per row
• Set constraint to DEFERRABLE in PostgreSQL for the same behavior
• Possible error: cannot use a deferrable unique constraint for referenced table
• Referenced columns by a foreign key must be a non-deferrable unique or primary key
INSERT INTO demo VALUES (1),(2);
UPDATE demo SET n=n+1;
ERROR: duplicate key value violates unique constraint "demo_pk"
DETAIL: Key (n)=(2) already exists.
CONSTRAINT behavior (demo)
Hint: INSERT of two rows at the same time does not work in Oracle.
VALUES (1),(2)
Source: Mathias Zarick, Trivadis Vienna

6. • Different behavior of SELECT INTO
• Oracle: Restrictive – allows only one value
• PostgreSQL: Tolerant – returns the first value of the SELECT
Oracle
CREATE OR REPLACE FUNCTION
get_bal(acc_no IN NUMBER)
RETURN NUMBER
IS
acc_bal NUMBER(11,2);
BEGIN
SELECT balance
INTO acc_bal
FROM accounts
WHERE account_id = acc_no;
RETURN acc_bal;
END;
/
FUNCTION in PL/SQL and PL/pgSQL
(demo)
PostgreSQL
CREATE OR REPLACE FUNCTION
get_bal(acc_no IN INTEGER)
RETURNS INTEGER
AS $$
DECLARE acc_bal INTEGER;
BEGIN
SELECT balance
INTO acc_bal
FROM accounts
WHERE account_id = acc_no;
RETURN acc_bal;
END;
$$ LANGUAGE plpgsql;
Source: Mathias Zarick, Trivadis Vienna

7. Statement error – ROLLBACK (demo)
• Statement error in PostgreSQL
• Uses by default AUTOCOMMIT
• Rollback to the beginning or to the last save point
• Transactional DDL support rollbacks DDL statements as well (e.g. CREATE TABLE)
• Statement error in Oracle
• Only the failed statement will be discarded
• Transaction is still open
Source: Mathias Zarick, Trivadis Vienna

8. What is crucial for the Migration?
Test, Test, Test…

9. PostgreSQL versus EDB
Postgres

10. EDB Postgres platform
Tools and support for enterprise level usage
Source: https://www.enterprisedb.com/

11. EDB subscriptions based on number of CPU
Source: https://www.enterprisedb.com/

12. EDB Postgres Advanced Server – the same
core
• EDB is based on PostgreSQL (binary compatible)
• Additional enterprise features
• Security: Password profiles, EDB*Wrap, etc.
• Performance features: Hints, extended analysis, resource manager, etc.
• Developer: Oracle PL/SQL, hierarchical query, synonyms, Oracle DBMS_* packages, etc.
• Tools: BART, failover manager, PEM, migration toolkit, etc.
• Oracle compatibility for simpler migration
• EDB documentation extends the community documentation by the specific EDB features
• https://www.postgresql.org/docs/11/index.html
• https://www.enterprisedb.com/docs/en/11.0/EPAS_Guide_v11/toc.html
• EDB has several well-known PostgreSQL developers on board

13. Project Experiences
Project 1

14. Project 1 - Energy supplier
Estimation for the migration effort to
PostgreSQL/EDB
• Strategy:
• 80% of all databases are in the AWS cloud until 2020
• Move away from commercial RDBMS and no vendor lock-in
• Motivation:
• Hardware renewal for Oracle RAC is not required
• For the time being, the existing hardware is sufficient if some applications are moved to AWS
(PostgreSQL)
• Project goal:
• Analysis tool for a migration feasibility assessment into the AWS cloud for the existing 800
Oracle schemas
• Target-RDBMS is either PostgreSQL (RDS, DBaaS) or EDB Postgres AS (EC2, IaaS)

15. Project 1 - Requirements
• No direct database access. Collecting and analyzing happens separately
• Step 1: Collect data as CSV
• Step 2: Analyze data and present them as HTML report
• No software installation, only a script deployment is required
• Gathering the required data:
• gather_db_details.sql: Feature usage, Object types and schema details
• gather_db_behavior.sql: Application behavior, e.g. programs, drivers accessing the application
• gather_db_metrics.sql: Metrics about the resource usage (CPU, I/O)
• Oracle RAC awareness
• Important to assign the Oracle instances to the corresponding AWS DB-instance-classes
https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Concepts.DBInstanceClass.html

16. • Execute the collector scripts in Oracle as DBA user
• Option -M is only available since 12.2, otherwise use set colsep in SQL*Plus
• Export the result from SQL Developer into a text file delimited by ;
export NLS_DATE_FORMAT='dd.mm.yyyy hh24:mi:ss'
sqlplus -S -M 'CSV on delimiter ;' system@REPO1
@../sql/gather_db_details.sql >gather_db_details.csv
sqlplus -S -M 'CSV on delimiter ;' system@REPO1
@../sql/gather_db_behavior.sql >gather_db_behavior.csv
sqlplus -S -M 'CSV on delimiter ;' system@REPO1
@../sql/gather_db_metrics.sql >gather_db_metrics.csv
Project 1 - Collect information as CSV

17. • CSV files are used as input for the report creation
perl -I ../lib mactl.pl
--input-files "REPO1;../csv/gather_db_details.csv"
--behavior-input-files "REPO1;../csv/gather_db_behavior.csv"
--metric-input-files "REPO1;../csv/gather_db_metrics.csv"
--output-directory ../html --schemas-only
Project 1 - Analyze the collected data

18. Project 1 - Report demo

19. Project 1 - Next steps
• Identify easily portable applications and migrate to PostgreSQL or EDB Postgres AS
• After first experience try the harder ones
• Migration tools for schema or database
• PostgreSQL: ora2pg http://ora2pg.darold.net
• EDB Postgres AS: Migration Toolkit
https://www.enterprisedb.com/products/edb-postgres-platform/edb-migration-tool-kit
• Get some Oracle compatibility in PostgreSQL with orafce tool
https://github.com/orafce/orafce
• EDB Postgres AS has comprehensive Oracle compatibility out-of-the-box

20. Project Experiences
Project 2

21. Project 2 - Aviation
Migration to EDB Postgres Advanced Server
• Strategy:
• Consolidation of vendors
• Analyzing cost saving potentials
• Motivation:
• Reducing maintenance and support costs
• Complete renewal of an application (Oracle Forms)
• Project goal:
• Find out the feasibility of replacing Oracle with EDB Postgres Advanced Server
• Knowledge about the migration scenarios and tools
• Know the required structural changes in the code

22. Project 2 - Requirements
• Perform a two-piece proof-of-concept
• EDB: Database migration assessment (DMA) and online EDB Postgres migration portal
• Trivadis: Practical migration part
• Timely limited EDB Postgres Advanced Server license
• Doing based on a complex and critical Oracle database
• Customer provided a server with OS (Centos 7)
• EDB Postgres Advanced Server 11
• Access to the Oracle database via SQLNet

23. Project 2 - EDB database migration
assessment
• Input was based on a structure export of the source database (export script from EDB)
• PDF report from EDB Professional Services
• It looks worse than it really is as we will see later during the practical part!

24. Project 2 - DMA classification
• Classification of objects like procedures, functions, tables, indizes, etc.
• The status Invalid occurs often because of dependencies to objects in other schemas
• Incompatible means not impossible
• Requires some additional effort in the code
• Examples: Index organized table must be replaced or some reserved words must be enclosed
by quotes

25. Project 2 - EDB Postgres migration portal
(1)
• Upload of the structure export incl. analysis (https://www.enterprisedb.com/edb-postgres-
migration-portal)

26. Project 2 - EDB Postgres migration portal
(2)
• EDB offers many ready-to-use solutions in Postgres for most of the known incompatibilites

27. • EDB MTK is a CLI tool
• Documentation: https://www.enterprisedb.com/docs/en/52.0/MTK_Guide_v52.0/toc.html
• Installation via EDB YUM repository (yum install edb-migrationtoolkit)
• Configuration file contains connection details to Postgres and Oracle
vi /usr/edb/migrationtoolkit/etc/toolkit.properties
SRC_DB_URL=jdbc:oracle:thin:@192.168.38.186:1521:DB01
SRC_DB_USER=system
SRC_DB_PASSWORD=pw
TARGET_DB_URL=jdbc:edb://localhost:5444/edb
TARGET_DB_USER=enterprisedb
TARGET_DB_PASSWORD=pw
/usr/edb/migrationtoolkit/bin/runMTK.sh –help
Project 2 - EDB migration toolkit (1)

28. Project 2 - EDB migration toolkit (2)
• Online or offline migration
• Online exports from Oracle and imports directly to EDB Postgres
• Offline (-offlineMigration) exports into SQL files for manual import
• Only structure and/or data
• -schemaOnly export/import only the sturcture (DDL)
• -dataOnly export/import only the data
• Without explicit parameter definition it copies structure and data
• Different options for importing of different objects types:
-allTables, -allSequences, -skipFKConst, etc.
• Oracle specific options: -allProfiles, -allDBLinks, -allSynonyms, etc.
• Further migration options, e.g. to change data types during the migration

29. • Modify/correct DDL scripts within the export directory ~/mig
• Create Postgres database user, schema and tablespace
• Load the structure
• Offline structure export from Oracle (select any dictionary privilege)
runMTK.sh -offlineMigration ~/mig -schemaOnly -logDir ~/mig/logs
-sourcedbtype oracle -targetSchema schema1 schema1
create user admin password 'xxx';
create database mydb with owner admin;
c mydb enterprisedb
create user schema1 with login identified by pw;
create tablespace ts_schema1 owner schema1 location '/opt/tbs/schema1';
edb-psql -f mtk_schema1_ddl.sql -o mtk_schema1_ddl.log mydb schema1
Project 2 - MTK approach

30. • Many initial errors because of dependencies
• Included further schemas
• Created a database link to another database (Instant Client required)
c mydb schema1
CREATE DATABASE LINK oradb.world CONNECT TO user1 IDENTIFIED BY 'pw'
USING '//192.168.40.223:1521/ORADB’;
select count(*) from table1@oradb.world;
Project 2 - Findings (1)
• Import order is key, order of schemas and DDL scripts
• Permissions (GRANT) are required between schemas as well
grant select on all tables in schema schema1 to schema2,schema3,schema4;

31. • ROWID functionality in EDB Postgres activated
Parameter default_with_rowids=on in $PGDATA/postgresql.conf
• Several syntax fixes required where the EDB parser is somehow more restrictive than Oracle
• Few keywords as column names had to be enclosed in quotation marks
• BLOB data type changed for one table to BYTEA
• Granted access on SYS package UTL_FILE and a directory object created
• search_path extended in DDL files with “public” for the visibility of public synonyms
• Tablespace names defined in DDL files to create the objects correctly
SET search_path=schema1,public;
SET default_tablespace = ts_schema1;
Project 2 - Findings (2)

32. Project 2 - Recommendations
• Having the right persons at the table (DBA and developer)!
• Work in parallel on application and database related issues
• Agile, interactive approaching - migrate, verify, correct, rollback and the same again
• Do not try to fix all potential problems at the first run
• Start with the structure because loading is fast but the potential for issues is higher
• Use offline migration to modify scripts before importing them
• Data migration should be well thought out due to the long run time
• Identify large objects and migrate them separately without MTK (delta-migration)
• Basically we expect less issues during data migration
• BUT: The run time can be bad especially with the standard JDBC copy (downtime)
• Search for alternatives like database link (parallelism)

33. Project 2 - Next steps
• Complete the structure migration including all dependencies
• Feasibility of the data migration
• Accepted downtime defines the maximum of run time for the go-live
• Based on that the migration concept has to be created (database link, etc.)
• Test very well the connectivity of interfaces to Postgres (drivers)
• Create a Postgres operation concept
• The 15 days for porting the structure alone is not enough!

34. Conclusion

35. Conclusion
• Many things are technically possible with corresponding effort
• EDB Postgres simplifies the migration of Oracle features heavily (PL/SQL)
• Which path does the further development of ported applications follow, Oracle or Postgres?
• Postgres as alternative RDMBS in the company is a wise decision
• EDB offers support for community PostgreSQL as well
• Oracle compatibility is less important for new project without an Oracle history
• Knowledge level about PostgreSQL varies heavily in companies (training)