Migration from one database engine to another one for existing applications is not a trivial task. There are a lot of technical challenges, including but not limited to database features, object/data types and different behavior. In this presentation we would like to talk about a customer PoC, with the goal to migrate as many databases from Oracle to MySQL as possible. This includes categorization of several hundred Oracle databases into migration classes, based on custom criteria, as well as tool development to support schema (DDL) conversion and data migration.

1. BASEL BERN BRUGG DÜSSELDORF FRANKFURT A.M. FREIBURG I.BR. GENF
HAMBURG KOPENHAGEN LAUSANNE MÜNCHEN STUTTGART WIEN ZÜRICH
Oracle to MySQL Migration – Challenges
Robert Bialek
Principal Consultant
Robert Kruzynski
Principal Consultant

2. Who Am I
Principal Consultant and Trainer at Trivadis GmbH in Munich.
– MSc in Computer Engineering.
– At Trivadis since 2004.
Focus:
– Oracle Database High Availability.
– Database Architecture/Internals.
– Backup/Recovery.
– Troubleshooting/Performance Tuning.
– Linux.
– Trainer: O-GRINF, O-RAC, O-DG.
15.07.20172 Oracle to MySQL Migration – Challenges

3. Who Am I
Principal Consultant and Trainer at Trivadis GmbH in Munich.
– Dipl.-Ing. Technische Informatik (TU Berlin)
– At Trivadis since 2002
Focus:
– Troubleshooting/Performance Tuning.
– Capacity and Resource Management.
– Automatization/Optimization.
– Backup/Recovery.
– Tool Owner: TVD-CapMan, TIB (Trivadis Intelligent Backup)
– Trainer: O-TUN, O-TUN-PRACT.
15.07.20173 Oracle to MySQL Migration – Challenges

4. Customer-Driven Trend
Two strategic Trivadis IMS customers switched the „default“ database engine from
Oracle to MySQL for new projects/applications.
Target environment:
– MySQL Enterprise Edition (optionally with data replication).
– Red Hat Enterprise Virtualization (Live Migration).
– Two security classes: standard and confidential (secret as of now not available).
– Goal: automatic full software stack provisioning.
In 2017, one of the customers decided to start a Proof-of-Concept project for
migration of exising applications (fixed-price).
15.07.2017 Oracle to MySQL Migration – Challenges4

5. Customer Project Goals
Which and how many Oracle applications/schemas can be migrated to MySQL?
What are the technical differences and challenges?
Categorize the existing Oracle databases (~1700) into migration (difficulty) classes.
– Should be performed automatically and based on customer thresholds, which can
be changed at any time.
Migration method should be successfully demonstrated on 5 Oracle applications.
15.07.2017 Oracle to MySQL Migration – Challenges5

6. Our Challenges
Create Oracle2MySQL migration matrix (features, data types, ...).
Define database migration (difficulty) classes.
Create a framework to scan the whole Oracle database environment and categorize
schemas/databases into migration (difficulty) classes.
Generate MySQL schema DDLs.
Migrate data from Oracle to MySQL database and check data correctness after
migration.
There is no framework on the market which supports all, or at least most of the
requirements.
– What we need is a small, but yet powerful framework.
– This way, a new tool was born: TVD-Espresso ☺.
15.07.2017 Oracle to MySQL Migration – Challenges6

7. Agenda
Oracle to MySQL Migration – Challenges15.07.2017
1. Oracle Database/Schema Classification
2. MySQL Schema Generator
3. Data Migration
4. Summary
7

8. Oracle Database/Schema
Classification
15.07.2017 Oracle to MySQL Migration – Challenges8

9. Migration Classes – Overview
Database/Schema classification based on features, data types, usage patterns, ...
New classes can be easily introduced.
15.07.2017
Migration Class Description
Easy (1) Automatic migration is possible.
Intermediate (2) Migration ist possible, requires some manual tasks
and/or re-coding.
Difficult (3) Migration ist possible, requires many and complex
manual tasks and/or re-coding.
Very Difficult (4) Migration ist not recommended (technical and/or
financial reasons).
Oracle to MySQL Migration – Challenges9

10. Example Migration Classes (1)
15.07.2017 Oracle to MySQL Migration – Challenges10
Class – Easy
Heap Table
Index-Organized Table
B-Tree Index
PK,FK,UK,NOT NULL Constraints
DATA_TYPE_CATEGORY_1:
▪ LOB (<4GB), (VAR)CHAR, LONG,
RAW, DATE, TIMESTAMP (<7),
NUMBER, ...
Class – Intermediate
Bitmap Index
View
Oracle Outer-Join Syntax
Synonym
Database Job

11. Example Migration Classes (2)
15.07.2017 Oracle to MySQL Migration – Challenges11
Class – Difficult
Temporary Table
Function-Based Index
Materialized View
Maximum index key length 3072
bytes
CHAR data type length > 255
Class – Intermediate/Difficult
Check Constraints:
▪ >0 Intermediate <=20
▪ >20 Difficult
Sequences:
▪ >0 Intermediate <=10
▪ >10 Difficult

12. Example Migration Classes (3)
15.07.2017 Oracle to MySQL Migration – Challenges12
Class – Very Difficult
Advanced Queueing
PL/SQL Package
Database Link
TDE, Encrypted columns
TIMESTAMP WITH [LOCAL]
TIMEZONE
Class – Difficult/Very Difficult
Procedures, Functions, Triggers:
▪ >0 Difficult <=10
▪ >10 Very Difficult
Object Source Code Lines:
▪ >0 Difficult <=100
▪ >100 Very Difficult

13. Migration Classes – Configuration
Metadata is stored in Oracle table.
15.07.2017 Oracle to MySQL Migration – Challenges13
SQL> SELECT FEATURE_NAME, DIFFICULTY "DIF.", MAXIMUM_COUNT MAXC, MAXIMUM_LENGTH MAXL,
DIFFICULTY_HIGH DIFH, MAXIMUM_COUNT_HIGH MAXH, DESCRIPTION
FROM MIGSCAN.MIGRATION_FEATURE;
FEATURE_NAME DIF. MAXC MAXL DIFH MAXH DESCRIPTION
------------------------------ ---- ---- ---- ---- ---- ----------------------------------------------
DATA_TYPE BFILE 4 Does not exist in MySQL
DATA_TYPE CATEGORY_NOT_1 2 0 Simple data types
DATA_TYPE CHAR_TOO_LONG 3 0 CHAR is restricted to 255 bytes in MySQL
DATA_TYPE TIMESTAMP_WITH_TIMEZONE 4 Does not exist in MySQL
MAX INDEX KEY LENGTH 3 3072 Max index key length is hard limited
OBJECT CONSTRAINT CHECK 2 0 3 20 Does not exist in MySQL
OBJECT CREDENTIAL 4 Does not exist in MySQL
OBJECT DATABASE_LINK 4 Does not exist in MySQL
OBJECT ENCRYPTION_COLUMN 4 0 Binlogs are not encrypted in MySQL
OBJECT INDEX FUNCTION_BASED 3 0 Does not exist in MySQL
OBJECT SYNONYM 2 Does not exist in MySQL
OBJECT SOURCE_CODE_LINES 3 0 4 100 Require re-engineering
SYS GRANTS 1 Application re-coding could be necessary

14. Database/Schema Classification (1)
Job scheduled from a central location:
– Implemented in shell, PL/SQL, SQL.
15.07.2017
DB-Server List
global_mig_scan.sh
target_sql_all_dbs.sh
target_mig_scan.sql
target_sql_all_dbs.sh
target_mig_scan.sql
target_sql_all_dbs.sh
target_mig_scan.sql
target_sql_all_dbs.sh
target_mig_scan.sqlAnalysis
Oracle to MySQL Migration – Challenges14
Repository-DB
- Meta Information
- Scan Results
SQL
SQL
SQL
SQL
SQL, SQL, ...
ETL

15. Database/Schema Classification (2)
What the scan job cannot automatically find out:
– Application usage of specific SQL functions.
– Usage of dictionary objects.
– Usage of anonymous PL/SQL code.
Some information is extracted from V$ views, but this might be not reliable.
15.07.2017 Oracle to MySQL Migration – Challenges15

16. Database/Schema Classification Results (1)
Example classification results
– Fine tuning still necessary!
15.07.2017
6%
22%
20%
52%
Cat. 1
Cat. 2
Cat. 3
Cat. 4
2% 2%
15%
81%
Cat. 1
Cat. 2
Cat. 3
Cat. 4
MAX_DIFFICULTY_DB [%]
MAX_DIFFICULTY_SCHEMA [%]
Oracle to MySQL Migration – Challenges16

17. Database/Schema Classification Results (2)
Classification Results are stored by ETL job in Oracle table.
15.07.2017 Oracle to MySQL Migration – Challenges17
SQL> SELECT MAX_DIFFICULTY_DB mdb, MAX_DIFFICULTY_SCHEMA msch, SCHEMA_NAME, DB_SCHEMAS, HEAP_TABLES,
DIFFICULT_FEATURES
FROM MIGSCAN.MIGRATION_REPORT;
MDB MSCH SCHEMA_NAME DB_SCHEMAS HEAP_TABLES DIFFICULT_FEATURES
---- ---- -------------------- ---------- ----------- ------------------------------------------------
3 1 A4MBBBPRSPOI 5 1
3 3 A4MBBBDWA1 5 5 OBJECT SEQUENCE=3,OBJECT SOURCE_CODE_LINES=18,
OBJECT TRIGGER=3,
3 3 A4MBBBMVS 5 7 OBJECT SEQUENCE=2,
3 3 A4MBBBMVSGEO 5 7 OBJECT SEQUENCE=2,
3 3 A4MBBBONUP 5 87 MAX INDEX KEY LENGTH,OBJECT FUNCTION=1,
OBJECT PROCEDURE=1,OBJECT SEQUENCE=3,
OBJECT SOURCE_CODE_LINES=39,
4 1 A4MBBAONUP 87 8
4 1 A4MBBBMOBKEY2 55 19
4 1 A4MBBBPRSPOI 7 1
4 3 A4MBBBDESTIM 87 31 OBJECT FUNCTION=1,OBJECT SEQUENCE=1,
OBJECT SOURCE_CODE_LINES=1,
4 3 A4MBBBDESTIM 89 31 OBJECT PROCEDURE=1,OBJECT SEQUENCE=1,
OBJECT SOURCE_CODE_LINES=27,

18. MySQL Schema Generator
15.07.2017 Oracle to MySQL Migration – Challenges18

19. MySQL Schema Generator – Overview
Generates in the source environment MySQL schema DDLs:
– Tables, Indexes, Constraints.
Responsible for data type mapping.
15.07.2017
SET SERVEROUTPUT ON
BEGIN
oracle2mysql.gen_tab_ddl(
in_schema => '<APP_NAME>',
in_cascade => TRUE,
in_file => '<APP_NAME>_all.sql');
END;
/
Gen. DDL
MySQL
DDL
Oracle
Oracle to MySQL Migration – Challenges19
mysql> source <APP_NAME>_all.sql

20. MySQL Schema Generator – Data Mapping
15.07.2017
Oracle MySQL
VARCHAR2,NVARCHAR2
• <4001 Bytes
• >4000 Bytes
• VARCHAR
• TEXT
CHAR,NCHAR
• <256
• >255
• CHAR (different padding behaviour)
• Needs clarification.
LONG (max. 2GB) LONGTEXT (max. 4GB)
LONG RAW (max. 2GB) LONGBLOB (max. 4GB)
CLOB,NCLOB (max. 8 - 128 TB) LONGTEXT (max. 4GB)
Oracle to MySQL Migration – Challenges20

21. MySQL Schema Generator – Data Mapping
15.07.2017
Oracle MySQL
BLOB (max. 8 - 128 TB) LONGBLOB (max. 4GB)
RAW (max. 2000 bytes) VARBINARY (4000 bytes)
TIMESTAMP
• max. PRECISION 6
• PRECISION 7-9
• TIMESTAMP
• <NOT_SUPPORTED>
DATE DATETIME
NUMBER(precision,scale) DECIMAL(precision,scale)
NUMBER(precision) DECIMAL(precision)
NUMBER(precision,-scale) <NOT_SUPPORTED>
Oracle to MySQL Migration – Challenges21

22. MySQL Schema Generator – Data Mapping
15.07.2017
Oracle MySQL
NUMBER (*) • Default: DECIMAL(65,30)
BINARY_FLOAT DOUBLE
BINARY_DOUBLE DOUBLE
FLOAT DOUBLE
DEFAULT (Constant-Value, Functions) DEFAULT (Constant-Value, Functions*)
Oracle to MySQL Migration – Challenges22

23. Data Migration
15.07.2017 Oracle to MySQL Migration – Challenges23

24. Data Migration – Overview
Data migration performed by TVD-Espresso using JDBC-Thin API:
– Automatic data type migration and conversion (including LOBs, LONGs and RAW).
– Makes sure that all migrated rows are identical (column values in every row).
– Migrates all or selected tables in a schema, optionally in parallel.
15.07.2017
java -classpath ojdbc6.jar:mysql-connector-java-5.1.42-bin.jar:.
com.trivadis.espresso.Espresso
--truncate
--tablepattern=%
--noflashback
--parallel=10
--sourceowner=<SCHEMA>
--sourcejdbc=jdbc:oracle:thin:@//<SERVER>:<PORT>/<SERVICE_NAME>
--sourceuser=<MIGRATION_USER>
--sourcepass=xxxxxxxxxxxxxxx
--destjdbc=jdbc:mysql://<SERVER>:<PORT>/<SCHEMA>
--destuser=<MYSQL_USER>
--destpass=xxxxxxxxxxxxxxx
Oracle to MySQL Migration – Challenges24

25. Data Migration – Example Output
Example data migration output
15.07.2017
INFO: loading oracle driver
INFO: loading mysql driver
INFO: connecting to jdbc:oracle:thin:@//<SERVER>:<PORT>/<SERVICE_NAME> user mysqlmig
INFO: connecting to jdbc:mysql://<SERVER>:<PORT>/<SCHEMA>...
INFO: copying table
CEIDMAP_Z6:..........................................................................
.................................................... 12664975 rows processed in
466617 ms
INFO: row count of table CEIDMAP_Z6 is 12664975 OK
INFO: comparing column values in table CEIDMAP_Z6: finished in 299267 ms
CEIDMAP_Z6.SAPDOCID CHAR OK
CEIDMAP_Z6.ARCDOCID CHAR OK
CEIDMAP_Z6.DOCPROT NUMBER OK
CEIDMAP_Z6.NOTE NUMBER OK
CEIDMAP_Z6.STATE NUMBER OK
Oracle to MySQL Migration – Challenges25

26. Data Migration – Example Output
Summary
– Performance („SELECT * FROM table ORDER BY PK”)
15.07.2017
EXECUTOR STATS: started worker threads 66
EXECUTOR STATS: finished worker threads 66
EXECUTOR STATS: copied tables 66
EXECUTOR STATS: copy exceptions 0
EXECUTOR STATS: rows copied 109376002
EXECUTOR STATS: compare exceptions 0
EXECUTOR STATS: select-order-by time source 201 s
EXECUTOR STATS: select-order-by time destination 1048 s
Oracle to MySQL Migration – Challenges26

27. Summary
15.07.2017 Oracle to MySQL Migration – Challenges27

28. Summary (1)
With the newly developed framework/solution, we are able to fulfill all the customer
project requirements/goals.
– Find the best migration candidates.
– Perform the migration including data quality check.
– Support re-engeneering tasks.
With our experience and the tools we can support similar projects in any environment
and any other DBMS.
– JDBC connection to both systems is required.
15.07.2017 Oracle to MySQL Migration – Challenges28

29. Summary (2)
Does it make sense to migrate existing applications between different DBMS
products?
– It depends…
It might be the only choice for systems using cloud infrastructure.
Necessary changes on application level should not be underestimated! This part
might kill many migration projects ☺.
There could be also a lot of potential to optimize the usage of existing environments
(better resource management, tuning, software editions, ...).
15.07.2017 Oracle to MySQL Migration – Challenges29

30. Session Feedback – now
30 15.07.2017
Please use the Trivadis Events mobile app to give feedback on each session
Use "My schedule" if you have registered for a session
Otherwise use "Agenda" and the search function
If the mobile app does not work (or if you have a Windows smartphone), use your
smartphone browser
– URL: http://trivadis.quickmobileplatform.eu/
– User name: <your_loginname> (such as "svv")
– Password: sent by e-mail...
Oracle to MySQL Migration – Challenges