Database migration
from Sybase ASE to PostgreSQL

Achim Eisele and Jens Wilke
1&1 Internet AG
30.10.2013
Agenda

Introduction
Analysis
Differences between Sybase ASE and PostgreSQL
Porting the application
Migration strategies
Re...
Introduction

Agenda

Introduction
Analysis
Differences between Sybase ASE and PostgreSQL
Porting the application
Migration...
Introduction

Goals

Why we had been dissatisfied
many DBs on one System
the availability was depending on third party Syst...
Introduction

Why PostgreSQL

Most advanced Open Source RDBMS
Best TCO

1&1 Internet AG 2013

Database migration

2/25
Analysis

Agenda

Introduction
Analysis
Differences between Sybase ASE and PostgreSQL
Porting the application
Migration str...
Analysis

3 Applications

250 GB size
20 Mio. Queries per day
up to 1.500 transactions per second

1&1 Internet AG 2013

D...
Analysis

Architecture

Sybase
all reading and writing operations on one system
many databases, user and tools on a single...
Differences between Sybase ASE and PostgreSQL

Agenda

Introduction
Analysis
Differences between Sybase ASE and PostgreSQL
P...
Differences between Sybase ASE and PostgreSQL

Limitations of Postgres

Sybase offers bi-directional replication using the r...
Differences between Sybase ASE and PostgreSQL

Advantages of Postgres 1/3

Postgres’ MVCC vs Sybase’s locking
Postgres is c...
Differences between Sybase ASE and PostgreSQL

Advantages of Postgres 2/3

On Sybase ASE
strings longer than varchar/char o...
Differences between Sybase ASE and PostgreSQL

Advantages of Postgres 3/3

Postgres’ sequences are not limited to a single ...
Porting the application

Agenda

Introduction
Analysis
Differences between Sybase ASE and PostgreSQL
Porting the applicatio...
Porting the application

Database driver

Java (Hibernate)
loading JDBC driver
setting Dialect and DriverClass
PHP
On Debi...
Porting the application

Sybase vs PostgresSQL Dialect 1/2

Date/time
Sybase
SELECT getdate();

-- Oct 30 2013 03.52PM

SE...
Porting the application

Sybase vs PostgresSQL Dialect 2/2

concatenation
Sybase
SELECT zip + ’ ’ + city FROM address
Post...
Migration strategies

Agenda

Introduction
Analysis
Differences between Sybase ASE and PostgreSQL
Porting the application
M...
Migration strategies

Procedures

Migration strategies:
Dump Reload
Replication

1&1 Internet AG 2013

Database migration
...
Migration strategies

Dump - Reload 1/5

adjust and import the DDL
stop the application/write access
export Data, search a...
Migration strategies

Dump - Reload 2/5

Advantages
simple
less effort
rollback is also simple
Disadvantages
longer downtim...
Migration strategies

Dump - Reload 3/5

Export with bcp
bcp $db..$tablename out $fifo_path/$tablename.pipe -Jutf8 -c -t ’...
Migration strategies

Dump - Reload 4/5

Import in Postgres using COPY and fix the timestamps:
mkfifo $fifo_path/$tablename...
Migration strategies

Dump - Reload 5/5

the image type is converted to binary
image is imported into a text field and deco...
Migration strategies

Sybase → PostgreSQL - Replication

Closed source replication tool based on transaction logs
setup re...
Migration strategies

Pseudo-replication

Top requirement: minimal downtime!
should we write our own trigger based replica...
Migration strategies

Pseudo-replication - DDL 1/2

Automated Export of DDL
sp_tables
sp_columns
sp_pkeys
sp_fkeys
sp_stat...
Migration strategies

Pseudo-replication - DDL 2/2

// iterate over all tables
for $table in sp_tables
"CREATE TABLE $tabl...
Migration strategies

Pseudo-replication - Data 1/2

Data import
to reduce the system load, batch jobs were suspended
impo...
Migration strategies

Pseudo-replication - Data 2/2

Finalisation
create indices
during downtime
create sequences
create f...
Retrospects

Agenda

Introduction
Analysis
Differences between Sybase ASE and PostgreSQL
Porting the application
Migration ...
Retrospects

12 months with PostgreSQL

achivement of objectives
autonomy: dedicated hardware
stability: not one single ou...
Retrospects

Questions

???

1&1 Internet AG 2013

Database migration

25/25
Nächste SlideShare
Wird geladen in ...5
×

Database migration from Sybase ASE to PostgreSQL @2013.pgconf.eu

1,834

Published on

The talk explains some differences between Sybase ASE and PostgreSQL and shows two different migration strategies - the dump reload process and replication.

Published in: Technologie
0 Kommentare
0 Gefällt mir
Statistiken
Notizen
  • Hinterlassen Sie den ersten Kommentar

  • Be the first to like this

Keine Downloads
Views
Gesamtviews
1,834
Bei Slideshare
0
Aus Einbettungen
0
Anzahl an Einbettungen
1
Aktionen
Geteilt
0
Downloads
9
Kommentare
0
Gefällt mir
0
Einbettungen 0
No embeds

No notes for slide

Database migration from Sybase ASE to PostgreSQL @2013.pgconf.eu

  1. 1. Database migration from Sybase ASE to PostgreSQL Achim Eisele and Jens Wilke 1&1 Internet AG 30.10.2013
  2. 2. Agenda Introduction Analysis Differences between Sybase ASE and PostgreSQL Porting the application Migration strategies Retrospects
  3. 3. Introduction Agenda Introduction Analysis Differences between Sybase ASE and PostgreSQL Porting the application Migration strategies Retrospects
  4. 4. Introduction Goals Why we had been dissatisfied many DBs on one System the availability was depending on third party Systems many downtimes performance dependencies Our goals autonomy performance improvements high availability ⇒ satisfied customers :-) 1&1 Internet AG 2013 Database migration 1/25
  5. 5. Introduction Why PostgreSQL Most advanced Open Source RDBMS Best TCO 1&1 Internet AG 2013 Database migration 2/25
  6. 6. Analysis Agenda Introduction Analysis Differences between Sybase ASE and PostgreSQL Porting the application Migration strategies Retrospects
  7. 7. Analysis 3 Applications 250 GB size 20 Mio. Queries per day up to 1.500 transactions per second 1&1 Internet AG 2013 Database migration 3/25
  8. 8. Analysis Architecture Sybase all reading and writing operations on one system many databases, user and tools on a single system PostgreSQL Heartbeat / DRBD WAL / Streaming Replication (not synchronous!) SR-Standby for dedicated Statements 1&1 Internet AG 2013 Database migration 4/25
  9. 9. Differences between Sybase ASE and PostgreSQL Agenda Introduction Analysis Differences between Sybase ASE and PostgreSQL Porting the application Migration strategies Retrospects
  10. 10. Differences between Sybase ASE and PostgreSQL Limitations of Postgres Sybase offers bi-directional replication using the replication agent on Sybase it is possible to switch the replication direction Sybase offers replication of single tables with few impact, using the transaction log Sybase offers a replication solution for Road Warriors RAW Devices, advantage on modern file systems is doubtable I/O can be audited by process Plans of running queries can be shown 1&1 Internet AG 2013 Database migration 5/25
  11. 11. Differences between Sybase ASE and PostgreSQL Advantages of Postgres 1/3 Postgres’ MVCC vs Sybase’s locking Postgres is conforming to SQL-standard Postgres does know schemes and complex datatypes too Postgres does support grant for colums, not just revoke on Postgres it is possible to restrict the connection limit per user Logins can be restricted to IP adresses in Postgres 1&1 Internet AG 2013 Database migration 6/25
  12. 12. Differences between Sybase ASE and PostgreSQL Advantages of Postgres 2/3 On Sybase ASE strings longer than varchar/char or a page are cropped or fail different behaviour on insert and update default ASE pagesize is 2k Sybase creates a fixed size container for the DB and the transaction log. shrinking the DB container is only possible with version 15.7 1&1 Internet AG 2013 Database migration 7/25
  13. 13. Differences between Sybase ASE and PostgreSQL Advantages of Postgres 3/3 Postgres’ sequences are not limited to a single column as Sybase’s identity columns are and the value can be overwritten bcp (COPY) is neither capable to escape nor to quote Postgres does support multiple trigger for one event Sybase is planning to implement advisory locks for future versions index concurrently since ASE 15.7 SP 100, May 2013 1&1 Internet AG 2013 Database migration 8/25
  14. 14. Porting the application Agenda Introduction Analysis Differences between Sybase ASE and PostgreSQL Porting the application Migration strategies Retrospects
  15. 15. Porting the application Database driver Java (Hibernate) loading JDBC driver setting Dialect and DriverClass PHP On Debian: apt-get install php5-pgsql PDO only need another data source name “Plain” PHP sybase_connect(), sybase_select_db() => pg_connect() sybase_query() => pg_query() sybase_fetch_object() => pg_fetch_object() 1&1 Internet AG 2013 Database migration 9/25
  16. 16. Porting the application Sybase vs PostgresSQL Dialect 1/2 Date/time Sybase SELECT getdate(); -- Oct 30 2013 03.52PM SELECT convert(varchar(30), getdate(), 109); -- Oct 30 2013 03:52:34:126PM PostgreSQL SELECT current_timestamp; -- 2013-10-30 15:52:34.126305+01 SELECT to_char(current_timestamp, ’Mon DD YYYY HH12:MI:SS:MSAM’); 1&1 Internet AG 2013 Database migration 10/25
  17. 17. Porting the application Sybase vs PostgresSQL Dialect 2/2 concatenation Sybase SELECT zip + ’ ’ + city FROM address PostgreSQL SELECT zip || ’ ’ || city FROM address field length Sybase set textsize 512000 -- limits the size of a text or image column that is -- returned through a select statement. The default is 32KB. PostgreSQL n/a 1&1 Internet AG 2013 Database migration 11/25
  18. 18. Migration strategies Agenda Introduction Analysis Differences between Sybase ASE and PostgreSQL Porting the application Migration strategies Retrospects
  19. 19. Migration strategies Procedures Migration strategies: Dump Reload Replication 1&1 Internet AG 2013 Database migration 12/25
  20. 20. Migration strategies Dump - Reload 1/5 adjust and import the DDL stop the application/write access export Data, search and replace dates and times, import create indices, foreign keys and set sequences import and index/key creation by multiple processes in parallel logical checks re-configure and start the application 1&1 Internet AG 2013 Database migration 13/25
  21. 21. Migration strategies Dump - Reload 2/5 Advantages simple less effort rollback is also simple Disadvantages longer downtime limits regardiung quoting, escaping and content (date/time) empty strings are converted to NULL 1&1 Internet AG 2013 Database migration 14/25
  22. 22. Migration strategies Dump - Reload 3/5 Export with bcp bcp $db..$tablename out $fifo_path/$tablename.pipe -Jutf8 -c -t ’;’ Sybase bcp is not capable to write in unnamed pipes, just in files Date and time formats of Sybase are not conforming to any standard: milliseconds are separated by colons and a white space is missing before AM/PM 1&1 Internet AG 2013 Database migration 15/25
  23. 23. Migration strategies Dump - Reload 4/5 Import in Postgres using COPY and fix the timestamps: mkfifo $fifo_path/$tablename.pipe cat $fifo_path/$tablename.pipe | sed s/(;|^|[A-Z])([^;]*):([0-9]+)(AM|PM) (;|$)/12.3 45/g | psql -d $IMPORTDB -c "COPY $tablename from stdin CSV DELIMITER ’;’" alter sequence $seq restart with max(id)+1 create indices and foreign keys 1&1 Internet AG 2013 Database migration 16/25
  24. 24. Migration strategies Dump - Reload 5/5 the image type is converted to binary image is imported into a text field and decoded after that CREATE TABLE $tablename2 as (select id, decode(image, ’hex’) as image, ... from $tablename); DROP TABLE $tablename; ALTER TABLE $tablename2 RENAME to $tablename; 1&1 Internet AG 2013 Database migration 17/25
  25. 25. Migration strategies Sybase → PostgreSQL - Replication Closed source replication tool based on transaction logs setup replication, wait until it’s synchronised and restart the re-configured application. Advantages flexible point of time for the switch minimal downtime → less organisational overhead :) no impact on the production DB Disadvantages expensive closed source without warranty 1&1 Internet AG 2013 Database migration 18/25
  26. 26. Migration strategies Pseudo-replication Top requirement: minimal downtime! should we write our own trigger based replication tool? too much resource consuming on the server analysis of the table usage static tables autoincrement tables non-static tables tool based migration Optimizations (indices, field lengths) 1&1 Internet AG 2013 Database migration 19/25
  27. 27. Migration strategies Pseudo-replication - DDL 1/2 Automated Export of DDL sp_tables sp_columns sp_pkeys sp_fkeys sp_statistics sp_helpuser sp_helprotect 1&1 Internet AG 2013 Database migration 20/25
  28. 28. Migration strategies Pseudo-replication - DDL 2/2 // iterate over all tables for $table in sp_tables "CREATE TABLE $table->name (" // now iterate over all columns of table for $column in sp_columns $table->name "$column->name $column->type DEFAULT $column->default" // add primary key "ALTER TABLE $table->name ADD PRIMARY KEY "+sp_pkeys // grant statements for $right in sp_helprotect "GRANT $right->action ON $table->name TO $right->user" 1&1 Internet AG 2013 Database migration 21/25
  29. 29. Migration strategies Pseudo-replication - Data 1/2 Data import to reduce the system load, batch jobs were suspended import by 10 processes in parallel (horizontal partition) chunks of 5.000 rows per select Setup Up-Sync (auto increment) 80 tables migrated by triggers (non static tables) using a transitional migration table (table name, primary key, action) 1&1 Internet AG 2013 Database migration 22/25
  30. 30. Migration strategies Pseudo-replication - Data 2/2 Finalisation create indices during downtime create sequences create foreign keys content checks re-configure the applications’ DB connection parameter tests result: 10 minutes downtime 1&1 Internet AG 2013 Database migration 23/25
  31. 31. Retrospects Agenda Introduction Analysis Differences between Sybase ASE and PostgreSQL Porting the application Migration strategies Retrospects
  32. 32. Retrospects 12 months with PostgreSQL achivement of objectives autonomy: dedicated hardware stability: not one single outage in 12 months performance: dedicated hardware, read-only standby 1&1 Internet AG 2013 Database migration 24/25
  33. 33. Retrospects Questions ??? 1&1 Internet AG 2013 Database migration 25/25
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×