The talk will be about the project to find a replacement for all IBM products in the company with the example for the databases. What was the goal of the project, the learning, a short overview about the options we migrated about 500 db2 databases to EnterpriseDB. The database size was from a small size up to 4 TB and we implemented a completely new fully automated deployment of VM and database. Databases are now 11 month in production. The talk will have an overview of the project, the learnings, a few parameters and technical parameters that were found for stability and performance.

1. Migration DB2 to EDB - Project Experience
Harald Stefan
Munich, December 2020

2. Table of contents
2
December 2020
Migration DB2 to EDB - Project Experience
Introduction
Situation, scope of project and decision
Technology
Status, lessons learned and conclusion
01
02
03
04
Additional information and reference05

3. Introduction
3
An overview of company and speaker01

4. Company BG-Phoenics GmbH
4
December 2020
Migration DB2 to EDB - Project Experience
 BG-Phoenics GmbH has
 > 500 employees, thereof 250 in IT
 14 locations in Germany
 We are THE service provider for the German Social
Accident Insurance.
 We manage over 8,500 IT users with more than
21,000 end devices.
 We as a full service provider develop and operate
complete applications for our customers including
infrastructure.

5. About me
5
 Head of platform databases
 15 years leadership experience for technology teams
 25 years database experience in development and
administration of various database systems (Relational,
NoSQL, Shared Nothing, Hadoop, ..)
 many technology projects and experience in various
technologies
 since December 2017 at BG-Phoenics GmbH
 Contact
harald.stefan@bg-phoenics.de
NDecember 2020
Migration DB2 to EDB - Project Experience

6. Situation, scope of project and
decision
6
02

7. Situation 2017
7
 Nearly 1,500 Databases in operation, thereof about 500 are DB2
 Many databases highly consolidated and complex to operate due to license situation
 7 different database systems
 Mission Critical Systems in HA network on VMWare
 long set-up times for systems deployment by many participating teams
 many weekend activities because of license-driven architecture
December 2020
Migration DB2 to EDB - Project Experience
DB2 DBRMS
Instanz1
Instanz2
Instanz20
Service
1
Service
3
Service
2

8. Mission and scope of project
8
 Replacement of IBM products until End of 2019 and therefore large parts of the core
infrastructure
- Content management systems
- about 500 DB2 databases
- Backup system TSM
 Database size from < 100 GB up to 2.5 TB compressed, partly with cluster HA setup
 Conversion of the databases to characterset UTF8
 Content management systems with about 1.5 billion documents
 Looking for replacement products for Mission Critical Systems in the Enterprise Network
 Reduction of license dependencies to best technical architecture for the company
 Replacement of application servers from websphere to Jboss
 Replacement of IBM ITM monitoring system
December 2020
Migration DB2 to EDB - Project Experience

9. Analysis and procedure for databases
9
 What else do we want to achieve?
- automatic deployment of server and database with all parts
- simple license model
- high throughput and stability
- open architecture for expansion to cloud and container
- backups to S3 storage, if possible
 Create matrix with all necessary features
(HA, Partitioning, Hints, Point in Time Recovery, procedural ability
easily accessible support, future-proof product and support, Linux basis)
 be careful, the devil is in the details !
 after market analysis comparison of 16 database systems with 32 weighted criteria
 decision on EDB
December 2020
Migration DB2 to EDB - Project Experience

10. Why EDB?
10
 Fast and competent support, meet our SLA demands
 Security roadmap fits our planning
 DB2 code migratable
 Oracle databases are also migratable (as a further option)
 All necessary tools included and without additional costs
ressource manager, HA, standby, system wait diagnostic, pooling etc.
 ACID Compliant
 Application support for all of our core applications
 Open minded for customer wishes (adding options and features to the code)
 Sustainable and stable with good product roadmap
December 2020
Migration DB2 to EDB - Project Experience

11. Migration Method
11
 No direct link from db2 to EDB possible
 No direct way to migrate character set
 We used a tool to do the migration including data-type and character conversion
 12 batches in parallel to migrate and transport data
 Reworked code by developers for the procedural business logic
December 2020
Migration DB2 to EDB - Project Experience
DB2
Linux
EDB
Linux
Convert
Tool
Convert
Tool
Convert
Tool

12. Technology
12
03

13. Technology – Agenda
13
• Private Cloud EDB
• Problems & Solutions
• Future Vision
December 2020
Migration DB2 to EDB - Project Experience

14. Private Cloud - Requirement
14
 EDB, private cloud  everything automated
 Vrealize (Puppet)
 We developed the complete functionality with BASH shell
script, then handover to cloud automation team
 Private Cloud, RAM (8-128GB) + CPU (2-12) + HDD
(130GB-4TB) selectable
 EDB, each DB into a separate VM (x 350 DBs)
 HA Solution Standby with EFM
 Day-2 Operations
December 2020
Migration DB2 to EDB - Project Experience
success
what most
people believe
In reality

15. Components and Versions
15
December 2020
Migration DB2 to EDB - Project Experience
Product Version Description
VMware 6.x (6.5 or higher) Hypervisor
RHEL 7.8 Operating System
EDB EPAS 10.12.20 Database
EFM 3.4 -> 3.9 Failover manager (HA)
OpenJDK 1.8.0 Java JDK
Commvault 11(BUILD80) SP12
(1303739)
Backup solution
ZIS Monitoring
Automic (was
UC4)
10.0.8+hf.5.build.1413
, changelist
1489053929
Job Management
ITM 18/08/20cit_2.8.0.900
0
Monitoring
PEM Agent 7.6.0-3 -> 7.13 Monitoring
PEM Server 7.6.0-3 -> 7.13 Monitoring
BART Server 2.2 Backup
Die Werkzeuge
vRealize
Puppet
Satellite

16. Order VM/DB via vRA
16
December 2020
Migration DB2 to EDB - Project Experience

17. Problems & Solution
17
 What users do we need?
DB/OS
 Documentation
 Kernel Parameters
 Limits
 Postgresql.conf
 EFM & Network
 Init Parameter
checksum
December 2020
Migration DB2 to EDB - Project Experience

18. EFM & Network
18
/etc/edb/efm-3.4/efm.properties
virtualIp=10.11.12.236
virtualIp.interface=ens192
virtualIp.prefix=16
virtualIp.single=true
(facepalm!)
Virtual IP = Main Gateway
All network traffic over virtual IP
NOTHING works
December 2020
Migration DB2 to EDB - Project Experience

19. Current Status of Deployment
19
December 2020
Migration DB2 to EDB - Project Experience
0
5
10
15
20
25
Backup (Hours) Restore (Hours) Deployment (Hours) Umlage (Hours)
DB2 vs. EDB
DB2 EDB

20. Status, lessons learned and
conclusion
20
04

21. Current situation and status
21
 Simple and good license model in place
 Stable and high performant platform that passes HA and high load tests
 Live since December 2019
few problems from the DB layer and hardly any failure in 1 year of operation
 Databases with 4.5 TB and 128 GB RAM without any problem
-> suitable and tested for large environments
 Fully automated provisioning of VM and DB from private Cloud Single and HA
with all necessary processes around (monitoring, backup)
 Day2 operations for Online extensions possible in minutes, including recalculation of Linux
and performance values
 Online deployment for database configuration without service interruption
-> reload for nearly every parameter possible, caches and cursors remain valid
 UTF8 character conversion in DB done (for database without any problem)
-> check application layer and client settings
December 2020
Migration DB2 to EDB - Project Experience

22. Lessons Learned
22
 Do not underestimate the emotional component
 Communicate as much as possible what you want
to achive and the benefit of the migration
 Do a complete review of your application an how
it is working
 Testing, testing, testing
 Have no fear about the migration
 Start with POC and testing, not only paperwork
 enterpriseDB is ready for high load databases in
business critical environments
 Implement patch process as early as possible to
avoid problems
(we have seen xfs bug and efm memory leak
-> both solved and running fine)
 Think about automation
December 2020
Migration DB2 to EDB - Project Experience

23. Conclusion
23
Projekt BSW – EDB Private Cloud
IT-B/MD/DM
• Provision in hours vs. provision in weeks
• Batch Quantities of deployments using vRO workflow
• No reworking necessary after provision
• Facilitate daily business by Day2 operations (root; disk+ram enlargement, …)
• Avoidance of errors by onTop-Automation (see documentation and passwords)
• Performance by precisely calculated parameters, limits, heaps...etc
• With the 1:1 Setup we have smooth operation and very rare incidents
• PrivateCloud open for other RDBMS
• Permanent expansion of our Cloud-Service planned !!
Currently Secure Linux in DMZ and migration V12 in work

24. Ihre Fragen . Ihr Feedback .
24
Questions - Feedback

25. Additional information and
reference
25
05

26. Users (1/2)
26
December 2020
Core renovation, Project experience EDB
 OS users
 Enterprisedb (EDB binary & process owner)
 Efm (EFM binary & process owner)
 Database Owner (application)
 DB Users
 Enterprisedb (superuser)
 Efm (DB Status for HA)
 S48deploy (HA - Standby)
 Database Owner (application)

27. Users (2/2)
27
December 2020
Migration DB2 to EDB - Project Experience
Name Purpose OS/DB user
Root OS Superuser OS
Enterprisedb EDB Binary & Prozess Owner OS
<DB_NAME> Db Owner DB & OS
<DB_NAME>a Phoenics Schema-user DB & OS
<DB_NAME>e Phoenics Schema-user DB & OS
Efm EFM (HA Solution) Binary & process owner DB & OS
S48commv OS User for Commvault DB & OS
S48deploy DB User for Replication DB & OS
S48itbdm DBA Group DB & OS
S48ZIS ZIS Agent – Monitoring DB & OS
(any other user) Checked through LDAP DB / OS

28. Kernel Parameters (Doc according link)
28
December 2020
Migration DB2 to EDB - Project Experience
T-Shirt sizes & clear rules?
https://www.postgresql.org/docs/10/kernel-resources.html

29. Kernel Parameters Rules (1/2)
29
December 2020
Migration DB2 to EDB - Project Experience
 kernel.shmmni = 32768
 kernel.shmmin = 1
 kernel.shmmax = max(RAM / 2 (bytes), 2147483648)
 kernel.shmall = max(shmmax / 4096, 2097152)
 kernel.sem = 250 256000 100 32768
 fs.aio-max-nr = 4194304
 kernel.sysrq = 0
 kernel.core_uses_pid = 1
 kernel.msgmni = 131072
 kernel.msgmax = 65536
 kernel.msgmnb = 65536
 vm.overcommit_memory = 2
 vm.overcommit_ratio = 80

30. Kernel Parameters Rules (2/2)
30
December 2020
Migration DB2 to EDB - Project Experience
 vm.swappiness = 1
 vm.dirty_ratio = 2
 vm.dirty_background_ratio = 1
 vm.dirty_expire_centisecs = 500
 vm.dirty_writeback_centisecs = 250
 vm.nr_hugepages = <%= (@memorysize_mb.to_i / 4).to_s %>
 fs.file-max = 512000
 net.ipv4.ip_local_port_range = 1024 65000
 net.core.rmem_default = 1048576
 net.core.wmem_default = 262144
 net.core.rmem_max = 1048536
 net.core.wmem_max = 262144

31. Shell Limits (Doc)
31
December 2020
Migration DB2 to EDB - Project Experience
 The PostgreSQL server uses one process per connection so you
should provide for at least as many processes as allowed
connections, in addition to what you need for the rest of your
system. This is usually not a problem but if you run several
servers on one machine things might get tight.
 The factory default limit on open files is often set to “socially
friendly” values that allow many users to coexist on a machine
without using an inappropriate fraction of the system resources.
If you run many servers on a machine this is perhaps what you
want, but on dedicated servers you might want to raise this
limit.

32. Shell Limits Rules
32
December 2020
Migration DB2 to EDB - Project Experience
 @enterprisedb soft nproc 262144
 @enterprisedb hard nproc 262144
 @enterprisedb soft nofile 65536
 @enterprisedb hard nofile 65536
 @enterprisedb soft stack unlimited
 @enterprisedb hard stack unlimited
 @enterprisedb soft core 6291456
 @enterprisedb hard core 6291456
 @enterprisedb soft sigpending 256904
 @enterprisedb hard sigpending 256904
 @enterprisedb soft memlock <%= (@memorysize_mb.to_i * 900).to_s %>
 @enterprisedb hard memlock <%= (@memorysize_mb.to_i * 900).to_s %>

33. Postgresql.conf (RAM)
33
December 2020
Migration DB2 to EDB - Project Experience
VAL_SHARED_BUFFERS=RAM / 4
VAL_MAX_CONNECTIONS=1000
VAL_BUFSZ=8192
VAL_WORK_MEM=(shared_buffers / 4) / max_connections +
768k / BUFSZ
VAL_EFFECTIVE_CACHE_SIZE=RAM *0,75
VAL_MAINTENANCE_WORK_MEM = 5% RAM
huge_pages=on
# https://www.postgresql-archive.org/Auto-tuning-work-mem-and-maintenance-work-
mem-td5773852.html
# https://wiki.postgresql.org/wiki/Tuning_Your_PostgreSQL_Server
# http://pgconfigurator.cybertec.at/
# https://www.postgresql.org/docs/current/runtime-config-resource.html

34. Postgresql.conf (excerpt)
34
December 2020
Migration DB2 to EDB - Project Experience
stats_temp_directory = '/data/edb/pg_stat_temp‘
archive_command= 'test ! -f ${DIR_ARCLOG}/%f && cp %p ${DIR_ARCLOG}/%f‘
wal_sync_method = fdatasync
lc_messages = 'en_US.UTF8' # locale for system error
message strings
max_worker_processes = ${MIN_WORKER_PROCESSES} # must be set
to 8 minimum
max_parallel_workers_per_gather = ${HALF_NUMBER_OF_CORES}
max_parallel_workers = ${MIN_WORKER_PROCESSES}

35. Filesystems
35
December 2020
Migration DB2 to EDB - Project Experience
Name Purpose FS/Dir/RAM Size
/ Root directory Filesystem 6 GB
/home Home directory Filesystem 100 MB
/var/log Systemlog Filesystem 4 GB
/var/log/commvault Commvault Error Logs Filesystem 2 GB
/usr/edb/as10 EDB Binaries Filesystem 500 MB
/data/db/scripts Scripts Filesystem 500 MB
/data/commvault Commvault Temp mount Filesystem 35 GB
/data/edb/db DB files Filesystem Handover HIERA
/data/edb/pg_stat_temp Temp EDB statistics RAM drive 500 MB
/var/log/EDB_Logs Script Logs Filesystem 1 GB
/data/edb/db/arclog Archived WAL Logs Filesystem 10% of DB-Obj.
/opt/commvault Commvault Filesystem 6 GB
/data/edb/db/pg_wal WAL (Transaction) Logs Directory
/var/log/edb/as10 Error logs EDB Directory
/var/log/efm-3.4 Error Logs EFM Directory
/var/log/pem Error Logs PEM Agent Directory

36. Security pg_hba.conf
36
December 2020
Migration DB2 to EDB - Project Experience
 By default, only local connections are allowed for postgres
 extend entries in pg_hba.conf what is allowed.
 We allow certain networks, 10.48, 10.248, …
 as enterprisedb
 or LDAP authenticated users
 We allow replication with special user s48deploy & efm (HA- P/M/S)
 Future SHA-256, TDE

37. Pg_hba.conf (1/3)
37
December 2020
Migration DB2 to EDB - Project Experience
IP_PRIMARY=$(nslookup ${PRIMARY_HOST} |grep Address | awk 'NR==2{print $2}')
IP_PRIMARY=$(getent ahostsv4 ${PRIMARY_HOST} | awk 'NR==1{ print $1 }')
NETWORK_CLASS=$(ip addr | grep $IP_PRIMARY | cut -d/ -f 2 | awk '{print $1}')
MY_NETWORK=$(ipcalc ${IP_PRIMARY}/${NETWORK_CLASS} -n| cut -d= -f 2)
# Allow pemagent to use local connection without password
host all ${UNIX_EDB_USER} ${IP_PRIMARY}/32 trust
host all ${UNIX_EDB_USER} ${IP_STANDBY}/32 trust

38. Pg_hba.conf (2/3)
38
December 2020
Migration DB2 to EDB - Project Experience
# database user Entry for standby
host replication ${THE_REPLICATION_DB_USER} ${IP_STANDBY}/32 md5
host replication ${THE_REPLICATION_DB_USER} ${IP_PRIMARY}/32 md5
# Entries for efm must come before the LDAP entry
host all ${EFM_USER} ${IP_PRIMARY}/32 md5
host all ${EFM_USER} ${IP_STANDBY}/32 md5
host all ${EFM_USER} ${IP_WITNESS}/32 md5

39. Pg_hba.conf (3/3)
39
December 2020
Migration DB2 to EDB - Project Experience
# We trust all connections as enterprisedb
host all ${UNIX_EDB_USER} ${MY_NETWORK}/${NETWORK_CLASS} md5
host replication ${UNIX_EDB_USER} ${MY_NETWORK}/${NETWORK_CLASS} md5
# Remote Connections are authenticated in OpenLDAP (if not enterprisedb)
host all all ${MY_NETWORK}/${NETWORK_CLASS} ldap ldapserver=ldap-ro.company.com ldaptls=1
ldapbinddn="uid=USERNAME,ou=users,ou=technical,ou=accounts,ou=linux,dc=ms,dc=DOMAIN,dc=de"
ldapbindpasswd=PASSWORD ldapbasedn="ou=accounts,ou=linux,dc=ms,dc=DOAMIN,dc=de"