Presentation from AWS Summit London 2023 In this session CDL an InsurTech from the UK will walk through the entire journey of real-life large-scale database migration from Oracle on premise to PostgreSQL in AWS. We’ll cover the process and share unique details, hits and tips covering zero downtime database migration, change data capture and schema conversion. We’ll show how we made the databases not only cheaper by removing licence costs but also how we sped up our development teams by proving self-service features and made our applications run faster. Finally, we’ll share the extra features we built for PostgreSQL with the AWS Database Freedom team that were missing and how we open sourced these.

1. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved.
CDL - EXTERNAL
© 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved.
15:30
Migrating 600 databases to AWS and making
them better, faster, and cheaper
A W S S U M M I T L O N D O N 2 0 2 3
Matt Houghton (he/him)
Data Architect
CDL

2. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved.
CDL - EXTERNAL
© 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Migrating 600 databases to AWS
and making them better, faster,
and cheaper
A W S S U M M I T L O N D O N 2 0 2 3
Matt Houghton (he/him)
Data Architect
CDL

3. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved.
CDL - EXTERNAL

4. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved.
CDL - EXTERNAL
High availability

5. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved.
CDL - EXTERNAL
Enterprise-grade database

6. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved.
CDL - EXTERNAL
Hybrid cloud
Primary Secondary
Read-Only

7. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved.
CDL - EXTERNAL
Move up the stack
• Learn by failing
• Undifferentiated heavy lifting
• Focus on the business problem

8. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved.
CDL - EXTERNAL
Migration steps
• AWS Schema Conversion Tool
(AWS SCT)
• Change data capture (CDC)
• Database objects
• Database code
• Engine-specific functions
• Code/rewrite/rearchitect

9. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved.
CDL - EXTERNAL
Change data capture

10. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved.
CDL - EXTERNAL
Demo

11. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved.
CDL - EXTERNAL
Event-driven
-

12. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved.
CDL - EXTERNAL
Migration monitoring

13. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved.
CDL - EXTERNAL
Automation
• Infrastructure as code
• Parameterize
• Switchover
• Rollback
• Reduce risk
• Consistency
• Speed of deployment

14. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved.
CDL - EXTERNAL
Tuning
• join_collapse_limit
• from_collapse_limit
• work_mem
• pg_stat_statements
• pg_prewarm
• Long-running SQL
• Tools
• The organization

15. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved.
CDL - EXTERNAL
What do the DBAs do now?
• Infrastructure as code
• More automation
• Config-driven deployments
• Faster upgrades/fixes
• Scale the business

16. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved.
CDL - EXTERNAL
Developer experience
• Self-service
• Database clones
• Privacy automation
• Speed
• Preapproved queries
• Health checks
• Database-as-a-service product
• New skills

17. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved.
CDL - EXTERNAL
© 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Amazon RDS Ready

18. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved.
CDL - EXTERNAL
Connection pooling

19. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved.
CDL - EXTERNAL
Application logging

20. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved.
CDL - EXTERNAL
Data encryption

21. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved.
CDL - EXTERNAL
Database failure and performance

22. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved.
CDL - EXTERNAL
Open source

23. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved.
CDL - EXTERNAL
Fast refresh materialized views

24. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved.
CDL - EXTERNAL
Fast refresh materialized views – Walkthrough

25. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved.
CDL - EXTERNAL
Fast refresh materialized views – Walkthrough

26. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved.
CDL - EXTERNAL
Fast refresh materialized views – Walkthrough

27. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved.
CDL - EXTERNAL
Fast refresh materialized views – Walkthrough

28. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved.
CDL - EXTERNAL
Demo

29. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved.
CDL - EXTERNAL
Final tip

30. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved.
CDL - EXTERNAL
Thank you!
Matt Houghton
@mattdevdba