Weitere ähnliche Inhalte Ähnlich wie Lessons from Migrating Oracle Databases to Amazon RDS or Amazon Aurora (20) Kürzlich hochgeladen (20) Lessons from Migrating Oracle Databases to Amazon RDS or Amazon Aurora 1. © 2020, Amazon Web Services, Inc. or its Affiliates.© 2020, Amazon Web Services, Inc. or its Affiliates.
Shailesh Rangani, Cloud Database Engineer, Datavail
Lessons from Migrating
Oracle Databases to Amazon
RDS or Amazon Aurora
2. © 2020, Amazon Web Services, Inc. or its Affiliates.
Shailesh Rangani
• Shailesh Rangani is a Cloud Database Engineer with 15
years’ experience in database domain.
• He holds various certifications on cloud platforms like
AWS, Azure and OCI along with database platforms like
PostgreSQL, MongoDB, Oracle and DB2 LUW.
• He is an expert in the design, deployment,
administration, and management of data-intensive
applications that enable organizations to effectively
analyze and process large volumes of structured and
unstructured data.
• Shailesh specializes in Cloud platforms and DBMS
technologies. He has successfully delivered the data
architecture strategy for projects and large-scale
platforms.
Shailesh Rangani
Cloud Database Engineer, Datavail
3. © 2020, Amazon Web Services, Inc. or its Affiliates.
About Datavail
Databases
• Oracle
• SQL Server
• MySQL
• MariaDB
• PostgreSQL
• Cassandra
• MongoDB 100+ Cloud SA’s
and Engineers
16+ Years
Database Services
700+ Customers
8+ Years AWS
Cloud Experience
200,000+
Databases Managed
150+ AWS
Migrations and
Modernizations
Expertise Experience Outcome
Analytics and Data
Integration
4. © 2020, Amazon Web Services, Inc. or its Affiliates.© 2020, Amazon Web Services, Inc. or its Affiliates.
• Why break free? The business case for
moving to AWS databases
• Common migration paths to
modernization
• Introduction to Amazon Aurora
• Database migration
• Best practices
• Hands on lab (Oracle to Aurora PostgreSQL)
Agenda
5. © 2020, Amazon Web Services, Inc. or its Affiliates.© 2020, Amazon Web Services, Inc. or its Affiliates.
Fill Out Cloud
Adoption
Industry
Benchmark
Survey
Fill out the survey for a chance to WIN
a pair of our cool Bose noise cancelling
headphones!
https://www.datavail.com/cloud-survey/
6. © 2020, Amazon Web Services, Inc. or its Affiliates.© 2020, Amazon Web Services, Inc. or its Affiliates.
Flexible Ways to Break Free
7. © 2020, Amazon Web Services, Inc. or its Affiliates.
A typical database migration lifecycle
Data
migration
Application
code
Schema
objects
Cutover
Analysis
Testing
8. © 2020, Amazon Web Services, Inc. or its Affiliates.
We offer flexible ways to help you migrate
AWS Migration
Tools
AWS ProServe and
Migration Partners
AWS Database
Freedom Program
9. © 2020, Amazon Web Services, Inc. or its Affiliates.
Flexible, powerful migration tooling
Most sources and targets, higher conversion automation
Source DB or DW AWS SCT Destination DB or DW
Source DB or DW Destination DB or DWAWS DMS
Copy or convert
Data
schema
Step 1: Convert or copy your schema
Step 2: Move your data
10. © 2020, Amazon Web Services, Inc. or its Affiliates.
AcceleratedAmazon Aurora Migration
Case Study 1 – Midsized Utility Firm
• Single monolithic DB
• Lot of historical data
• Oracle licensing cost
• Encryption (TDE)
• Scalability
• Credential rotation
• Quick testing environment
• Native utilities for static
data migration
• AWS Database Migration
Service (DMS) & AWS
Schema ConversionTool
• Databases runningAurora
PostgreSQL
• Encryption via KMS
• Auto Scaling (CPU based)
• Native partitions
• Database clone for quick
test environment
Challenges Accelerated Migration Outcome
AWS Schema
ConversionTool
AWS Database
Migration Service
Oracle instance Aurora PostgreSQL
11. © 2020, Amazon Web Services, Inc. or its Affiliates.
Case Study 2 – Midsized Financial Company
• Oracle licensing cost
• Multiple Sources
• Heavy read intensive workload
over weekend
• 150,000 lines of code to convert
• 3000 + objects to convert
• AWS Database Migration
Service (DMS) & AWS Schema
ConversionTool
• Expert PostgreSQL
developers/AWS SA
• Code conversion 2 months
• RegressingTesting
• Databases runningAurora
PostgreSQL
• Faster reports
• Lower licensing costs
• IAM authentication at DB level
• DB code moved to app layer
• Auto Scaling (scheduled
based)
Challenges Accelerated Migration Outcome
AcceleratedAmazon Aurora Migration
AWS Schema
ConversionTool
AWS Database
Migration Service
Oracle instance Aurora PostgreSQL
12. © 2020, Amazon Web Services, Inc. or its Affiliates.© 2020, Amazon Web Services, Inc. or its Affiliates.
Introduction to Amazon Aurora
13. © 2020, Amazon Web Services, Inc. or its Affiliates.
Benefits of Amazon Aurora
AmazonAurora
Speed and availability of high-end commercial databases
Simplicity and cost-effectiveness of open source
databases
Drop-in compatibility with MySQL and PostgreSQL
Simple pay as you go pricing
So, we designed Amazon Aurora - enterprise database at open source price,
delivered as a managed service
14. © 2020, Amazon Web Services, Inc. or its Affiliates.
Scale-out, distributed, multi-tenant storage architecture
• Purpose-built log-
structured distributed
storage
• Storage volume is striped
across hundreds of
storage nodes
• Storage nodes with locally
attached SSDs
• Continuous backup to
Amazon S3. AZ 1 AZ 2 AZ 3
SHAREDCLUSTER STORAGEVOLUME
Writer
Transactions
Caching
SQL
Cluster
Endpoint
Amazon S3
Reader
Transactions
Caching
SQL
Reader
Transactions
Caching
SQL
Reader
Endpoin
t
15. © 2020, Amazon Web Services, Inc. or its Affiliates.
Leverages the AWS Cloud ecosystem
AWS Lambda Amazon S3 AWS IAM
Amazon
CloudWatch
Invoke AWS Lambda events
from stored
procedures/triggers
Load from, save to Amazon
S3, store snapshots and
backups in S3
Use AWS Identity &
Access Management
(IAM) roles to manage
database access control
Upload systems metrics and audit logs to
Amazon CloudWatch
Amazon Sagemaker
Amazon Comprehend
Make inferences directly from your database
using SQL calls
16. © 2020, Amazon Web Services, Inc. or its Affiliates.
Automates administrative tasks
Schema design
Query construction
Query optimization
Automatic fail-over
Backup & recovery
Isolation & security
Industry compliance
Push-button scaling
Automated patching
Advanced monitoring
Routine maintenance
Takes care of your time-
consuming database
management tasks, freeing you
to focus on your applications and
business
YOU
AWS
17. © 2020, Amazon Web Services, Inc. or its Affiliates.
Who is adopting Amazon Aurora?
• Higher performance
• Better availability
and durability
• Easy migration; no
application change
• One-tenth of the
cost; no licenses
• Comparable
performance and
availability
• Migration tooling and
services
• Integration with
other
AWS services
• Cloud-native
capabilities and
access
mechanisms
• Scalability,
availability,
managed service
Customers migrating from
open source PostgreSQL
Customers using
commercial DB engines
Customers building new
applications
18. © 2020, Amazon Web Services, Inc. or its Affiliates.© 2020, Amazon Web Services, Inc. or its Affiliates.
Schema Conversion
19. © 2020, Amazon Web Services, Inc. or its Affiliates.
AWS Schema ConversionTool
• Assessment report
• Project interface
• Code browser
• Automates many conversion tasks
Packages
Stored procedures
Functions
Triggers
User defined types
Schemas
Tables
Indexes
Views
Sequences
Synonyms
Tab with the assessment report Manual conversion tips
Side by side code view
20. © 2020, Amazon Web Services, Inc. or its Affiliates.
AWS Schema Conversion Tool tips
Save as an SQL file
• Allows you to apply only
Table DDL & PK
• Save the secondary DDL for
after migration
Memory management
• Global settings
• JVM settings
21. © 2020, Amazon Web Services, Inc. or its Affiliates.
SCT migration assessment report
Connect SCT to “source”
& “target”
Run “assessment report”
Read “executive
summary” & follow
detailed instructions
More: https://docs.aws.amazon.com/SchemaConversionTool/latest/userguide/CHAP_AssessmentReport.Create.html
22. © 2020, Amazon Web Services, Inc. or its Affiliates.
SQL scripts
Packages
Stored procedures
Functions
Triggers
User defined types
Schemas
Tables
Indexes
Views
Sequences
Synonyms
SCT does a great job of converting
your schema and code objects
Users, roles, grants
https://aws.amazon.com/blogs/database/use-sql-to-map-users-roles-and-grants-from-oracle-to-postgresql/
23. © 2020, Amazon Web Services, Inc. or its Affiliates.
Oracle to Amazon
Aurora PostgreSQL
Migration Playbook –
Example
More:
https://d1.awsstatic.com/whitepapers/Migration/oracle-
database-amazon-aurora-postgresql-migration-
playbook.pdf
24. © 2020, Amazon Web Services, Inc. or its Affiliates.© 2020, Amazon Web Services, Inc. or its Affiliates.
Data Migration
25. © 2020, Amazon Web Services, Inc. or its Affiliates.
Move Data using Data Migration Service
Oracle Databases AWS Database
Migration Service
Amazon Aurora
AWS DMS Replication
Instances
AWS DMS
ReplicationTasks
Read Write
26. © 2020, Amazon Web Services, Inc. or its Affiliates.
Support for the following conversions
Source* Database Target* Database on AWS
Oracle database Amazon Aurora, MySQL, PostgreSQL, Oracle
Oracle data warehouse Amazon Redshift
Azure SQL Amazon Aurora, MySQL, PostgreSQL
Microsoft SQL Server Amazon Aurora, Amazon Redshift, MySQL PostgreSQL
Teradata Amazon Redshift
IBM Netezza Amazon Redshift
Greenplum Amazon Redshift
HPEVertica Amazon Redshift
MySQL and Maria DB PostgreSQL
PostgreSQL Amazon Aurora, MySQL
Amazon Aurora PostgreSQL
IBM DB2 LUW Amazon Aurora, MySQL, PostgreSQL
Apache Cassandra Amazon DynamoDB
Source: https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Source.html Target: https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Target.html
27. © 2020, Amazon Web Services, Inc. or its Affiliates.
AWS Database Migration Service
Working with DMS components
• Replication Instance
‒ C - compute R - memory T - burstable
• Endpoint
‒ Connect source / target from replication instance
‒ Choose advanced override settings
• Task
‒ Full Load
‒ Change Data Capture
28. © 2020, Amazon Web Services, Inc. or its Affiliates.
AWS Database Migration Service
Best Practices choosing a replication instance
• How to choose between different instance types.
• Things to know about EBS storage on replication instance
• Important Amazon CloudWatch metrics to watch on the replication
instance
29. © 2020, Amazon Web Services, Inc. or its Affiliates.
AWS Database Migration Service
Best Practices Designing the source & target endpoint
• Oracle (source)
‒ When to use ‘binary reader’
‒ When to use ‘log miner’
‒ Using NumberDataTypeScale
‒ Using standbyDelayTime
• Aurora PostgreSQL (target)
‒ Using ‘maxFileSize’
30. © 2020, Amazon Web Services, Inc. or its Affiliates.
AWS Database Migration Service
Components of a DMS task
• Choosing a “migration type”
‒ Full Load
‒ Change Data Capture (CDC)
• Choosing a target table prep mode
‒ DMS can create tables, however, better pre-create table with SCT
‒ Flexibility to drop or truncate tables.
• Include LOB columns in replication
‒ Why limited LOB (Large Object) mode is a better choice?
• Selection & transformation rules, logging, exceptions, and others
‒ Flexible in selection but limited transformations
‒ How to work around transformations
31. © 2020, Amazon Web Services, Inc. or its Affiliates.
Task – Migration type
Choose migration
type
Existing data
Existing data
and replicate
changes
Replicate
changes only
Creates files or tables in the target database
Populates the tables with data from the source
Migrate existing data option in the AWS console and Full Load in the API
Captures changes on the source during migration
Once initial migration completes, changes are applied to the target as units of
completed transactions
Migrate existing data and replicate ongoing changes option in the AWS console
and full-load-and-cdc in the API.
Reads the recovery file on the source database
Groups together transactions and applies them to the target. Buffering as needed
Replicate data changes only option in the AWS console
32. © 2020, Amazon Web Services, Inc. or its Affiliates.
Task – Target preparation
Target
preparation
Do nothing
Drop tables on
target
Truncate
In Do nothing mode, AWS DMS assumes target tables are pre-created.
In full load or full load plus CDC, ensure that the target tables are empty
before starting the migration.
In Drop tables on target mode, AWS DMS drops the target tables and
recreates them before starting the migration.This ensures that the target
tables are empty when the migration starts.
In Truncate mode, AWS DMS truncates all target tables before
the migration starts.
33. © 2020, Amazon Web Services, Inc. or its Affiliates.
Task – Include LOBs
Include LOBs
Don't include
LOBs
Full LOB
mode
Limited LOB
mode*
LOB columns are excluded from the migration.
Migrate complete LOBs regardless of size.AWS DMS migrates LOBs
piecewise in chunks controlled by the Max LOB size parameter.This mode
is slower than using Limited LOB mode.
Truncate LOBs to the value of the Max LOB size parameter.This mode
is faster than using Full LOB mode.
34. © 2020, Amazon Web Services, Inc. or its Affiliates.
Task – Selection and Transformation rules
35. © 2020, Amazon Web Services, Inc. or its Affiliates.
AWS Database Migration Service
AWS Cloud
VPCCustomer On-premises
VPN
Oracle Aurora
PostgreSQL
DMS Replication
Instance
Application users
• Start a replication instance
• Connect the source and target endpoints
• DMS FULL LOAD the data from Source to
Target• DMS Change Data Capture to replicate
ongoing changes
• At steady state – take an outage, validate &
redirect connection
End Point Connection
Full Load
Change Data
Capture
36. © 2020, Amazon Web Services, Inc. or its Affiliates.© 2020, Amazon Web Services, Inc. or its Affiliates.
Best Practices for Production
Rollout
37. © 2020, Amazon Web Services, Inc. or its Affiliates.
PostgreSQL is a lowercase
data dictionary
Use “exception handlers”
when needed, not by default
PostgreSQL has six
different index types.
Store your BLOBs in
Amazon S3 instead of the
database
search_path replaces
PUBLIC SYNONYM
PostgreSQL
has 64 datatypes
Understand basic database engine differences
38. © 2020, Amazon Web Services, Inc. or its Affiliates.© 2020, Amazon Web Services, Inc. or its Affiliates.
Overview of migration steps
39. © 2020, Amazon Web Services, Inc. or its Affiliates.
Start full load
Source TargetReplication
Instance
40. © 2020, Amazon Web Services, Inc. or its Affiliates.
While loading data also capture changes
Source TargetReplication
Instance
App
41. © 2020, Amazon Web Services, Inc. or its Affiliates.
Load complete - apply captured changes
Source TargetReplication
Instance
App
42. © 2020, Amazon Web Services, Inc. or its Affiliates.
Changes reach steady state
Source TargetReplication
Instance
App
43. © 2020, Amazon Web Services, Inc. or its Affiliates.
Shutdown apps and apply remaining changes
Source TargetReplication
Instance
App
44. © 2020, Amazon Web Services, Inc. or its Affiliates.
Change your application endpoint
Source TargetReplication
Instance
App
45. © 2020, Amazon Web Services, Inc. or its Affiliates.© 2020, Amazon Web Services, Inc. or its Affiliates.
Fill Out Cloud
Adoption
Industry
Benchmark
Survey
Fill out the survey for a chance to WIN
a pair of our cool Bose noise cancelling
headphones!
https://www.datavail.com/cloud-survey/
46. © 2020, Amazon Web Services, Inc. or its Affiliates.© 2020, Amazon Web Services, Inc. or its Affiliates.
Workshop Architecture
47. © 2020, Amazon Web Services, Inc. or its Affiliates.
DMS Workshop - Instructions for Hands-on Lab
• Workshop Instructions https://dms-immersionday.workshop.aws/en/oracle-aurora.html
• Account Access: https://dashboard.eventengine.run
• Region – US-West-2 (Oregon)
48. © 2020, Amazon Web Services, Inc. or its Affiliates.
Lab Architecture
VPC
AWS Cloud
Windows
Remote Desktop
User
Amazon EC2
AWS Schema Conversion
Tool (SCT)
Source
Amazon RDS
(Oracle)
AWS Database
Migration Service
Target
Amazon Aurora
RDS (PostgreSQL)
49. © 2020, Amazon Web Services, Inc. or its Affiliates.
Lab Instructions
Part 1: (Approx. time 1 hour)
• Remote desktop to EC2
• Install Schema Conversion Tool (SCT)
• Create a SCT project
• Convert the Oracle Schema
Part 2: (Approx. time 1hour 30
min)
• Connect to Source (Oracle) and
configure DMS access
• Connect to Target (Aurora PG) and
prepare for DMS load
• Create Replication Instance
• Create Source and Target
Endpoints
• Create DMS Task
• Verify the data
50. © 2020, Amazon Web Services, Inc. or its Affiliates.© 2020, Amazon Web Services, Inc. or its Affiliates.
Appendix
51. © 2020, Amazon Web Services, Inc. or its Affiliates.
Migrate & Modernize Your Databases with Datavail &
AWS
• Break free from commercial databases with migrations and modernizations to
the AWS Cloud with Datavail as your trusted partner.
• Contact us to learn more about our AWS Cloud capabilities and to set up a
discovery workshop at no cost.
• Additional Datavail resources:
• Transition from Commercial Databases to Amazon Web Services
• AWS Database Migration Jump Start for Aurora
• 10 Best Practices to Secure PostgreSQL AWS RDS/Aurora
52. © 2020, Amazon Web Services, Inc. or its Affiliates.
AWS Database Partners
Migration SI Partners License Advisory Partners
53. © 2020, Amazon Web Services, Inc. or its Affiliates.© 2020, Amazon Web Services, Inc. or its Affiliates.
Purpose-built Databases
aws.amazon.com/products/databases/
Database Migration Service
aws.amazon.com/dms/resources/
Database Freedom
aws.amazon.com/solutions/databasemigrations/database-freedom/
AWS Database Migration Partners
aws.amazon.com/dms/partners/
Migration Resources Library
aws.amazon.com/solutions/database-migrations/resources/
Other resources for you to leverage
Hinweis der Redaktion Datavail is in process of doing a benchmark to determine where organizations are in their cloud adoption with a cloud adoption industry benchmark survey. The results of the survey will be shared on a future Database Trends & Applications (DBTA) webinar and you’ll be able to learn how your company compares to your peers in the industry on cloud adoption. We’ll be selecting a winner from our lab today to win a pair of Bose noise cancelling headphones. As organizations look to migrate to the cloud, there are 3 core types of database migration paths:
Self-service: For many migrations, the self-service path through the use of the Database Migration Service (DMS) and Schema Conversion Tool (SCT) offers the tools necessary to execute With over 200,000 migrations completed through DMS, customers have successfully migrating their instances to AWS.
Migration partners: For assistance in the migration process, there are several migration partners that offer expertise to
Database Freedom program – this type of migration is the best for customers looking to move away from the punitive licensing costs of commercial database vendors and avoiding the vendor lock-in. This is Database Freedom. Most of these migrations have been from Oracle and SQL Server to our open source databases and Aurora but there are use cases for migrating to NoSQL databases as well. For example, an online store may have started on a commercial or open source database but now is growing so fast that they would need a NoSQL database like DynamoDB to scale to millions of transactions per minute. Re-factoring, however, typically requires application changes and does more time to migrate than the other migration methods. Using the Database Migration Service (DMS), you can make homogeneous migrations from your legacy database service to a managed service on AWS, such as from Oracle to RDS Oracle.
Alternatively, leveraging both DMS and the Schema Conversion Tool (SCT), heterogeneous conversions are possible, such as converting from SQL Server to Amazon Aurora.
The Schema Conversion Tool assesses the source compatibility and recommends the best target engine. The tool attempts conversion of all schema and code objects to the target engine, including stored procedures and functions. It scans and converts embedded SQL statements in app code, and generates a report with recommendations. Level 200 Level 300 Level 200
Level 200
Level 200 It wasn’t possible to backfill without tooling.. We found DMS
Backfill process involves moving data from Oracle to DynamoDB without invoking the application
Backfill using DMS involves creation of replication instances, tasks and endpoints
DMS configurations are used to specify source, destination, attribute mapping, conditional expressions for writing to DDB etc… Log miner (D) Oracle utility (Impact on the Source) , ** CDC with BLOB and CDC 12C
Binary reader Copy the redo on to DMS and we mine the redo log (Decrease cpu, memory on On source but Network usage)
NumberDataTypeScale 38 precision, pg -- > 128
Standbydelaytime Delayed replication
maxFileSize mine the data and creates csv files
Session replication role = replica
session_replication_role='replica’ FK disable
LOB columns are excluded from the migration.
Full Lob Mode - Migrate complete LOBs regardless of size. AWS DMS migrates LOBs piecewise in chunks controlled by the Max LOB size parameter. This mode is slower than using Limited LOB mode.
Limited Lob Mode - Truncate LOBs to the value of the Max LOB size parameter. This mode is faster than using Full LOB mode.
In Limited LOB mode, LOB columns which exceed Maximum LOB size will be truncated to “Max LOB size”
EM EM EM EM EM EM Level 200