Migrating Your Databases to AWS: Deep Dive on Amazon RDS and AWS Database Migration Service - AWS Summit Cape Town 2017
•
7 gefällt mir•1,313 views
Amazon RDS allows you to launch an optimally configured, secure and highly available database with just a few clicks. It provides cost-efficient and resizable capacity, automates time-consuming database administration tasks, and provides you with six familiar database engines to choose from: Amazon Aurora, Oracle, Microsoft SQL Server, PostgreSQL, MySQL and MariaDB. In this session, we will take a close look at the capabilities of Amazon RDS and explain how it works. We’ll also discuss the AWS Database Migration Service and AWS Schema Conversion Tool, which help you migrate databases and data warehouses with minimal downtime from on-premises and cloud environments to Amazon RDS and other Amazon services. Gain your freedom from expensive, proprietary databases while providing your applications with the fast performance, scalability, high availability, and compatibility they need. AWS Speaker: Andrew Kane, Solutions Architect - Amazon Web Services
Empfohlen
Empfohlen
Weitere ähnliche Inhalte
Was ist angesagt?
Was ist angesagt? (20)
Ähnlich wie Migrating Your Databases to AWS: Deep Dive on Amazon RDS and AWS Database Migration Service - AWS Summit Cape Town 2017
Ähnlich wie Migrating Your Databases to AWS: Deep Dive on Amazon RDS and AWS Database Migration Service - AWS Summit Cape Town 2017 (20)
Mehr von Amazon Web Services
Mehr von Amazon Web Services (20)
Kürzlich hochgeladen
Kürzlich hochgeladen (20)
Migrating Your Databases to AWS: Deep Dive on Amazon RDS and AWS Database Migration Service - AWS Summit Cape Town 2017
- 1. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Dr. Andrew Kane, Solutions Architect July 5, 2017 Migrating Your Databases to AWS: Deep Dive on Amazon Relational Database Service and AWS Database Migration Service
- 2. Agenda The WHAT The WHY The WHEN The HOW The WHO
- 3. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. The What : Amazon Relational Database Service
- 4. Amazon Relational Database Service (RDS) Scale up/down Cost-effective Instant provisioning Application compatibility No infrastructure management
- 5. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. The Why : Amazon Relational Database Service
- 6. • Multi-engine support: Amazon Aurora, MySQL, MariaDB, PostgreSQL, Oracle, SQL Server • Automated provisioning, patching, scaling, backup/restore, failover • High availability with RDS Multi-AZ – 99.95% SLA for Multi-AZ deployments Highlights of Amazon RDS
- 7. Security
- 8. Amazon Virtual Private Cloud (VPC) Securely Control Network Configuration Availability Zone AWS Region 10.1.0.0/16 10.1.1.0/24 Manage connectivity AWS Direct Connect VPN Connection VPC Peering Internet Gateway Routing Rules
- 9. Security groups Database IP firewall protection Protocol Port Range Source TCP 3306 172.31.0.0/16 TCP 3306 “Application security group” Corporate address admins Application tier
- 10. Encryption in Transit Available for all six engines Database traffic encryption with SSL/TLS
- 11. Encryption At Rest • DB instance storage • Automated backups • Read Replicas • Snapshots • Available for all six engines • No additional cost • Support compliance requirements
- 14. Minimal Deployment - Single AZ Availability Zone AWS Region 10.1.0.0/16 10.1.1.0/24 Amazon Elastic Block Store volume PROD Env DEV Env
- 15. High Availability - Multi-AZ Availability Zone A AWS Region 10.1.0.0/16 10.1.1.0/24 Availability Zone B 10.1.2.0/24 Synchronous replication Same instance type as master
- 16. High Availability – Multi-AZ to DNS Availability Zone A AWS Region Availability Zone B M S DNS mydatabase.eu-west-2.rds.amazonaws.com:3306 M
- 17. High Availability – Amazon Aurora Nodes • Cluster contains primary node and up to 15 secondary nodes • Customer application can scale out read traffic across secondary nodes • Failing nodes are automatically detected and replaced • Failing database processes are automatically detected and recycled • Secondary nodes automatically promoted on persistent outage AZ 1 AZ 3AZ 2 Primary Node Primary Node Primary Node Primary Node Primary Node Secondary Node Primary Node Primary Node Secondary Node
- 18. High Availability – Amazon Aurora Storage • Storage volume automatically grows up to 64 TB • 10 GB segments as unit of repair or hotspot rebalance • Continuous backup to Amazon S3, and continuous monitoring of disk for repair • Can withstand loss of whole AZ or 2 storage nodes without any loss of client availability • Peer-to-peer gossip replication to fill in holes, 10Gb storage segments restore in parallel AZ 1 AZ 2 AZ 3 Amazon S3
- 19. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. The How : Getting onto Amazon Relational Database Service
- 21. What are DMS and SCT? AWS Database Migration Service (DMS) easily and securely migrates and/or replicate your databases and data warehouses to AWS AWS Schema Conversion Tool (SCT) converts your commercial database and data warehouse schemas to open-source engines, Amazon Aurora and Redshift. Converts and loads data warehouse data into Amazon Redshift We have migrated over 28,000 unique databases. And counting…
- 22. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. The When : AWS Database Migration Service
- 23. When to use DMS and SCT? Modernize Migrate Replicate Modernise your database tier – • Commercial to open-source • Commercial to Amazon Aurora • Open-source to Amazon Aurora Modernise your Data Warehouse – • Commercial to Redshift • Migrate business-critical applications • Migrate from Classic to VPC • Migrate data warehouse to Redshift • Upgrade to a minor version • Consolidate shards into Aurora • Create cross-regions Read Replicas • Run your analytics in the cloud • Keep your dev/test and production environment sync
- 24. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. …. But there’s more!
- 26. Fanning-Out Target Source Target Target Replication instance
- 27. Homogenous or heterogeneous SQL Server Amazon Aurora Oracle Oracle Oracle Amazon S3
- 28. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. The Why : AWS Database Migration Service
- 29. Why Use DMS and SCT? Secure Cost Effective Remove Barriers to Entry Allow DB Freedom Keep a Leg in the Cloud Easy to Use, but Sophisticated… Near-Zero Downtime
- 30. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. The How : How The AWS DMS Works
- 31. Database Migration Process Step 1: Convert or Copy your Schema Source DB or DW AWS SCT Native Tool Destination DB or DW Step 2: Move your data Source DB or DW AWS SCT Destination DB or DW AWS DMS
- 32. Customer premises Application users AWS Internet VPN Keep Your Apps Running During the Migration • Start a replication instance • Connect to source and target databases • Select tables, schemas, or databases Let AWS DMS create tables, load data, and keep them in sync Switch applications over to the target at your convenience AWS DMS
- 33. New SCT data extractors Extract Data from your data warehouse and migrate to Amazon Redshift • Extracts through local migration agents • Data is optimized for Redshift and Saved in local files • Files are loaded to an Amazon S3 bucket (through network or Amazon Snowball) and then to Amazon Redshift Amazon Redshift AWS SCT S3 Bucket
- 34. New NoSQL support Migrate to AWS • Move from MongoDB to Amazon DynamoDB • Move from MongoDB to relational db’s Move between NoSQL and SQL • Change technologies Amazon Aurora DynamoDB DynamoDB RDS
- 35. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. The Who : Customer Use Cases
- 36. Heterogeneous Migration • Oracle private DC to RDS PostgreSQL migration • Used the AWS Schema Conversion Tool to convert their database schema • Used on-going replication (CDC) to keep databases in sync until they reached the cutover window • Benefits: • Improved reliability of the cloud environment • Savings on Oracle licensing costs • SCT Assessment Report let them understand the scope of the migration ”The SCT Assessment Report was the key enabler to allow us to understand the scope of effort required to complete an Oracle to PostgreSQL migration. What was originally thought to be a largely manual task that no one was particularly excited about having to do became a very straight-forward quick and easy process."
- 37. Scale-up Migration • RDS MySQL to Amazon Aurora Migration • Used DMS with on-going replication (CDC) to migrate the data • Benefits: • Aurora handles their larger data storage requirements. Per regulations they are storing 120 TB of data for 2 years • Reduced cost and improved performance when compared to large MySQL instances “We are in the process of migrating some databases to Amazon Aurora. The ease by which we can do this using the AWS Database Migration Service has simplified this process for us and enabled us to accelerate our migration efforts. The ability to closely monitor the process, the detailed logging feature, and the support we received from AWS have given us a great deal of confidence in a successful migration.”
- 38. Homogeneous Migration • RDS MySQL on EC2 Classic to VPC • Database ran an end-user application so could not take downtime • 70 RDS instances have been migrated • Benefits: • Leveraging CDC, they could decrease outage per database to less than 5 mins • 70 RDS instances have been seamlessly migrated
- 39. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Wrapping up
- 40. Recap Minimal Operational Overhead Reliable, Scalable and Secure deployments Consistent Migrations
- 41. AWS Database Migration Partners
- 42. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Thank You! aws.amazon.com/dms
Hinweis der Redaktion
- In this session we will look at the Amazon Relational Database service (RDS), and then dive into the details behind our services dedicated to migrating your database into AWS, namely the Database Migration Service (DMS) and the Schema Conversion Tool (SCT).
- What why how when who. We're going to tackle this topic by looking at from these 5 different perspectives, so if you’re here to find the answers to all of these questions from Amazon Relational Database Service and Database Migration Service then you’re in the right place!
- RDS is a managed database service. Managed being the key word here No Infra Mgmt RDS service fully manages the host, operating system, and database version that you are running on, storage Less administrative task, so You can focus on your application Cost-effective No Cost to get started AND Pay only for what you consume Application Compatibility There are many popular applications, or even your own custom code, that you may be running on your own infrastructure and it can still work on RDS. If you are using one of the supported engines, then there is a good chance you can get them working on RDS. Instant Pro/Scale Simple and Fast to deploy and scale your RDS instances launch a new database or change your existing one at any point in time with no need to wait for infrastructure to be ordered or configured. In short, you could save yourself 10-12 weeks weeks worth of ordering and configuring to get a new DB infrastructure up and running. But with RDS, you could achieve this in a matter of minutes.
- When migrating your database to the cloud you do more than just change location. You leverage the power of the cloud. Customers appreciate the flexibility that platforms like EC2 offer, and many do run database engines there, but the RDS takes things to a WHOLE NEW level. Here’s how. AWS offers Six different engines to choose from AWS does patching, handles backup and replication, provides HA and automated failover management AWS manages the underlying RDS Infrastructure making sure that it is healthy, providing you with a large number of detailed operational metrics Provision a database in minutes Protect your backups and logs with 11 9s of durability Recover to any point in time from nightly backups + logs 99.95% => 4.38h / year You as technologist can now focus on enhancing your applications and find techniques to automate the rest of your infrastructure rather than fire-fighting
- Security at AWS is a top priority as it is with our customers. Being able to secure their database environment is very important for our customers and there are many features within RDS that will allow you to achieve the security profile you are looking for.
- When you launch an RDS instance it launches inside of Amazon Virtual Private Cloud or VPC. As some of you may be aware, Amazon VPC gives you the ability to define a private address space within AWS. You can also carve out different subnets within your VPC address space to further segment and isolate your application components, including your database. You can implement IAM policies that restrict who can do what against your RDS infrastructure – you govern who can launch, modify or terminate an instance, but it’s important to note that IAM does not govern who can login to your database. You get to control which users and applications can access your database and how they access it via functionality within your database platform. From a shared responsibility model, you are the DBA, we are SYSDBA. You have many different options to manage connectivity to your database. [click] You can use Direct Connect to link your data center to an AWS region giving you a connection with consistent over the internet or over Direct Connect [click] You can peer two different VPCs together allowing applications in one VPC to access your database in another VPC over what is essentially a private route [click] You can control the routing of your VPC using route tables that you attach to each of the subnets in your VPC [click] You can allow or disallow access to your database from the Internet via an Internet Gateway
- Within RDS you get the ability to control the flow of traffic into your database with security groups. If you have used EC2 you are probably already familiar with security groups. A Security group is similar to a virtual firewall that is placed around your database environment and you get to control what gets in and out of your database. You can control the protocol, port range, and source of the traffic that you allow into your database. For the source you can restrict it to a specific IP address, a particular CIDR block covering multiple IP addresses, or even another security group meaning that your RDS instance will only accept traffic if it comes from instances in that particular security group. This gives you the flexibility to have a multi tier archicture where you only grant connections from the parts of the tier that actually need to access the database.
- RDS gives you the ability to encrypt the traffic to and from your database using SSL. Each of the six engines supports the ability to configure an SSL connection into your database. RDS allows you to block connections that are not encrypted
- Encryption of data that is stored in a database is another important requirement for a lot of our customers. Whether it be to meet external compliance needs or internal compliance needs. In general, wherever possible AWS recommends that you encrypt your data to have as much control and protection over your data as possible. With RDS AWS provides you the ability to encrypt the data at rest for all six engines at NO additional cost When you encrypt your RDS database with AWS provided encryption that encryption covers: The Database Instance Storage Automated Backups AND Read Replicas of the data Snapshots that you generate of the database AWS Key Management Service (KMS) is a managed service that makes it easy for you to create and control the encryption keys used to encrypt your data, and uses Hardware Security Modules (HSMs) to protect the security of your keys. ------------- AWS CloudHSM provides you with a dedicated hardware device installed in your Amazon Virtual Private Cloud (VPC) that provides a FIPS 140-2 Level 2 validated single-tenant HSM to store and use your keys. -------------
- Compliance is another requirement for many of our customers. As you can see here that we have a wide range of certifications covering MySQL, Oracle, Microsoft SQL Server, and PostgreSQL around key areas of compliance.
- Having a highly available database is an important part of your overall application architecture. If the rest of your application is highly available and your database is not then application downtime is inevitable in case of failures
- When you launch an RDS database the bare minimum you can have is a single Availability Zone configuration. With this you get a single RDS instance inside a VPC with the necessary attached storage. [click] This is great for getting an initial database up and running for some testing or POC work [click] ...but as you move into production workloads this is not going to provide you with a highly available application.
- For a more robust database architecture you are going to want to look at having a Multi Availability Zone configuration. Config: Pick Primary AZ, Standby in other AZ, same type, synchronous replication The failover conditions that this configuration handles are: Loss of availability in primary AZ Loss of network connectivity to primary Compute unit failure on primary Storage failure on primary OS of the DB instance is being patched during the maintenance window [Infrastructure] As a result of AZ / Region setup, sub-2ms (generally sub-1ms) (Steve Seymour, previous session, Day in the Life of a Billion Packets) Health: RDS checks Primary state, Standby in state to recover to, Standby NOT available to read (or login) ----- SQL Server uses mirroring to support this functionality -----
- You DB access is via a URL/port, which is mapped to the Master [click] When the Master fails RDS begins the failover process [click] The Standby is promoted to Master and the DNS record is updated to point to the Standby This is all seamless from a user perspective, but you will need to re-establish connections to the DB, and be careful with Java DNS caching to ensure it ‘forgets’ about the old Master. You can expect failover to complete within 60-120 seconds
- Amazon Aurora is a MySQL- and PostgreSQL-compatible Relational Database engine that combines the speed and availability of high-end commercial databases with the simplicity and cost-effectiveness of open source databases. It provides up to five times better performance than MySQL with the security, availability, and reliability of a commercial database at one tenth the cost. In this configuration, you have a primary node in AZ1 and can have upto 15 secondary nodes across the AZs in your Region. These secondary nodes are actually Read Replicas that customers can scale out to support read operations. When a failure is detected in the primary node the service will automatically promote one of the secondary nodes to be the primary node And the failing database processes are automatically detected and recycled.
- If you’re using Aurora then this failover that we’ve just discussed is much faster than on other RDS engines. Amazon Aurora is built on a fully distributed and self-healing storage system that keeps your data safe. Let’s see how we do this. Your data is available across three availability zones 6 copies of data – 2 in each AZ Storage is shared across multiple AZ’s and is available to every node no matter where it is. We can lose 2 nodes anywhere, anyhow, without impacting the system. 10Gb segments restore in parallel across all AZs if necessary, but if any erorrs in data – or holes – are found then
- So now that we have gone deeper in to how some of the key features work on RDS we’re now going to spend some time highlighting how you can move your existing database onto RDS. I'm sure that many of you have attempted large-scale database migrations... POP-QUIZ
- Migrations can be Complex, Time consuming and Expensive. They may involve application downtime and expert resources AND license-intensive replication software to move data Customers who want to switch engines had to deal with differences in datatypes, table syntax, sp’s, functions, triggers But this doesn't’ have to be the case!!! AWS being a customer centric company, addressed these by introducing DMS – Amazon Database Migration Service and AWS Schema Conversation Tool
- So it was with all these factors in mind that we developed the database migration service We designed it to be simple! So you can get started in less than ten minutes. We designed it to enable near-zero-downtime migrations. And we designed it to be a kind of Replication Swiss Army Knife, to securely replicate data between on-premises systems, RDS, EC2, and across database engine types The AWS Schema Conversion Tool makes heterogeneous database migrations easy by automatically converting the source database schema and a majority of the custom code to a format compatible with the target database. More than 28,000 databases have been migrated using AWS Database Migration Service. A few days ago Andy Jassy tweeted that it’s now over 30,000 – it moves fast!
- DMS (Database migration service) and SCT (AWS Schema Conversion Tool) can help you modernize, migrate, and replicate Modernise: Convert from one database engine to another, and update the associated application code using the code conversion features of SCT Migrate: You’re looking to move data from A to B. Whether that is on premise to the cloud, between EC2 and RDS, or between database engines/warehouses Replicate: Leverage the power of Change Data Capture and keep changes flowing from source to target on an ongoing basis
- by now it should be pretty obvious that data can move from A to B, and many customers have used DMS to do just that. Some customers have been pretty ingenious with DMS and have fanned in, or consolidated systems. DMS will move data where you ask it to, so if you want to merge data from a number of legacy systems into a single repository, DMS can help with this. you can take information from a number of different platforms and consolidate it in a single, cloud database. Replicate from multiple sources and consolidate to a single target server
- Or you can FAN OUT we’ve had a customer who has taken their database and “fanned-out” different portions to different targets. From their ERP system they replicate the customer list out to the CRM system AND also take some of the general ledger transactions and funnel them over to a BI system. We’ve also had a customer who, for legacy reasons, used a single database for three different applications. With DMS you can split this information out to different databases. One could be PostgreSQL, one SQL Server, and one Aurora
- Just in case it wasn’t abundantly clear – DMS can move your data from pretty much anywhere to pretty much anywhere else.
- The net result is that DMS and SCT bring a lot of advantages to our customers and can make it easier to move to the cloud. DMS is recommended when you're after consistency, standardizing the process, cost effectiveness AND best of all there near zero down time. DMS allows you to Keep your apps running during the migration ~ DMS minimizes impact to users by capturing and applying data changes You can cut over when you're ready
- DB Migration process has two key steps to it - Convert or Copy your database or data-warehousing schema using AWS SCT or your Native tool from Source to Destination The second step is to Move your actual data If you need to copy data from your database then leverage AWS DMS If you’re looking at copying data from a data-warehouse then you can leverage AWS SCT SCT is almost like a verification process, it clearly marks any code that the tool cannot convert automatically so that you can convert it yourself Bulk ingest via s3 COPY rather than table-by-table or row-by-row - COPY loads large amounts of data much more efficiently than using INSERT statements, and stores the data more effectively as well
- Let’s explore how it is that DMS can enable near zero downtime migration. We have a database on-premise on the left, and we wish to move it to the VPC in AWS on the right, using a VPC IPsec tunnel. (click) You launch a DMS instance in your AWS account. (click) You then provide database connection information to connect out to you on-premises database, and in to your AWS database. (click) You then select which tables, schemas or databases you want to migrate, (click) and DMS will load the data, and keep it in sync on an ongoing basis. (click) Finally, you simply change the application to point to the new AWS database, instead of their old on-premises database whenever you’re ready And this is the power of using AWS Database Migration Service! Its Simple, its cost effective and less time consuming.
- Data warehouses generally have very large data sets that cannot be moved over the wire in a reasonable amount of time Redshift is a fully managed, petabyte-scale data warehouse that makes it simple and cost-effective to analyze all your data using your existing business intelligence tools. SCT can now convert the warehouse schema to Amazon Redshift and extract the data from the source warehouse in a format that Redshift can ingest
- Recently added support for NoSQL databases using DMS For Mongo Two modes: Document mode > JSON data becomes a single column in a target table Table mode > DMS scans the Mongo documents and creates a set of all the keys and types to use a columns in a target table For Dynamo: DMS supports the scalar data types in Dynamo Use mapping rules to define record to record or record to document mappings for relational sources Also mention S3
- Trimble are a real estate and workplace solution provider They successfully completed an Oracle to PostGreSQL migration using DMS Trimble valued the STRAIGHTFORWARD & QUICK transition that DMS provides Two production databases 4TB and 7TB, a 4TB release/staging database, plus a few Dev and QA databases
- Expedia is a travel company Needed to move to Aurora (MySQL) as MySQL does not scale well when data volumes grow. They are in the process of migrating some DB’s to Amazon Aurora Expedia valued the SIMPLIFICATION & EASE of USE that DMS provides, as well as the detailed monitoring as logging within the DMS process to give them the confidence that everything is progressing.
- Flipboard bring together news, popular stories and conversations around any interest or passion They aim to provide one place for reading, collecting, and sharing stories Needed to move from Classic to VPC but could not take the required outage
- We have a great Ecosystem and we have a number of AWS Migration Partners who are here with us. So if you like what you see and if you'd like to know more, please feel free to reach out to us after the session. There is extensive documentation available for the same so go ahead and get your hands dirty. Getting started – use partners who are out in the expo If you would like to know more, or have questions please see us OR feel free to share your experiences with us.