Whether you are building a new application, refactoring an existing one or working on the next big startup, you want to focus on your application and your business and not on time consuming database administration tasks. In this session we will cover how a managed relational database service offloads the undifferentiated heavy lifting around High Availability, DR, Security, Backup/Recovery and Patching while improving your ability to innovate using the capabilities of the PostgreSQL database. We will also dive into the recent changes to the service including support for logical replication and 9.6 as well as lessons learned running a large fleet of PostgreSQL instances.
3. Amazon Relational Database Service
RDS is a managed Relational database service that is simple to deploy,
easy to scale, reliable and cost-effective
Managed Service
Easy to Scale and Operate
Choice of Database Engines
High Availability
High Performance
Amazon Relational Database Service (RDS)
29. Amazon Aurora with PostgreSQL Compatibility
Performance By The Numbers
Measurement Result
PgBench >= 2x faster
SysBench 2x-3x faster
Data Loading 3x faster
Response Time >2x faster
Throughput Jitter >3x more consistent
Throughput at Scale 3x faster
Recovery Speed Up to 85x faster
31. Minor Version Patching
• Customer Control Patching
• Execute immediately or in Weekly Maintenance Window
• Auto Patching can be enabled
• Allows for hands off upgrade in next Maintenance Window
41. Forcing SSL on all connections
DB
Instance
Snapshot
Application
Host
Log Backups
42. Forcing SSL on all connections
DB
Instance
Snapshot
Application
Host
Log Backups
Security Group
43. Forcing SSL on all connections
DB
Instance
Snapshot
Application
Host
SSL
Log Backups
Security Group
44. Forcing SSL on all connections
DB
Instance
Snapshot
Application
Host
SSL
Log Backups
Security Group
VPC
45. Forcing SSL on all connections
DB
Instance
Snapshot
Application
Host
SSL
Log Backups
Security Group
VPC
Encryption at Rest
46. Forcing SSL on all connections
DB
Instance
Snapshot
Application
Host
SSL
Log Backups
Security Group
VPC
Encryption at Rest
ssl_mode=disable
47. Forcing SSL on all connections
DB
Instance
Snapshot
Application
Host
SSL
Log Backups
Security Group
VPC
Encryption at Rest
ssl_mode=disable
48. Forcing SSL on all connections
DB
Instance
Snapshot
Application
Host
SSL
Log Backups
Security Group
VPC
Encryption at Rest
ssl_mode=disable
rds.force_ssl=1 (default 0)
49. Forcing SSL on all connections
DB
Instance
Snapshot
Application
Host
SSL
Log Backups
Security Group
VPC
Encryption at Rest
ssl_mode=disable
rds.force_ssl=1 (default 0)
64. HIPAA-eligible service & FedRAMP
• RDS PostgreSQL is now a HIPAA-eligible service
• https://aws.amazon.com/compliance/hipaa-compliance/
• FedRAMP in AWS GovCloud (US) region
• https://aws.amazon.com/compliance/fedramp/
66. Move data to the same or different database engine
Keep your apps running during the migration
Start your first migration in 10 minutes or less
Replicate within, to, or from AWS EC2 or RDS
AWS
Database Migration
Service
(DMS)
69. Customer
Premises
Application Users
EC2
or
RDS
Internet
VPN
Start a replication instance
Connect to source and target databases
Select tables, schemas, or databases
Let the AWS Database Migration
Service create tables and load data
Keep your apps running during the migration
AWS Database
Migration Service
70. Customer
Premises
Application Users
EC2
or
RDS
Internet
VPN
Start a replication instance
Connect to source and target databases
Select tables, schemas, or databases
Let the AWS Database Migration
Service create tables and load data
Uses change data capture to keep
them in sync
Keep your apps running during the migration
AWS Database
Migration Service
71. Customer
Premises
Application Users
EC2
or
RDS
Internet
VPN
Start a replication instance
Connect to source and target databases
Select tables, schemas, or databases
Let the AWS Database Migration
Service create tables and load data
Uses change data capture to keep
them in sync
Switch applications over to the target
at your convenience
Keep your apps running during the migration
AWS Database
Migration Service
72. AWS Database Migration Service - PostgreSQL
• Source - on premise or EC2 PostgreSQL (9.4+)
RDS (9.4.9+ or 9.5.4+ or 9.6.1)
• Destination can be EC2 or RDS
• Initial bulk copy via consistent select
• Uses PostgreSQL logical replication support to provide
change data capture
https://aws.amazon.com/dms/
73. Logical Replication Support
• Supported with 9.6.1+, 9.5.4+ and 9.4.9+
• Set rds.logical_replication parameter to 1
• As user who has rds_replication & rds_superuser role
SELECT * FROM pg_create_logical_replication_slot('test_slot', 'test_decoding');
pg_recvlogical -d postgres --slot test_slot -U master --host $rds_hostname -f - --start
• Added support for Event Triggers
93. Stats on RAMDISK
• Set rds.pg_stat_ramdisk_size in MB’s
• Creates a RAM disk and sets stats_temp_directory to
use it.
• Reduces IOPS
• Good for instances with many tables/indexes and
databases.
94. Burst mode: GP2 and T2
T2 – Amazon EC2 instance with burst capability
• Base performance + burst
• Earn credits per hour when below base performance
• Can store up to 24 hours worth of credits
• Amazon CloudWatch metrics to see credits and usage
GP2 – SSD-based Amazon EBS storage
• 3 IOPS per GB base performance
• Earn credits when usage below base
• Burst to 3000+ IOPS
101. Vacuum parameters
Will auto vacuum when
• autovacuum_vacuum_threshold +
autovacuum_vacuum_scale_factor * pgclass.reltuples
How hard auto vacuum works
• autovacuum_max_workers
• autovacuum_nap_time
• autovacuum_cost_limit
• autovacuum_cost_delay
102. RDS autovacuum logging (9.4.5+)
log_autovacuum_min_duration = 5000 (i.e. 5 secs)
rds.force_autovacuum_logging_level = LOG
…[14638]:ERROR: canceling autovacuum task
…[14638]:CONTEXT: automatic vacuum of table "postgres.public.pgbench_tellers"
…[14638]:LOG: skipping vacuum of "pgbench_branches" --- lock not available
http://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Appendix.PostgreSQL.Comm
onDBATasks.html#Appendix.PostgreSQL.CommonDBATasks.Autovacuum
103. RDS autovacuum visibility(9.3.12, 9.4.7, 9.5.2, 9.6.1)
pg_stat_activity
BEFORE
usename | query
----------+-------------------------------------------------------------
rdsadmin | <insufficient privilege>
rdsadmin | <insufficient privilege>
gtest | SELECT c FROM sbtest27 WHERE id BETWEEN 392582 AND 392582+4
gtest | select usename, query from pg_stat_activity
NOW
usename | query
----------+----------------------------------------------
rdsadmin | <insufficient privilege>
gtest | select usename, query from pg_stat_activity
gtest | COMMIT
rdsadmin | autovacuum: ANALYZE public.sbtest16
Slide should be self-explanatory. Backup and disaster recovery can be a costly exercise for many businesses. So much so that often the strategy for DR is to simply accept the risk of data loss. RDS for SQL Server takes regular snapshots of the DB and the retention of backups can be set to 35 days if needed. PiT restore against a 5 minute window enables more granularity of recoverability.
Questions to ask:
How do you manage backup, recovery and snapshots in your current DB environment
How quickly can you recover your DB instance if it fails … usually this is much slower than if AWS RDS has to replace a failed host
What is an acceptable level of data loss for the workload
Kmj
To summarize what we have covered in the last few performance slides: we showed results from both pgbench and sysbench, demonstrating throughput 2x to 3x that of PostgreSQL; we showed data loading 3x faster, we showed response time to be more than 2x faster, and both response time and throughput to be much more consistent; we showed throughput at larger scale databases to be 3x faster, and we showed recovery speed to be much much faster due to the innovations in Amazon Aurora storage.
Line data type
Reg* data types
Open prepared transactions
Line data type
Reg* data types
Open prepared transactions
Line data type
Reg* data types
Open prepared transactions
Line data type
Reg* data types
Open prepared transactions
Line data type
Reg* data types
Open prepared transactions
Line data type
Reg* data types
Open prepared transactions
Line data type
Reg* data types
Open prepared transactions
Line data type
Reg* data types
Open prepared transactions
Add a Key for the encrypted snapshot and then show that it needs to be shared for this to work. Note that this doesn’t work with default keys.
Add a Key for the encrypted snapshot and then show that it needs to be shared for this to work. Note that this doesn’t work with default keys.
Add a Key for the encrypted snapshot and then show that it needs to be shared for this to work. Note that this doesn’t work with default keys.
Add a Key for the encrypted snapshot and then show that it needs to be shared for this to work. Note that this doesn’t work with default keys.
Add a Key for the encrypted snapshot and then show that it needs to be shared for this to work. Note that this doesn’t work with default keys.
Add a Key for the encrypted snapshot and then show that it needs to be shared for this to work. Note that this doesn’t work with default keys.
Add a Key for the encrypted snapshot and then show that it needs to be shared for this to work. Note that this doesn’t work with default keys.
Move data to the same or different database engine
~ Supports Oracle, Microsoft SQL Server, MySQL, PostgreSQL, MariaDB, Amazon Aurora, Amazon Redshift
Keep your apps running during the migration
~ DMS minimizes impact to users by capturing and applying data changes
Start your first migration in 10 minutes or less
~ The AWS Database Migration Service takes care of infrastructure provisioning and allows you to setup your first database migration task in less than 10 minutes
Replicate within, to or from AWS EC2 or RDS
~ After migrating your database, use the AWS Database Migration Service to replicate data into your Redshift data warehouses, cross-region to other RDS instances, or back to on-premises
Using the AWS Database Migration Service to migrate data to AWS is simple.
(CLICK) Start by spinning up a DMS instance in your AWS environment
(CLICK) Next, from within DMS, connect to both your source and target databases
(CLICK) Choose what data you want to migrate. DMS lets you migrate tables, schemas, or whole databases
Then sit back and let DMS do the rest. (CLICK) It creates the tables, loads the data, and best of all, keeps them synchronized for as long as you need
That replication capability, which keeps the source and target data in sync, allows customers to switch applications (CLICK) over to point to the AWS database at their leisure. DMS eliminates the need for high-stakes extended outages to migrate production data into the cloud. DMS provides a graceful switchover capability.
Using the AWS Database Migration Service to migrate data to AWS is simple.
(CLICK) Start by spinning up a DMS instance in your AWS environment
(CLICK) Next, from within DMS, connect to both your source and target databases
(CLICK) Choose what data you want to migrate. DMS lets you migrate tables, schemas, or whole databases
Then sit back and let DMS do the rest. (CLICK) It creates the tables, loads the data, and best of all, keeps them synchronized for as long as you need
That replication capability, which keeps the source and target data in sync, allows customers to switch applications (CLICK) over to point to the AWS database at their leisure. DMS eliminates the need for high-stakes extended outages to migrate production data into the cloud. DMS provides a graceful switchover capability.
Using the AWS Database Migration Service to migrate data to AWS is simple.
(CLICK) Start by spinning up a DMS instance in your AWS environment
(CLICK) Next, from within DMS, connect to both your source and target databases
(CLICK) Choose what data you want to migrate. DMS lets you migrate tables, schemas, or whole databases
Then sit back and let DMS do the rest. (CLICK) It creates the tables, loads the data, and best of all, keeps them synchronized for as long as you need
That replication capability, which keeps the source and target data in sync, allows customers to switch applications (CLICK) over to point to the AWS database at their leisure. DMS eliminates the need for high-stakes extended outages to migrate production data into the cloud. DMS provides a graceful switchover capability.
Using the AWS Database Migration Service to migrate data to AWS is simple.
(CLICK) Start by spinning up a DMS instance in your AWS environment
(CLICK) Next, from within DMS, connect to both your source and target databases
(CLICK) Choose what data you want to migrate. DMS lets you migrate tables, schemas, or whole databases
Then sit back and let DMS do the rest. (CLICK) It creates the tables, loads the data, and best of all, keeps them synchronized for as long as you need
That replication capability, which keeps the source and target data in sync, allows customers to switch applications (CLICK) over to point to the AWS database at their leisure. DMS eliminates the need for high-stakes extended outages to migrate production data into the cloud. DMS provides a graceful switchover capability.
Using the AWS Database Migration Service to migrate data to AWS is simple.
(CLICK) Start by spinning up a DMS instance in your AWS environment
(CLICK) Next, from within DMS, connect to both your source and target databases
(CLICK) Choose what data you want to migrate. DMS lets you migrate tables, schemas, or whole databases
Then sit back and let DMS do the rest. (CLICK) It creates the tables, loads the data, and best of all, keeps them synchronized for as long as you need
That replication capability, which keeps the source and target data in sync, allows customers to switch applications (CLICK) over to point to the AWS database at their leisure. DMS eliminates the need for high-stakes extended outages to migrate production data into the cloud. DMS provides a graceful switchover capability.
Push button scaling is a capability of RDS that enables you to quickly and easily scale your instance nodes up or down.
Slide content should be self-explanatory – questions to ask:
How do you predict future scale requirements for the application and how far in advance would you need to order, and provision hardware in order to scale the infrastructure up
How quickly are you able to respond to peaks in demand – and / or a drop in application load, resulting in excess provisioned hardware and storage
