Databases in the Cloud em Amazon Web Services

3/14/19
1
© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Mv – Marcus Vinicius Ferreira
Sr. SA, Public Sector, Education
Fev/2019
EBC: Databases in the Cloud
Databases, DataWarehouses, DataLakes
Mv – Marcus Vinicius Ferreira
mvferr@amazon.com
Solutions Architect
BR, Public Sector, Education
Previous: Oracle, Sun, Abril, Baby, Nubank
Presenter
Mv

3/14/19
2
Agenda
• Databases: RDS e Aurora
• Datawarehouse: Amazon Redshift
• Data Lake
• Database Freedom
• Q&A
Databases:
Amazon RDS

3/14/19
3
Corporate data
center
Database DB on EC2
instance RDS
Databases: SQL, NoSQL
EC2
Server
Volume Volume Volume Volume Volume Volume

3/14/19
4
Amazon Relational Database Service
• Introduced in 2009
• Multi-engine support: Amazon Aurora, MySQL,
MariaDB, PostgreSQL, Oracle, Microsoft SQL
Server
• Automated provisioning, patching, scaling,
replicas, backup/restore, monitoring, failover
• Easy to manage via console or API
• Low-cost ‘pay for what you use’
• Built-in High availability with RDS Multi-AZ and
Amazon Aurora
Commercial Open source Cloud native
MySQL Compatible
PostgreSQL Compatible
Amazon Aurora

3/14/19
5
If you host your databases on premises
Power, HVAC, net
Rack and stack
Server maintenance
OS patches
DB s/w patches
Database backups
High availability
DB s/w installs
OS installation
Scaling
App optimization
you
Power, HVAC, net
Rack and stack
Server maintenance
OS patches
DB s/w patches
Database backups
Scaling
High availability
DB s/w installs
OS installation
App optimization
If you host your databases on premises
?
you

3/14/19
6
If you host your databases in EC2
Power, HVAC, net
Rack and stack
Server maintenance
OS patches
DB s/w patches
Database backups
Scaling
High availability
DB s/w installs
OS installation
App optimization
you
OS patches
DB s/w patches
Database backups
Scaling
High availability
DB s/w installs
App optimization
Power, HVAC, net
Rack and stack
Server maintenance
OS installation
If you host your databases in EC2
?
you

3/14/19
7
If you choose a managed database service
Power, HVAC, net
Rack and stack
Server maintenance
OS patches
DB s/w patches
Database backups
App optimization
High availability
DB s/w installs
OS installation
Scaling
Database tuning
Design and Review
App optimization
Best practices
you
How is Database Practitioner time spent with
Amazon RDS?
New Apps
App upgrades
Schema design
Query construction
Query optimization
Automatic fail-over
Backup & recovery
Isolation & security
Industry compliance
Push-button scaling
Automated patching
Advanced monitoring
Routine maintenance
RDS takes care of your time-
consuming database management
tasks, freeing you to focus on your
applications and business
DBA AWS

3/14/19
8
Focus: application optimization
App optimization
Database tuning
Design and Review
App optimization
Best practices
you
Amazon RDS Performance Insights
DB Load : Average Active
Sessions
• Easy
• Powerful
• Identifies database
bottlenecks
Top SQL
• Identifies source of
bottleneck
Enables Problem Discovery
• Adjustable Time frame
• Hour, day, week and longer
• Investigate: SQL, waits, user,
hosts

3/14/19
9
© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Tamanhos de Instâncias
1 x c4.8xlarge 2 x c4.4xlarge
≈
4 x c4.2xlarge
≈
8 x c4.xlarge
≈
~$1.60 ~$0.80 ~$0.40 ~$0.20
Scaling Instances Up/Down
AWS Management Console

3/14/19
10
RDS Multi-AZ: High-Availability comes FIRST.
172.31.0.0/16
sa-east-1a sa-east-1b sa-east-1c
Availability Zone A
AWS Region
10.1.0.0/16
10.1.1.0/24
Availability Zone B
10.1.2.0/24
Synchronous Replication
M S
Single
Availability Zone
Configuration
HA Multi Availability Zone Configuration

3/14/19
11
High availability—Multi-AZ to DNS
Availability Zone A
AWS Region
Availability Zone B
M S
DNS
mydatabase.us-west-2.rds.amazonaws.com:3306
M
Database Resize
Availability Zone
AWS Region
10.1.2.0/24
Availability Zone
10.1.1.0/24
Prod

3/14/19
12
Empty
Architectures
can evolve…
Database Resize
Availability Zone
AWS Region
10.1.2.0/24
Availability Zone
10.1.1.0/24
Prod

3/14/19
13
Database Resize... and separation.
Availability Zone
AWS Region
10.1.2.0/24
Availability Zone
10.1.1.0/24
ERP
Website 1
Website 2
ERP
Website 1
Website 2
Amazon CloudWatch for RDS
l CPU utilization
l Storage
l Memory
l 50+ system/OS metrics
l 1–60 second granularity
l DB connections
l Selects per second
l Latency (read and write)
l Cache hit ratio
l Replica lag
CloudWatch alarms
l Similar to on-premises custom
monitoring tools

3/14/19
14
Availability Zone
AWS Region
10.1.2.0/24
Availability Zone
10.1.1.0/24
ERP
Website 1
Website 2
ERP
Website 1
Website 2
Read Replicas
Read scaling and disaster recovery
RDS for MySQL, PostgreSQL,
MariaDB
• Relieve pressure on your master node
with additional read capacity
• Bring data close to your applications
in different regions
• Promote a read replica to a master for
faster recovery in the event of disaster
Asynchronous
replication
BI/reporting
application server
Read only
Read/write Primary
Read replica
Slave

3/14/19
15
Availability Zone
AWS Region
10.1.2.0/24
Availability Zone
10.1.1.0/24
ERP
Website 1
Website 2
ERP
Website 1
Website 2
Automated backups
Point-in-time recovery for your DB instance
• Scheduled daily volume backup of
entire instance
• Archive database change logs
• 35–day maximum retention
• Minimal impact on database
performance
• Taken from standby when
running Multi-AZ
Every day during your backup
window, RDS creates a storage
volume snapshot of your instance
Every five minutes, RDS backs up the
transaction logs of your database

3/14/19
16
Database snapshots
Backups of your entire DB instance in Amazon S3
Bucket
Volume
Snapshot 2Snapshot 1 Snapshot 3
• Always incremental
• Amazon S3 à
99.999999999% durability
• Supports encryption
• Copy across accounts,
across regions
Database:
Amazon Aurora

3/14/19
17
BETTER SCALABILITY & LATENCY:
Scale out to millions of reads per second
Scale-Out Read Performance
Up to 15 Read Replicas Across Three AZs
Auto-Scale New Read Replicas
Auto-Scale Storage (no pre-provisioning needed)
FAST: Aurora MySQL – 5x Read/Write Throughput
WRITE PERFORMANCE READ PERFORMANCE
MySQL SysBench results
R3.8XL: 32 cores / 244 GB RAM
Five times higher throughput than stock MySQL
based on industry standard benchmarks.
0
25,000
50,000
75,000
100,000
125,000
150,000
0
100,000
200,000
300,000
400,000
500,000
600,000
700,000
Aurora MySQL 5.6 MySQL 5.7

3/14/19
18
BETTER DURABILITY: Aurora 6-way replicated storage
Six copies across three Availability Zones
• 4 out of 6 write quorum; 3 out of 6 read quorum
• Peer-to-peer replication for repairs
SQL
Transaction
AZ 1 AZ 2 AZ 3
Caching
SQL
Transaction
AZ 1 AZ 2 AZ 3
Caching
Read availabilityRead and write availability
Simplify storage management
• Automatic storage scaling up to 64 TB—no performance impact
• Continuous, incremental backups to Amazon S3
• Instantly create user snapshots—no performance impact
• Automatic restriping, mirror repair, hot spot management, encryption
Up to 64 TB of storage—autoincremented in 10 GB units
up to 64 TB

3/14/19
19
Datawarehouse:
Amazon Redshift
Purpose-
built
databases &
analytic
engines.
Right tool for the right job.

3/14/19
20
Traditionally, BI Looked Like This
OLTP ERP CRM LOB
Data Warehouse
Business Intelligence
• On-premises
• Relational data
• Terabyte to Petabyte scale
• Schema defined prior to data load
• Operational reporting and ad hoc
• Large initial capex + $10K–$50K / Terabyte / Year
• Difficult and slow to scale
Amazon Redshift is Secure
Security is Built-in
10 GigE
(HPC)
Customer VPC
Internal
VPC
JDBC/ODBC
Leader
Node
Compute
Node
Compute
Node
Compute
Node
• End-to-end data encryption
• Amazon VPC for network isolation
• Audit logging and AWS CloudTrail integration
• Communicate event-specific notifications via
email, text message, or call with Amazon SNS
• Certifications and compliance include
• SOC 1/2/3, PCI-DSS, FedRAMP, HIPAA
S3

3/14/19
21
Availability Zone
AWS Region
10.1.2.0/24
Availability Zone
10.1.1.0/24
Prod
Website 1
Website 2 ReportsAmazon
Redshift
Amazon Redshift is Integrated
Use Existing BI Tools
JDBC/ODBC
Tool based native drivers
On Prem
Amazon
QuickSight
Existing or new
BI tool
Amazon
Redshift

3/14/19
22
AWS Database Migration Service
Accelerate Migrations to Amazon Redshift
DMS usage is free for 6 months
For more information: https://aws.amazon.com/dms/free-dms/
“AWS Database Migration Service is
the most impressive migration service
we’ve seen.”
Azure SQL
Database
AWS DMS Amazon
Redshift
2. Relational Databases
1. Non-Relational Databases
3. Other sources
Amazon S3
Starting small is powerful, when you can scale up fast
Scaling up your analytics systems With AWS Traditional IT *
get a new BI server 20 minutes 3 months
upgrade your analytics server to the
newest Intel processors and add 16GB
memory
15 minutes 2 months
add 500TB of storage instant 2 months
grow a DWH cluster from 8GB to 1PB 1 hour 8 months
build a 1024-node Hadoop cluster 30 minutes unlikely
roll out multi-region production
environment
hours months
* actual provisioning times in a well-organized IT division

3/14/19
23
Data Lake
The world’s most
valuable resource is
no longer oil, but data.*
*Copyright: The Economist, 2017, David Parkins
“
”

3/14/19
24
Organizations that successfully
generate business value from their
data, will outperform their peers.*
24%
15%
Leaders Followers
Organic revenue growth
*Aberdeen: Angling for Insight in Today’s Data Lake, Michael Lock, SVP Analytics and Business Intelligence
To become a leader, data is becoming the differentiator
Data Continues to Grow
• Data volumes are growing faster
than the ability to analyze the data
• New types of data cannot be
analyzed in context with data in
the data warehouse
• Users and applications need faster
data ingest & analytics results

3/14/19
25
Sources:
Gartner: User Survey Analysis: Key Trends Shaping the Future of Data Center Infrastructure Through 2011
IDC: Worldwide Business Analytics Software 2012–2016 Forecast and 2011 Vendor Shares
Legacy Architecture Models Lead to Dark Data
Traditional data warehouses cannot keep pace with data growth
1990 2000 2010 2020
DataVolume
All Data Analyzed Data
Dark Data
is not available
for analysis
What do the analysts say?
https://www.promptcloud.com
https://john-popelaars.blogspot.com
https://ww.signiant.com
https://www.linkedin.com/pulse/world-today-data-rich-information-poor-guru-p-mohapatra-pmp/

3/14/19
26
Datalake: beyond the Datawarehouse
OLTP ERP CRM LOB
Data Warehouse
Business Intelligence
• S3 buckets
• File based storage
• Petabyte scale and beyond
• History information will never be deleted
Characteristics of a Data Lake
Future
Proof
Flexible
Access
Dive in
Anywhere
Collect
Anything

3/14/19
27
Data Lake Pipeline
Ingest Store Process Consume
ETL/Streaming
User Profile
Recommendation
Hot Homes
Similar Homes
Agent Follow-up
Agent Scorecard
Marketing
A/B Testing
Real Time Data
Insights
Business
intelligence
Redfin
website
Amazon
EMR/Spark
Amazon
Redshift
Amazon
DynamoDB
Amazon S3
Data lake
On premisesdata
Web appdata
Amazon RDS
Other databases
Devices
Your data
Social Media
Central Storage
Secure, Cost Effective
Storage in S3
S3
Catalog & Search
Access & Search Metadata
DynamoDB Amazon ES
Access & User Interface
Give your users easy & secure access
API Gateway IAM Cognito
Protect & Secure
Use entitlements to ensure data is secure and users identities are verified
Security Token
Service
Cloudwatch Cloudtrail KMS
Athena Quicksight EMR Redshift
Processing & Analytics
Use predictive and prescriptive
analytics to gain better understanding
Firehose Direct Connect Snowball DMS
Data Ingestion
Get your data into S3
quickly and securely

3/14/19
28
Big Data Pipeline
CONSUME
Amazon QuickSight
Apps & Services
Analysis&visualizationNotebooksAPI
ETL
Streaming
Amazon Kinesis
Analytics
Amazon KCL
apps
AWS Lambda
Amazon Redshift
PROCESS/ANALYZE
Amazon Machine
Learning
Presto
Amazon
EMR
BatchInteractiveStreamML
Amazon EC2
COLLECT
Mobile apps
Web apps
Devices
Sensors &
IoT solutions AWS IoT
Analytics
Enterprise
apps
Logging
Amazon
CloudWatch
AWS
CloudTrail
LoggingIoTApplications
STORE
Amazon Elasticsearch
Service
Apache Kafka
Amazon Kinesis
Streams
Amazon Kinesis
Firehose
Amazon DynamoDB
Amazon S3
Amazon RDS
Amazon DynamoDB
Streams
SearchSQLNoSQLFileStream
Amazon Redshift
Leadership

3/14/19
29
Compliance
• SOC 1, 2, 3
• ISO 27001/9001
• ISO 27017/27018
• PCI
• FedRAMP
(MySQL)
• HIPAA BAA
Aurora
• SOC 1, 2, 3
• ISO 27001/9001
• ISO 27017/27018
• PCI
• FedRAMP
• HIPAA BAA
• UK Gov. Programs
• Singapore MTCS
MySQL
• SOC 1, 2, 3
• ISO 27001/9001
• ISO 27017/27018
• PCI
• FedRAMP
• HIPAA BAA
• Singapore MTCS
Oracle
• SOC 1, 2, 3
• ISO 27001/9001
• ISO 27017/27018
• PCI
• HIPAA BAA
MariaDB
• SOC 1, 2, 3
• ISO 27001/9001
• ISO 27017/27018
• PCI
• FedRAMP
• HIPAA BAA
• Singapore MTCS
PostgreSQL
• SOC 1, 2, 3
• ISO 27001/9001
• ISO 27017/27018
• PCI
• HIPAA BAA
• Singapore MTCS
SQL Server
Singapore MTCS
27001/9001
27017/27018
Gartner, Magic Quadrant for Data Management Solutions for Analytics, 13 February 2018. This graphic was
published by Gartner, Inc. as part of a larger research document and should be evaluated in the context of
the entire document. The Gartner document is available upon request from Amazon. Gartner does not
endorse any vendor, product or service depicted in its research publications, and does not advise technology
users to select only those vendors with the highest ratings or other designation. Gartner research
publications consist of the opinions of Gartner's research organization and should not be construed as
statements of fact. Gartner disclaims all warranties, expressed or implied, with respect to this research,
including any warranties of merchantability or fitness for a particular purpose.

3/14/19
30
The Forrester Wave™ is copyrighted by Forrester Research, Inc. Forrester and Forrester
Wave™ are trademarks of Forrester Research, Inc. The Forrester Wave™ is a graphical
representation of Forrester's call on a market and is plotted using a detailed spreadsheet
with exposed scores, weightings, and comments. Forrester does not endorse any vendor,
product, or service depicted in the Forrester Wave. Information is based on best available
resources. Opinions reflect judgment at the time and are subject to change.
The Forrester Wave™
Database-As-A-Service, Q2 2017
The Forrester Wave™ is copyrighted by Forrester Research, Inc. Forrester and Forrester Wave™ are trademarks of Forrester Research, Inc. The Forrester Wave™ is a graphical representation of Forrester's call on a market and is plotted using a
detailed spreadsheet with exposed scores, weightings, and comments. Forrester does not endorse any vendor, product, or service depicted in the Forrester Wave. Information is based on best available resources. Opinions reflect judgment at
the time and are subject to change.
“Amazon Redshift has the largest adoption of
BDW in the cloud.”
“With more than 5,000 deployments,
Amazon Redshift has the largest data
warehouse deployments in the cloud – some
over 10 petabytes in size.”
AWS received a score of 5/5 (the highest score
possible) in the: customer base, market
awareness, ability to execute,
road map, support, and partners criteria

3/14/19
31
Starting small is powerful, when you can scale up fast
Scaling up your analytics systems With AWS Traditional IT *
get a new BI server 20 minutes 3 months
upgrade your analytics server to the
newest Intel processors and add 16GB
memory
15 minutes 2 months
add 500TB of storage instant 2 months
grow a DWH cluster from 8GB to 1PB 1 hour 8 months
build a 1024-node Hadoop cluster 30 minutes unlikely
roll out multi-region production
environment
hours months
* actual provisioning times in a well-organized IT division
Customer Successes

3/14/19
32
Hundreds of thousands of customers use Amazon RDS
Thousands of Organizations Run Mission
Critical Workloads on Amazon Redshift

3/14/19
33
Airbnb moved its main MySQL database to
Amazon RDS with only 15 minutes of downtime
• RDS simplifies much of the time-consuming administrative
tasks associated with databases so engineers can spend more
time on features
• Airbnb uses asynchronous replication to improve website
performance launched via the RDS console or an API call
• Multi-Availability Zone (Multi-AZ) provides Airbnb with
high availability
Airbnb—Amazon RDS for MySQL

3/14/19
34
CUSTOMERS: Provide scalable web services quickly
The Government of Ontario makes
government information and
services accessible to the 14
million people in Ontario, Canada's
largest province.
Database Freedom

3/14/19
35
What is Database Freedom?
Launched in early 2018, Database Freedom is an AWS
Database Migration Program focused on accelerating
enterprise migrations from commercial database engines
to AWS native database services.
What we hear from customers about legacy
databases...
AUDIT
Very expensive Proprietary Lock-in Punitive
licensing
You’ve got mail
https://www.theregister.co.uk/2017/01/30/oracle_effectively_doubles_licence_fees_to_run_in_aws/
https://www.infoworld.com/article/3164517/cloud-computing/oracle-is-pricing-itself-out-of-amazons-cloud.html

3/14/19
36
Programs
Discover Database Freedom with AWS
Innovation
• Tools: Database Migration Service (DMS) and Schema Conversion Tool (SCT)
• Services: Aurora MySQL and Aurora PostgreSQL, DynamoDB, EMR, Redshift,
and others
• Optimized and new EC2 and RDS instance types
• ProServe, Partners, Service Teams
• Workload Qualification Framework
• Patterns and Recommendations
Expertise
• AWS Professional Services, Partners, Product Teams
• Migration Playbooks
• Patterns and Recommendations
• Proofs-of-Concepts
• Workshops
• Incentives / Credits
Database Freedom is an AWS Database Migration Program focused on accelerating enterprise migrations from
commercial database engines to AWS native database services.
AWS migration options
AWS DMS
Database migration and
replication managed
service
Benefit
• Easily and securely
migrate and/or
replicate your
databases and data
warehouses to AWS
• Migrate between
different database
engines
• Low cost and global
availability
AWS SCT
Development
tooling to convert
schemas between
databases and data
warehouses
Benefit
• Automates schema
conversion including
database structure and
code
• Minimizes manual
effort of performing a
schema re-write
• Allows conversion
from commercial to
open source platforms
Native Tooling
Migration options
included with the
engine
Benefit
• Leverage a familiar
environment
• Full support for native
features
Open Source
Ora2Pg and others
Benefit
• Tailored solution for a
specific problem
• Multiple tools can
offer a more complete
solution than any one
product
• Free
Commercial
Attunity, Golden
Gate and others
Benefit
• Commercial solutions
are available for a
wide range of
migration sources and
targets including
legacy mainframe
systems
Innovation

3/14/19
37
DynamoDB NeptuneRDS
Aurora CommercialCommunity
Timestream QLDBElastiCache
Relational Key-value Document In-memory Graph Time series Ledger
Innovation
AWS database partners
Migration SI Partners License Advisory Partners
Expertise

3/14/19
38
Verizon is migrating over 1,000 business-critical applications and database backend systems to AWS,
several of which also include the migration of production databases to Amazon Aurora.
By migrating from Microsoft SQL Server to Amazon Aurora, Ryanair can run one of the largest email
campaigns in Europe with higher performance at a fraction of the cost, sending out 22 million emails
daily to customers.
Trimble migrated their Oracle databases to Amazon RDS and project they will pay about 1/4th of
what they paid when managing their private infrastructure.
Intuit migrated from Microsoft SQL Server to Amazon Redshift to reduce data-processing timelines
and get insights to decision makers faster and more frequently.
Equinox Fitness migrated its Teradata on-premises data warehouse to Amazon Redshift. They went from
static reports to a modern data lake that delivers dynamic reports.
Migrated their Market Data system from SQL Server to Aurora MySQL using AWS Database Migration
Service(DMS) to replicate data nightly. Reduces their processing times from 8 hours to 3 hours.
By December 2018, Amazon.com will have migrated 88% of their Oracle DBs (and 97% of critical
system DBs) moved to Amazon Aurora and Amazon DynamoDB. They also migrated their 50 PB
Oracle Data Warehouse to AWS (Amazon S3, Amazon Redshift, and Amazon EMR).
Samsung Electronics migrated their Cassandra clusters to Amazon DynamoDB for their Samsung
Cloud workload with 70% cost savings.
Questions?

3/14/19
39
Thank You!
https://aws.amazon.com/rds/
AWS Global Infrastructure
18 Regions – 55 Availability Zones – 114 Edge Locations
Region & Number of Availability Zones
AWS GovCloud (2) EU
Ireland (3)
US West Frankfurt (2)
Oregon (3) London (2)
Northern California (3)
Asia Pacific
US East Singapore (2)
N. Virginia (5), Ohio (3) Sydney (2), Tokyo (3),
Seoul (2), Mumbai (2)
Canada
Central (2) China
Beijing (2)
South America
São Paulo (3)
Announced Regions
Paris, Ningxia

3/14/19
40
Region
New Region
Coming Soon
Edge Location
Region &
Number of Availability Zones#
The Global Infrastructure
REGION
~ 2ms latency
3
AZa
AZc
AZb
Redundant Tier-1 Internet &
Inter-Region Connectivity
TCTC
DC
DC
DC
DC
DCDC
DC
DC
DC

Databases in the Cloud em Amazon Web Services

Empfohlen

Empfohlen

Weitere ähnliche Inhalte

Was ist angesagt?

Was ist angesagt? (20)

Ähnlich wie Databases in the Cloud em Amazon Web Services

Ähnlich wie Databases in the Cloud em Amazon Web Services (20)

Mehr von Amazon Web Services LATAM

Mehr von Amazon Web Services LATAM (20)

Kürzlich hochgeladen

Kürzlich hochgeladen (20)

Databases in the Cloud em Amazon Web Services