SlideShare a Scribd company logo
1 of 16
Amazon Redshift
A whirl-wind tour

November 2013
Kel Graham
Data Architect
What is Redshift?
Amazon Product

Data Warehouse
Service

Fully managed

Redshift

Fast

Petabyte scale
(1PB == 1Billion MB)

1/10 cost of
traditional DW
As the universe expands, the wavelength of radiation from objects moving away
from an observer shifts towards the red end of the electromagnetic spectrum.
Redshift is a consequence of an expanding universe.
2
Where does Redshift sit within the Amazon database product suite?
Non-relational

Relational database
service

Data warehouse
service

Query flexibility

High availability

High availability

High availability

SimpleDB

DynamoDB

RDS

Redshift

(MySQL / Oracle / SQL Server)

(PostgreSQL base)

Web-services
interface

High scalability

Referential integrity

Cluster architecture

Smaller workloads

Run off SSDs

DB-dependent
feature-set (Multi-AZ)

Relational database

1MB response size

Integrates with
Redshift

Online Transaction
Processing

Horizontal scalability:
add more nodes

10GB hard limit
3

NoSQL service

Provisioned
throughput

Provisioned
throughput

Analytics
What differentiates Redshift from, say, a MySQL RDS instance?
Cluster Architecture

No RI by design

Redshift
Columnar storage

4

Read Optimised
What differentiates Redshift from, say, a MySQL RDS instance?
(i) Cluster architecture
a) Clients connect via existing protocols
to the Leader Node.
b) Leader node develops a query plan
and may generate and compile C++
code to be executed by the compute
nodes
c) Leader node will distribute work
across compute nodes using
Distribution Keys (more later)
d) Compute nodes receive work from
leader node and may transmit data
amongst themselves to answer the
query
e) Leader aggregates the results and
returns to client
f) Leader can distribute bulk data loads
across compute nodes: I have loaded
3G of raw data (gzipped to 500Mb)
on a single node in under 3 minutes)
5

source: http://docs.aws.amazon.com/redshift/latest/dg/c_high_level_system_architecture.html
What differentiates Redshift from, say, a MySQL RDS instance?
(ii) Column-store database
a) Relational databases tend to
store data on a tuple by
tuple basis.
b) When querying the data, the
engine needs to read more
blocks of data, discarding
much of the data just read in
order to return columns
being queried
c) A column-store stores
columns contiguously in the
same block
d) Result: the number of IO
operations involved in a
query can be significantly
reduced, dependent on the
shape of the data
source: http://docs.aws.amazon.com/redshift/latest/dg/c_columnar_storage_disk_mem_mgmnt.html

6
What differentiates Redshift from, say, a MySQL RDS instance?
(iii) Optimised for Read performance
a) Contrast block sizes with other databases:
a) Default MySQL installs on ext3 file-systems use 4k blocks
b) Default NTFS partitions use 4k blocks, so SQL Server on
NTFS defaults to 4k blocks as well.
b) Redshift’s focus on Data Warehousing (and hence read
optimisation) allows them to use a 1,024KB block size
c) Under a column-store architecture each block holds the same
kind of data, so datatype-specific compression enables even
more data to be stored per block, further reducing disk space
and IO
d) Reduced disk space and IO helps improve inter-node data
sharing and replication, where compute nodes may
redistribute data based on a table’s distribution key (more
later on that)

7

All your blox are
belong to me!
What differentiates Redshift from, say, a MySQL RDS instance?
(iv) No referential integrity

Ok, sounds g… WHAT?!!
•
•
•
•
•
•
•
•
•

8

Do tell Redshift about primary, foreign keys and
column uniqueness. It won’t enforce them, but it
will use these hints to better understand queries.

No primary key
No foreign key
No index support
No sequences
No user defined functions
No stored procedures
No common table expressions
No exotic data types – no arrays, JSON, Geospatial types, etc.
No ‘alter column’ syntax – drop and reload
How does Redshift locate data?
The Sort Key
•
•

Redshift will store data on disk in Sort Key order – so think of it as the single clustered index for
the table

•

Sort keys should be selected based on how the table is used:
• Columns that are frequently used to join to other tables should be included in the sort key
• Date and timestamp columns that are used in filtering operations should be included

•

Redshift stores metadata about each data block, including the min and max of each column value
– using this, Redshift can skip entire blocks when answering a query

•

9

Each table can have a single Sort Key – a compound key, comprised of 1 to 400 columns from the
table

After data loads or inserts, the ANALYZE command should be run
• ANALYZE updates the table metadata that is used by the query planner – very important for
column-based data storage and ongoing query performance
How does Redshift locate data?
The Distribution Key
•

•

10

Redshift will distribute and replicate data between compute nodes in order to get best use of
the parallelism available in the cluster
• By default, data will be spread evenly across all compute nodes (EVEN distribution)
• A node is further broken down into slices – one slice per CPU core
• Each slice participates in the parallel execution of a job sent from the Leader node, so the
even distribution of data across the nodes is vital to ensuring consistent query
performance
• If data is denormalised and does not participate in joins, then an EVEN distribution won’t
be problematic
Alternatively a Distribution key can be provided (KEY distribution)
• The Distribution key is important, in that it helps define which data is kept together on a
given node.
• The objective is to choose a key that helps distribute data across a node’s slices, but not
across the cluster’s nodes
• Similarly to the Sort Key, the Distribution key is defined on a per-table basis, but unlike a
Sort Key, the Distribution Key is comprised of only a single column
What typical RDBMS features does Redshift have?
Features
DataTypes (complete list):
• Transactions
• SmallInt
• Reasonable number of windowing functions
• Integer
• Rank, First, Last, Lag, Sum, Nth and so on
• Bigint
• Most types of relational joins
• Decimal
• Inner, Left, Right, Full, Cross
• Real
• Correlated sub-queries are supported, but only where • Double precision
the query planner can decorrelate them for
• Boolean
performance (sub-queries during a join are a no-go)
• Char
• Views
• Varchar
• Excellent locking and concurrent write capabilities
• Date
• Thanks PostgreSQL!
• Timestamp
• Schema management
• Identity columns (auto_increment)

11
Other features?
• Close integration with S3 and DynamoDB
• Our test instance was primed from S3:
COPY <tableName> from s3://bucket/file.csv.gz
header as 1
GZIP

• COPY command is central to the import process – can load data in parallel, using what it
knows about the structure of the target table to assign work to individual compute nodes
• UNLOAD will export data from a Redshift table out to an S3 bucket
• Excellent set of database system tables that allow one to monitor pretty much everything
that’s going on:
• Loads
• Queries
• Chatter between compute nodes
• Sort and distribution keys

12
Other features (cont)?
• Column compression
• Each column can have an optionally assigned compression algorithm, including:
• BYTEDICT – essentially a key-value lookup for up to 256 values. Useful for repeating
data, such as “State” in a property record
• DELTA – stores the initial value of a column as per its data type, and then stores only
the offset between the next value and the first value. Very useful for dates
• RUNLEGNTH – Stores the value of a column and the number of times the value is
repeated. Useful when the data is stored consecutively – relevant for sort-key
• MOSTLY8/16/32 – uses traditional numeric compression, but allows for outliers

13
Other features (cont)?
• Excellent monitoring and management console integration

14
How well does Redshift perform?
We ran some rudimentary queries over a realistic data set…
“Return the current list of all valid properties within a selected list
of states and tell me what the current number of bedrooms,
bathrooms, car spaces, land size, floor size and year built is.”

…and found that Redshift
outperformed our existing database
by a factory 2.5 – 3.5.

Correct select of a SORT KEY in Redshift is vital.
Any filtering or joins on a non-sortkey column will
result in (slow) a table scan. In our example, this
reduced performance by 30%.

However, this is not a particularly useful comparison as, these
were different machines with different hardware specifications.

15
Summary

Relational

No referential integrity

Column
Compression

Sort key

Column store

Fast

Accessible (JDBC, ODBC, S3)

Distribution Key

Massive Parallelism

Compute Node

Leader Node

Cluster Architecture

Redshift
16

More Related Content

What's hot

Uses and Best Practices for Amazon Redshift
Uses and Best Practices for Amazon Redshift Uses and Best Practices for Amazon Redshift
Uses and Best Practices for Amazon Redshift Amazon Web Services
 
Building Your Data Warehouse with Amazon Redshift
Building Your Data Warehouse with Amazon RedshiftBuilding Your Data Warehouse with Amazon Redshift
Building Your Data Warehouse with Amazon RedshiftAmazon Web Services
 
Data Warehousing with Amazon Redshift
Data Warehousing with Amazon RedshiftData Warehousing with Amazon Redshift
Data Warehousing with Amazon RedshiftAmazon Web Services
 
Data Warehousing with Amazon Redshift
Data Warehousing with Amazon RedshiftData Warehousing with Amazon Redshift
Data Warehousing with Amazon RedshiftAmazon Web Services
 
Big Data Analytics Architectural Patterns and Best Practices (ANT201-R1) - AW...
Big Data Analytics Architectural Patterns and Best Practices (ANT201-R1) - AW...Big Data Analytics Architectural Patterns and Best Practices (ANT201-R1) - AW...
Big Data Analytics Architectural Patterns and Best Practices (ANT201-R1) - AW...Amazon Web Services
 
Amazon Aurora Storage Demystified: How It All Works (DAT363) - AWS re:Invent ...
Amazon Aurora Storage Demystified: How It All Works (DAT363) - AWS re:Invent ...Amazon Aurora Storage Demystified: How It All Works (DAT363) - AWS re:Invent ...
Amazon Aurora Storage Demystified: How It All Works (DAT363) - AWS re:Invent ...Amazon Web Services
 
Amazon EMR Deep Dive & Best Practices
Amazon EMR Deep Dive & Best PracticesAmazon EMR Deep Dive & Best Practices
Amazon EMR Deep Dive & Best PracticesAmazon Web Services
 
BDA306 Building a Modern Data Warehouse: Deep Dive on Amazon Redshift
BDA306 Building a Modern Data Warehouse: Deep Dive on Amazon RedshiftBDA306 Building a Modern Data Warehouse: Deep Dive on Amazon Redshift
BDA306 Building a Modern Data Warehouse: Deep Dive on Amazon RedshiftAmazon Web Services
 
re:Invent 2022 DAT326 Deep dive into Amazon Aurora and its innovations
re:Invent 2022  DAT326 Deep dive into Amazon Aurora and its innovationsre:Invent 2022  DAT326 Deep dive into Amazon Aurora and its innovations
re:Invent 2022 DAT326 Deep dive into Amazon Aurora and its innovationsGrant McAlister
 

What's hot (20)

Amazon Redshift
Amazon Redshift Amazon Redshift
Amazon Redshift
 
Uses and Best Practices for Amazon Redshift
Uses and Best Practices for Amazon Redshift Uses and Best Practices for Amazon Redshift
Uses and Best Practices for Amazon Redshift
 
Building Your Data Warehouse with Amazon Redshift
Building Your Data Warehouse with Amazon RedshiftBuilding Your Data Warehouse with Amazon Redshift
Building Your Data Warehouse with Amazon Redshift
 
Introduction to Amazon Athena
Introduction to Amazon AthenaIntroduction to Amazon Athena
Introduction to Amazon Athena
 
Deep Dive on Amazon Redshift
Deep Dive on Amazon RedshiftDeep Dive on Amazon Redshift
Deep Dive on Amazon Redshift
 
Introduction to Amazon Athena
Introduction to Amazon AthenaIntroduction to Amazon Athena
Introduction to Amazon Athena
 
Data Warehousing with Amazon Redshift
Data Warehousing with Amazon RedshiftData Warehousing with Amazon Redshift
Data Warehousing with Amazon Redshift
 
AWS RDS
AWS RDSAWS RDS
AWS RDS
 
Amazon Redshift Masterclass
Amazon Redshift MasterclassAmazon Redshift Masterclass
Amazon Redshift Masterclass
 
Data Warehousing with Amazon Redshift
Data Warehousing with Amazon RedshiftData Warehousing with Amazon Redshift
Data Warehousing with Amazon Redshift
 
Big Data Analytics Architectural Patterns and Best Practices (ANT201-R1) - AW...
Big Data Analytics Architectural Patterns and Best Practices (ANT201-R1) - AW...Big Data Analytics Architectural Patterns and Best Practices (ANT201-R1) - AW...
Big Data Analytics Architectural Patterns and Best Practices (ANT201-R1) - AW...
 
Amazon Aurora Storage Demystified: How It All Works (DAT363) - AWS re:Invent ...
Amazon Aurora Storage Demystified: How It All Works (DAT363) - AWS re:Invent ...Amazon Aurora Storage Demystified: How It All Works (DAT363) - AWS re:Invent ...
Amazon Aurora Storage Demystified: How It All Works (DAT363) - AWS re:Invent ...
 
Masterclass - Redshift
Masterclass - RedshiftMasterclass - Redshift
Masterclass - Redshift
 
Amazon EMR Deep Dive & Best Practices
Amazon EMR Deep Dive & Best PracticesAmazon EMR Deep Dive & Best Practices
Amazon EMR Deep Dive & Best Practices
 
BDA306 Building a Modern Data Warehouse: Deep Dive on Amazon Redshift
BDA306 Building a Modern Data Warehouse: Deep Dive on Amazon RedshiftBDA306 Building a Modern Data Warehouse: Deep Dive on Amazon Redshift
BDA306 Building a Modern Data Warehouse: Deep Dive on Amazon Redshift
 
Deep Dive on Amazon Aurora
Deep Dive on Amazon AuroraDeep Dive on Amazon Aurora
Deep Dive on Amazon Aurora
 
re:Invent 2022 DAT326 Deep dive into Amazon Aurora and its innovations
re:Invent 2022  DAT326 Deep dive into Amazon Aurora and its innovationsre:Invent 2022  DAT326 Deep dive into Amazon Aurora and its innovations
re:Invent 2022 DAT326 Deep dive into Amazon Aurora and its innovations
 
HDInsight for Architects
HDInsight for ArchitectsHDInsight for Architects
HDInsight for Architects
 
ElastiCache & Redis
ElastiCache & RedisElastiCache & Redis
ElastiCache & Redis
 
Amazon Aurora: Under the Hood
Amazon Aurora: Under the HoodAmazon Aurora: Under the Hood
Amazon Aurora: Under the Hood
 

Viewers also liked

Building Your Data Warehouse with Amazon Redshift
Building Your Data Warehouse with Amazon RedshiftBuilding Your Data Warehouse with Amazon Redshift
Building Your Data Warehouse with Amazon RedshiftAmazon Web Services
 
AWS November Webinar Series - Advanced Analytics with Amazon Redshift and the...
AWS November Webinar Series - Advanced Analytics with Amazon Redshift and the...AWS November Webinar Series - Advanced Analytics with Amazon Redshift and the...
AWS November Webinar Series - Advanced Analytics with Amazon Redshift and the...Amazon Web Services
 
(BDT401) Amazon Redshift Deep Dive: Tuning and Best Practices
(BDT401) Amazon Redshift Deep Dive: Tuning and Best Practices(BDT401) Amazon Redshift Deep Dive: Tuning and Best Practices
(BDT401) Amazon Redshift Deep Dive: Tuning and Best PracticesAmazon Web Services
 
Best Practices for Migrating your Data Warehouse to Amazon Redshift
Best Practices for Migrating your Data Warehouse to Amazon RedshiftBest Practices for Migrating your Data Warehouse to Amazon Redshift
Best Practices for Migrating your Data Warehouse to Amazon RedshiftAmazon Web Services
 
Getting Started with Amazon Redshift
Getting Started with Amazon RedshiftGetting Started with Amazon Redshift
Getting Started with Amazon RedshiftAmazon Web Services
 

Viewers also liked (7)

Building Your Data Warehouse with Amazon Redshift
Building Your Data Warehouse with Amazon RedshiftBuilding Your Data Warehouse with Amazon Redshift
Building Your Data Warehouse with Amazon Redshift
 
AWS November Webinar Series - Advanced Analytics with Amazon Redshift and the...
AWS November Webinar Series - Advanced Analytics with Amazon Redshift and the...AWS November Webinar Series - Advanced Analytics with Amazon Redshift and the...
AWS November Webinar Series - Advanced Analytics with Amazon Redshift and the...
 
(BDT401) Amazon Redshift Deep Dive: Tuning and Best Practices
(BDT401) Amazon Redshift Deep Dive: Tuning and Best Practices(BDT401) Amazon Redshift Deep Dive: Tuning and Best Practices
(BDT401) Amazon Redshift Deep Dive: Tuning and Best Practices
 
Best Practices for Migrating your Data Warehouse to Amazon Redshift
Best Practices for Migrating your Data Warehouse to Amazon RedshiftBest Practices for Migrating your Data Warehouse to Amazon Redshift
Best Practices for Migrating your Data Warehouse to Amazon Redshift
 
Deep Dive on Amazon Redshift
Deep Dive on Amazon RedshiftDeep Dive on Amazon Redshift
Deep Dive on Amazon Redshift
 
Getting Started with Amazon Redshift
Getting Started with Amazon RedshiftGetting Started with Amazon Redshift
Getting Started with Amazon Redshift
 
Build Features, Not Apps
Build Features, Not AppsBuild Features, Not Apps
Build Features, Not Apps
 

Similar to A tour of Amazon Redshift

AWS June 2016 Webinar Series - Amazon Redshift or Big Data Analytics
AWS June 2016 Webinar Series - Amazon Redshift or Big Data AnalyticsAWS June 2016 Webinar Series - Amazon Redshift or Big Data Analytics
AWS June 2016 Webinar Series - Amazon Redshift or Big Data AnalyticsAmazon Web Services
 
Highlights of AWS ReInvent 2023 (Announcements and Best Practices)
Highlights of AWS ReInvent 2023 (Announcements and Best Practices)Highlights of AWS ReInvent 2023 (Announcements and Best Practices)
Highlights of AWS ReInvent 2023 (Announcements and Best Practices)Emprovise
 
Best Practices for Migrating Your Data Warehouse to Amazon Redshift
Best Practices for Migrating Your Data Warehouse to Amazon RedshiftBest Practices for Migrating Your Data Warehouse to Amazon Redshift
Best Practices for Migrating Your Data Warehouse to Amazon RedshiftAmazon Web Services
 
Deep Dive into DynamoDB
Deep Dive into DynamoDBDeep Dive into DynamoDB
Deep Dive into DynamoDBAWS Germany
 
Best Practices for Supercharging Cloud Analytics on Amazon Redshift
Best Practices for Supercharging Cloud Analytics on Amazon RedshiftBest Practices for Supercharging Cloud Analytics on Amazon Redshift
Best Practices for Supercharging Cloud Analytics on Amazon RedshiftSnapLogic
 
Rise of Column Oriented Database
Rise of Column Oriented DatabaseRise of Column Oriented Database
Rise of Column Oriented DatabaseSuvradeep Rudra
 
Implementing the Databese Server session 02
Implementing the Databese Server session 02Implementing the Databese Server session 02
Implementing the Databese Server session 02Guillermo Julca
 
AWS Redshift Introduction - Big Data Analytics
AWS Redshift Introduction - Big Data AnalyticsAWS Redshift Introduction - Big Data Analytics
AWS Redshift Introduction - Big Data AnalyticsKeeyong Han
 
Cassandra Tutorial
Cassandra Tutorial Cassandra Tutorial
Cassandra Tutorial Na Zhu
 
AWS re:Invent 2016| DAT318 | Migrating from RDBMS to NoSQL: How Sony Moved fr...
AWS re:Invent 2016| DAT318 | Migrating from RDBMS to NoSQL: How Sony Moved fr...AWS re:Invent 2016| DAT318 | Migrating from RDBMS to NoSQL: How Sony Moved fr...
AWS re:Invent 2016| DAT318 | Migrating from RDBMS to NoSQL: How Sony Moved fr...Amazon Web Services
 
Presentacion redislabs-ihub
Presentacion redislabs-ihubPresentacion redislabs-ihub
Presentacion redislabs-ihubssuser9d7c90
 
SPL_ALL_EN.pptx
SPL_ALL_EN.pptxSPL_ALL_EN.pptx
SPL_ALL_EN.pptx政宏 张
 

Similar to A tour of Amazon Redshift (20)

AWS June 2016 Webinar Series - Amazon Redshift or Big Data Analytics
AWS June 2016 Webinar Series - Amazon Redshift or Big Data AnalyticsAWS June 2016 Webinar Series - Amazon Redshift or Big Data Analytics
AWS June 2016 Webinar Series - Amazon Redshift or Big Data Analytics
 
Highlights of AWS ReInvent 2023 (Announcements and Best Practices)
Highlights of AWS ReInvent 2023 (Announcements and Best Practices)Highlights of AWS ReInvent 2023 (Announcements and Best Practices)
Highlights of AWS ReInvent 2023 (Announcements and Best Practices)
 
Cassandra training
Cassandra trainingCassandra training
Cassandra training
 
Nosql data models
Nosql data modelsNosql data models
Nosql data models
 
Lecture3.ppt
Lecture3.pptLecture3.ppt
Lecture3.ppt
 
Best Practices for Migrating Your Data Warehouse to Amazon Redshift
Best Practices for Migrating Your Data Warehouse to Amazon RedshiftBest Practices for Migrating Your Data Warehouse to Amazon Redshift
Best Practices for Migrating Your Data Warehouse to Amazon Redshift
 
Deep Dive on Amazon Redshift
Deep Dive on Amazon RedshiftDeep Dive on Amazon Redshift
Deep Dive on Amazon Redshift
 
Deep Dive into DynamoDB
Deep Dive into DynamoDBDeep Dive into DynamoDB
Deep Dive into DynamoDB
 
Processing and Analytics
Processing and AnalyticsProcessing and Analytics
Processing and Analytics
 
NoSQL.pptx
NoSQL.pptxNoSQL.pptx
NoSQL.pptx
 
Best Practices for Supercharging Cloud Analytics on Amazon Redshift
Best Practices for Supercharging Cloud Analytics on Amazon RedshiftBest Practices for Supercharging Cloud Analytics on Amazon Redshift
Best Practices for Supercharging Cloud Analytics on Amazon Redshift
 
Deep Dive on Amazon Redshift
Deep Dive on Amazon RedshiftDeep Dive on Amazon Redshift
Deep Dive on Amazon Redshift
 
Rise of Column Oriented Database
Rise of Column Oriented DatabaseRise of Column Oriented Database
Rise of Column Oriented Database
 
Implementing the Databese Server session 02
Implementing the Databese Server session 02Implementing the Databese Server session 02
Implementing the Databese Server session 02
 
AWS Redshift Introduction - Big Data Analytics
AWS Redshift Introduction - Big Data AnalyticsAWS Redshift Introduction - Big Data Analytics
AWS Redshift Introduction - Big Data Analytics
 
Cassandra Tutorial
Cassandra Tutorial Cassandra Tutorial
Cassandra Tutorial
 
AWS re:Invent 2016| DAT318 | Migrating from RDBMS to NoSQL: How Sony Moved fr...
AWS re:Invent 2016| DAT318 | Migrating from RDBMS to NoSQL: How Sony Moved fr...AWS re:Invent 2016| DAT318 | Migrating from RDBMS to NoSQL: How Sony Moved fr...
AWS re:Invent 2016| DAT318 | Migrating from RDBMS to NoSQL: How Sony Moved fr...
 
Presentacion redislabs-ihub
Presentacion redislabs-ihubPresentacion redislabs-ihub
Presentacion redislabs-ihub
 
SPL_ALL_EN.pptx
SPL_ALL_EN.pptxSPL_ALL_EN.pptx
SPL_ALL_EN.pptx
 
Database Technologies
Database TechnologiesDatabase Technologies
Database Technologies
 

Recently uploaded

Choosing the Right FDO Deployment Model for Your Application _ Geoffrey at In...
Choosing the Right FDO Deployment Model for Your Application _ Geoffrey at In...Choosing the Right FDO Deployment Model for Your Application _ Geoffrey at In...
Choosing the Right FDO Deployment Model for Your Application _ Geoffrey at In...FIDO Alliance
 
TEST BANK For, Information Technology Project Management 9th Edition Kathy Sc...
TEST BANK For, Information Technology Project Management 9th Edition Kathy Sc...TEST BANK For, Information Technology Project Management 9th Edition Kathy Sc...
TEST BANK For, Information Technology Project Management 9th Edition Kathy Sc...marcuskenyatta275
 
BT & Neo4j _ How Knowledge Graphs help BT deliver Digital Transformation.pptx
BT & Neo4j _ How Knowledge Graphs help BT deliver Digital Transformation.pptxBT & Neo4j _ How Knowledge Graphs help BT deliver Digital Transformation.pptx
BT & Neo4j _ How Knowledge Graphs help BT deliver Digital Transformation.pptxNeo4j
 
Working together SRE & Platform Engineering
Working together SRE & Platform EngineeringWorking together SRE & Platform Engineering
Working together SRE & Platform EngineeringMarcus Vechiato
 
Behind the Scenes From the Manager's Chair: Decoding the Secrets of Successfu...
Behind the Scenes From the Manager's Chair: Decoding the Secrets of Successfu...Behind the Scenes From the Manager's Chair: Decoding the Secrets of Successfu...
Behind the Scenes From the Manager's Chair: Decoding the Secrets of Successfu...CzechDreamin
 
Portal Kombat : extension du réseau de propagande russe
Portal Kombat : extension du réseau de propagande russePortal Kombat : extension du réseau de propagande russe
Portal Kombat : extension du réseau de propagande russe中 央社
 
Designing for Hardware Accessibility at Comcast
Designing for Hardware Accessibility at ComcastDesigning for Hardware Accessibility at Comcast
Designing for Hardware Accessibility at ComcastUXDXConf
 
1111 ChatGPT Prompts PDF Free Download - Prompts for ChatGPT
1111 ChatGPT Prompts PDF Free Download - Prompts for ChatGPT1111 ChatGPT Prompts PDF Free Download - Prompts for ChatGPT
1111 ChatGPT Prompts PDF Free Download - Prompts for ChatGPTiSEO AI
 
State of the Smart Building Startup Landscape 2024!
State of the Smart Building Startup Landscape 2024!State of the Smart Building Startup Landscape 2024!
State of the Smart Building Startup Landscape 2024!Memoori
 
Linux Foundation Edge _ Overview of FDO Software Components _ Randy at Intel.pdf
Linux Foundation Edge _ Overview of FDO Software Components _ Randy at Intel.pdfLinux Foundation Edge _ Overview of FDO Software Components _ Randy at Intel.pdf
Linux Foundation Edge _ Overview of FDO Software Components _ Randy at Intel.pdfFIDO Alliance
 
ERP Contender Series: Acumatica vs. Sage Intacct
ERP Contender Series: Acumatica vs. Sage IntacctERP Contender Series: Acumatica vs. Sage Intacct
ERP Contender Series: Acumatica vs. Sage IntacctBrainSell Technologies
 
FDO for Camera, Sensor and Networking Device – Commercial Solutions from VinC...
FDO for Camera, Sensor and Networking Device – Commercial Solutions from VinC...FDO for Camera, Sensor and Networking Device – Commercial Solutions from VinC...
FDO for Camera, Sensor and Networking Device – Commercial Solutions from VinC...FIDO Alliance
 
Structuring Teams and Portfolios for Success
Structuring Teams and Portfolios for SuccessStructuring Teams and Portfolios for Success
Structuring Teams and Portfolios for SuccessUXDXConf
 
Enterprise Knowledge Graphs - Data Summit 2024
Enterprise Knowledge Graphs - Data Summit 2024Enterprise Knowledge Graphs - Data Summit 2024
Enterprise Knowledge Graphs - Data Summit 2024Enterprise Knowledge
 
Powerful Start- the Key to Project Success, Barbara Laskowska
Powerful Start- the Key to Project Success, Barbara LaskowskaPowerful Start- the Key to Project Success, Barbara Laskowska
Powerful Start- the Key to Project Success, Barbara LaskowskaCzechDreamin
 
TopCryptoSupers 12thReport OrionX May2024
TopCryptoSupers 12thReport OrionX May2024TopCryptoSupers 12thReport OrionX May2024
TopCryptoSupers 12thReport OrionX May2024Stephen Perrenod
 
Your enemies use GenAI too - staying ahead of fraud with Neo4j
Your enemies use GenAI too - staying ahead of fraud with Neo4jYour enemies use GenAI too - staying ahead of fraud with Neo4j
Your enemies use GenAI too - staying ahead of fraud with Neo4jNeo4j
 
WebAssembly is Key to Better LLM Performance
WebAssembly is Key to Better LLM PerformanceWebAssembly is Key to Better LLM Performance
WebAssembly is Key to Better LLM PerformanceSamy Fodil
 
ASRock Industrial FDO Solutions in Action for Industrial Edge AI _ Kenny at A...
ASRock Industrial FDO Solutions in Action for Industrial Edge AI _ Kenny at A...ASRock Industrial FDO Solutions in Action for Industrial Edge AI _ Kenny at A...
ASRock Industrial FDO Solutions in Action for Industrial Edge AI _ Kenny at A...FIDO Alliance
 

Recently uploaded (20)

Choosing the Right FDO Deployment Model for Your Application _ Geoffrey at In...
Choosing the Right FDO Deployment Model for Your Application _ Geoffrey at In...Choosing the Right FDO Deployment Model for Your Application _ Geoffrey at In...
Choosing the Right FDO Deployment Model for Your Application _ Geoffrey at In...
 
TEST BANK For, Information Technology Project Management 9th Edition Kathy Sc...
TEST BANK For, Information Technology Project Management 9th Edition Kathy Sc...TEST BANK For, Information Technology Project Management 9th Edition Kathy Sc...
TEST BANK For, Information Technology Project Management 9th Edition Kathy Sc...
 
BT & Neo4j _ How Knowledge Graphs help BT deliver Digital Transformation.pptx
BT & Neo4j _ How Knowledge Graphs help BT deliver Digital Transformation.pptxBT & Neo4j _ How Knowledge Graphs help BT deliver Digital Transformation.pptx
BT & Neo4j _ How Knowledge Graphs help BT deliver Digital Transformation.pptx
 
Overview of Hyperledger Foundation
Overview of Hyperledger FoundationOverview of Hyperledger Foundation
Overview of Hyperledger Foundation
 
Working together SRE & Platform Engineering
Working together SRE & Platform EngineeringWorking together SRE & Platform Engineering
Working together SRE & Platform Engineering
 
Behind the Scenes From the Manager's Chair: Decoding the Secrets of Successfu...
Behind the Scenes From the Manager's Chair: Decoding the Secrets of Successfu...Behind the Scenes From the Manager's Chair: Decoding the Secrets of Successfu...
Behind the Scenes From the Manager's Chair: Decoding the Secrets of Successfu...
 
Portal Kombat : extension du réseau de propagande russe
Portal Kombat : extension du réseau de propagande russePortal Kombat : extension du réseau de propagande russe
Portal Kombat : extension du réseau de propagande russe
 
Designing for Hardware Accessibility at Comcast
Designing for Hardware Accessibility at ComcastDesigning for Hardware Accessibility at Comcast
Designing for Hardware Accessibility at Comcast
 
1111 ChatGPT Prompts PDF Free Download - Prompts for ChatGPT
1111 ChatGPT Prompts PDF Free Download - Prompts for ChatGPT1111 ChatGPT Prompts PDF Free Download - Prompts for ChatGPT
1111 ChatGPT Prompts PDF Free Download - Prompts for ChatGPT
 
State of the Smart Building Startup Landscape 2024!
State of the Smart Building Startup Landscape 2024!State of the Smart Building Startup Landscape 2024!
State of the Smart Building Startup Landscape 2024!
 
Linux Foundation Edge _ Overview of FDO Software Components _ Randy at Intel.pdf
Linux Foundation Edge _ Overview of FDO Software Components _ Randy at Intel.pdfLinux Foundation Edge _ Overview of FDO Software Components _ Randy at Intel.pdf
Linux Foundation Edge _ Overview of FDO Software Components _ Randy at Intel.pdf
 
ERP Contender Series: Acumatica vs. Sage Intacct
ERP Contender Series: Acumatica vs. Sage IntacctERP Contender Series: Acumatica vs. Sage Intacct
ERP Contender Series: Acumatica vs. Sage Intacct
 
FDO for Camera, Sensor and Networking Device – Commercial Solutions from VinC...
FDO for Camera, Sensor and Networking Device – Commercial Solutions from VinC...FDO for Camera, Sensor and Networking Device – Commercial Solutions from VinC...
FDO for Camera, Sensor and Networking Device – Commercial Solutions from VinC...
 
Structuring Teams and Portfolios for Success
Structuring Teams and Portfolios for SuccessStructuring Teams and Portfolios for Success
Structuring Teams and Portfolios for Success
 
Enterprise Knowledge Graphs - Data Summit 2024
Enterprise Knowledge Graphs - Data Summit 2024Enterprise Knowledge Graphs - Data Summit 2024
Enterprise Knowledge Graphs - Data Summit 2024
 
Powerful Start- the Key to Project Success, Barbara Laskowska
Powerful Start- the Key to Project Success, Barbara LaskowskaPowerful Start- the Key to Project Success, Barbara Laskowska
Powerful Start- the Key to Project Success, Barbara Laskowska
 
TopCryptoSupers 12thReport OrionX May2024
TopCryptoSupers 12thReport OrionX May2024TopCryptoSupers 12thReport OrionX May2024
TopCryptoSupers 12thReport OrionX May2024
 
Your enemies use GenAI too - staying ahead of fraud with Neo4j
Your enemies use GenAI too - staying ahead of fraud with Neo4jYour enemies use GenAI too - staying ahead of fraud with Neo4j
Your enemies use GenAI too - staying ahead of fraud with Neo4j
 
WebAssembly is Key to Better LLM Performance
WebAssembly is Key to Better LLM PerformanceWebAssembly is Key to Better LLM Performance
WebAssembly is Key to Better LLM Performance
 
ASRock Industrial FDO Solutions in Action for Industrial Edge AI _ Kenny at A...
ASRock Industrial FDO Solutions in Action for Industrial Edge AI _ Kenny at A...ASRock Industrial FDO Solutions in Action for Industrial Edge AI _ Kenny at A...
ASRock Industrial FDO Solutions in Action for Industrial Edge AI _ Kenny at A...
 

A tour of Amazon Redshift

  • 1. Amazon Redshift A whirl-wind tour November 2013 Kel Graham Data Architect
  • 2. What is Redshift? Amazon Product Data Warehouse Service Fully managed Redshift Fast Petabyte scale (1PB == 1Billion MB) 1/10 cost of traditional DW As the universe expands, the wavelength of radiation from objects moving away from an observer shifts towards the red end of the electromagnetic spectrum. Redshift is a consequence of an expanding universe. 2
  • 3. Where does Redshift sit within the Amazon database product suite? Non-relational Relational database service Data warehouse service Query flexibility High availability High availability High availability SimpleDB DynamoDB RDS Redshift (MySQL / Oracle / SQL Server) (PostgreSQL base) Web-services interface High scalability Referential integrity Cluster architecture Smaller workloads Run off SSDs DB-dependent feature-set (Multi-AZ) Relational database 1MB response size Integrates with Redshift Online Transaction Processing Horizontal scalability: add more nodes 10GB hard limit 3 NoSQL service Provisioned throughput Provisioned throughput Analytics
  • 4. What differentiates Redshift from, say, a MySQL RDS instance? Cluster Architecture No RI by design Redshift Columnar storage 4 Read Optimised
  • 5. What differentiates Redshift from, say, a MySQL RDS instance? (i) Cluster architecture a) Clients connect via existing protocols to the Leader Node. b) Leader node develops a query plan and may generate and compile C++ code to be executed by the compute nodes c) Leader node will distribute work across compute nodes using Distribution Keys (more later) d) Compute nodes receive work from leader node and may transmit data amongst themselves to answer the query e) Leader aggregates the results and returns to client f) Leader can distribute bulk data loads across compute nodes: I have loaded 3G of raw data (gzipped to 500Mb) on a single node in under 3 minutes) 5 source: http://docs.aws.amazon.com/redshift/latest/dg/c_high_level_system_architecture.html
  • 6. What differentiates Redshift from, say, a MySQL RDS instance? (ii) Column-store database a) Relational databases tend to store data on a tuple by tuple basis. b) When querying the data, the engine needs to read more blocks of data, discarding much of the data just read in order to return columns being queried c) A column-store stores columns contiguously in the same block d) Result: the number of IO operations involved in a query can be significantly reduced, dependent on the shape of the data source: http://docs.aws.amazon.com/redshift/latest/dg/c_columnar_storage_disk_mem_mgmnt.html 6
  • 7. What differentiates Redshift from, say, a MySQL RDS instance? (iii) Optimised for Read performance a) Contrast block sizes with other databases: a) Default MySQL installs on ext3 file-systems use 4k blocks b) Default NTFS partitions use 4k blocks, so SQL Server on NTFS defaults to 4k blocks as well. b) Redshift’s focus on Data Warehousing (and hence read optimisation) allows them to use a 1,024KB block size c) Under a column-store architecture each block holds the same kind of data, so datatype-specific compression enables even more data to be stored per block, further reducing disk space and IO d) Reduced disk space and IO helps improve inter-node data sharing and replication, where compute nodes may redistribute data based on a table’s distribution key (more later on that) 7 All your blox are belong to me!
  • 8. What differentiates Redshift from, say, a MySQL RDS instance? (iv) No referential integrity Ok, sounds g… WHAT?!! • • • • • • • • • 8 Do tell Redshift about primary, foreign keys and column uniqueness. It won’t enforce them, but it will use these hints to better understand queries. No primary key No foreign key No index support No sequences No user defined functions No stored procedures No common table expressions No exotic data types – no arrays, JSON, Geospatial types, etc. No ‘alter column’ syntax – drop and reload
  • 9. How does Redshift locate data? The Sort Key • • Redshift will store data on disk in Sort Key order – so think of it as the single clustered index for the table • Sort keys should be selected based on how the table is used: • Columns that are frequently used to join to other tables should be included in the sort key • Date and timestamp columns that are used in filtering operations should be included • Redshift stores metadata about each data block, including the min and max of each column value – using this, Redshift can skip entire blocks when answering a query • 9 Each table can have a single Sort Key – a compound key, comprised of 1 to 400 columns from the table After data loads or inserts, the ANALYZE command should be run • ANALYZE updates the table metadata that is used by the query planner – very important for column-based data storage and ongoing query performance
  • 10. How does Redshift locate data? The Distribution Key • • 10 Redshift will distribute and replicate data between compute nodes in order to get best use of the parallelism available in the cluster • By default, data will be spread evenly across all compute nodes (EVEN distribution) • A node is further broken down into slices – one slice per CPU core • Each slice participates in the parallel execution of a job sent from the Leader node, so the even distribution of data across the nodes is vital to ensuring consistent query performance • If data is denormalised and does not participate in joins, then an EVEN distribution won’t be problematic Alternatively a Distribution key can be provided (KEY distribution) • The Distribution key is important, in that it helps define which data is kept together on a given node. • The objective is to choose a key that helps distribute data across a node’s slices, but not across the cluster’s nodes • Similarly to the Sort Key, the Distribution key is defined on a per-table basis, but unlike a Sort Key, the Distribution Key is comprised of only a single column
  • 11. What typical RDBMS features does Redshift have? Features DataTypes (complete list): • Transactions • SmallInt • Reasonable number of windowing functions • Integer • Rank, First, Last, Lag, Sum, Nth and so on • Bigint • Most types of relational joins • Decimal • Inner, Left, Right, Full, Cross • Real • Correlated sub-queries are supported, but only where • Double precision the query planner can decorrelate them for • Boolean performance (sub-queries during a join are a no-go) • Char • Views • Varchar • Excellent locking and concurrent write capabilities • Date • Thanks PostgreSQL! • Timestamp • Schema management • Identity columns (auto_increment) 11
  • 12. Other features? • Close integration with S3 and DynamoDB • Our test instance was primed from S3: COPY <tableName> from s3://bucket/file.csv.gz header as 1 GZIP • COPY command is central to the import process – can load data in parallel, using what it knows about the structure of the target table to assign work to individual compute nodes • UNLOAD will export data from a Redshift table out to an S3 bucket • Excellent set of database system tables that allow one to monitor pretty much everything that’s going on: • Loads • Queries • Chatter between compute nodes • Sort and distribution keys 12
  • 13. Other features (cont)? • Column compression • Each column can have an optionally assigned compression algorithm, including: • BYTEDICT – essentially a key-value lookup for up to 256 values. Useful for repeating data, such as “State” in a property record • DELTA – stores the initial value of a column as per its data type, and then stores only the offset between the next value and the first value. Very useful for dates • RUNLEGNTH – Stores the value of a column and the number of times the value is repeated. Useful when the data is stored consecutively – relevant for sort-key • MOSTLY8/16/32 – uses traditional numeric compression, but allows for outliers 13
  • 14. Other features (cont)? • Excellent monitoring and management console integration 14
  • 15. How well does Redshift perform? We ran some rudimentary queries over a realistic data set… “Return the current list of all valid properties within a selected list of states and tell me what the current number of bedrooms, bathrooms, car spaces, land size, floor size and year built is.” …and found that Redshift outperformed our existing database by a factory 2.5 – 3.5. Correct select of a SORT KEY in Redshift is vital. Any filtering or joins on a non-sortkey column will result in (slow) a table scan. In our example, this reduced performance by 30%. However, this is not a particularly useful comparison as, these were different machines with different hardware specifications. 15
  • 16. Summary Relational No referential integrity Column Compression Sort key Column store Fast Accessible (JDBC, ODBC, S3) Distribution Key Massive Parallelism Compute Node Leader Node Cluster Architecture Redshift 16