In this session, we will take you through setting up an Amazon Redshift cluster and at the ways you can populate it with data. We will start by using AWS DMS to replicate the data as-is as well as doing some ETL on it. This will be followed by AWS Glue where you can do more advanced ETL operations. Lastly, we will look at how you can use Amazon Kinesis Firehose to stream event directly to the Redshift cluster.

1. Cobus Bernard
Sr Developer Advocate
Amazon Web Services
Getting Started with Data
Warehouses on AWS
@cobusbernard
cobusbernard
cobusbernard

2. Agenda
Amazon Redshift
Columnar Data
Compression
Query execution with Redshift Spectrum
Demo
Q&A

3. Load
Unload
Backup
Restore
Amazon Redshift architecture
Massively parallel, shared
nothing columnar architecture
Leader node
SQL endpoint
Stores metadata
Coordinates parallel SQL processing
Compute nodes
Local, columnar storage
Executes queries in parallel
Load, unload, backup, restore
Amazon Redshift Spectrum nodes
Execute queries directly against Amazon S3
SQL clients/BI tools
JDBC/ODBC
Compute
node
Compute
node
Compute
node
Leader
node
Amazon S3
...
1 2 3 4 N
Amazon
Redshift
Spectrum
Load
Query

4. Terminology and concepts: Node types
Amazon Redshift analytics – RA3
Amazon Redshift managed storage – Solid state disks + Amazon S3
Dense compute – DC2
Solid state disks
Dense storage – DS2
Magnetic disks
Instance type Disk type Size Memory CPUs Slices
RA3 4xlarge RMS Scales to 16 TB 96 GB 12 4
RA3 16xlarge RMS Scales to 64 TB 384 GB 48 16
DC2 large SSD 160 GB 16 GB 2 2
DC2 8xlarge SSD 2.56 TB 244 GB 32 16
DS2 xlarge Magnetic 2 TB 32 GB 4 2
DS2 8xlarge Magnetic 16 TB 244 GB 36 16

5. Terminology and concepts: Columnar
Amazon Redshift uses a columnar architecture for storing data on disk
Goal: reduce I/O for analytics queries
Physically store data on disk by column rather than row
Only read the column data that is required

6. Row-based storage
• Need to read everything
• Unnecessary I/O
aid loc dt
CREATE TABLE deep_dive (
aid INT --audience_id
,loc CHAR(3) --location
,dt DATE --date
);
aid loc dt
1 SFO 2017-10-20
2 JFK 2017-10-20
3 SFO 2017-04-01
4 JFK 2017-05-14
SELECT min(dt) FROM deep_dive;
Columnar architecture: Example

7. Column-based storage
• Only scan blocks for
relevant column
aid loc dt
CREATE TABLE deep_dive (
aid INT --audience_id
,loc CHAR(3) --location
,dt DATE --date
);
aid loc dt
1 SFO 2017-10-20
2 JFK 2017-10-20
3 SFO 2017-04-01
4 JFK 2017-05-14
SELECT min(dt) FROM deep_dive;
Columnar architecture: Example

8. Compression example
Add 1 of 13 different encodings
to each column
aid loc dt
CREATE TABLE deep_dive (
aid INT --audience_id
,loc CHAR(3) --location
,dt DATE --date
);
aid loc dt
1 SFO 2017-10-20
2 JFK 2017-10-20
3 SFO 2017-04-01
4 JFK 2017-05-14

9. Compression example
• More efficient compression is due
to storing the same data type in
the columnar architecture
• Columns grow and shrink
independently
• Reduces storage requirements
• Reduces I/O
aid loc dt
CREATE TABLE deep_dive (
aid INT ENCODE AZ64
,loc CHAR(3) ENCODE BYTEDICT
,dt DATE ENCODE RUNLENGTH
);
aid loc dt
1 SFO 2017-10-20
2 JFK 2017-10-20
3 SFO 2017-04-01
4 JFK 2017-05-14

10. CREATE TABLE deep_dive (
aid INT --audience_id
,loc CHAR(3) --location
,dt DATE --date
) SORTKEY (dt, loc);
Example: Sort key
Add a sort key to one or more
columns to physically sort
the data on disk
deep_dive
aid loc dt
1 SFO 2017-10-20
2 JFK 2017-10-20
3 SFO 2017-04-01
4 JFK 2017-05-14
deep_dive (sorted)
aid loc dt
3 SFO 2017-04-01
4 JFK 2017-05-14
2 JFK 2017-10-20
1 SFO 2017-10-20
deep_dive (sorted)
aid loc dt
3 SFO 2017-04-01
4 JFK 2017-05-14
2 JFK 2017-10-20
deep_dive (sorted)
aid loc dt
3 SFO 2017-04-01
4 JFK 2017-05-14
deep_dive (sorted)
aid loc dt
3 SFO 2017-04-01

11. SELECT count(*) FROM deep_dive WHERE dt = '06-09-2017';
MIN: 01-JUNE-2017
MAX: 06-JUNE-2017
MIN: 07-JUNE-2017
MAX: 12-JUNE-2017
MIN: 13-JUNE-2017
MAX: 21-JUNE-2017
MIN: 21-JUNE-2017
MAX: 30-JUNE-2017
Sorted by date
MIN: 01-JUNE-2017
MAX: 20-JUNE-2017
MIN: 08-JUNE-2017
MAX: 30-JUNE-2017
MIN: 12-JUNE-2017
MAX: 20-JUNE-2017
MIN: 02-JUNE-2017
MAX: 25-JUNE-2017
Unsorted table
Example: Zone maps and sorting

12. Data distribution
Distribution style is a table property that
dictates how that table’s data is distributed
throughout the cluster
KEY: Value is hashed, same value goes to same location (slice)
ALL: Full table data goes to the first slice of every node
EVEN: Round robin
AUTO: Combines EVEN and ALL
Goals
Distribute data evenly for
parallel processing
Minimize data movement
during query processing
KEY
Node 1
Slice
1
Slice
2
Node 2
Slice
3
Slice
4
keyA
keyB
keyC
keyD
ALL
Node 1
Slice
1
Slice
2
Node 2
Slice
3
Slice
4
EVEN
Node 1
Slice
1
Slice
2
Node 2
Slice
3
Slice
4

13. Node 1
Data distribution example
Slice 0 Slice 1
Node 2
Slice 2 Slice 3
Table: deep_dive
User columns System columns
aid loc dt ins del row
CREATE TABLE deep_dive (
aid INT --audience_id
,loc CHAR(3) --location
,dt DATE --date
) (EVEN|KEY|ALL|AUTO);

14. Node 1
Slice 0 Slice 1
Node 2
Slice 2 Slice 3
Data distribution, EVEN example
INSERT INTO deep_dive VALUES
(1, 'SFO', '2016-09-01'),
(2, 'JFK', '2016-09-14'),
(3, 'SFO', '2017-04-01'),
(4, 'JFK', '2017-05-14');
Table: deep_dive
User Columns System Columns
aid loc dt ins del row
Table: deep_dive
User Columns System Columns
aid loc dt ins del row
Table: deep_dive
User Columns System Columns
aid loc dt ins del row
Table: deep_dive
User Columns System Columns
aid loc dt ins del row
Rows: 0 Rows: 0 Rows: 0 Rows: 0Rows: 1 Rows: 1 Rows: 1 Rows: 1
CREATE TABLE deep_dive (
aid INT --audience_id
,loc CHAR(3) --location
,dt DATE --date
) DISTSTYLE EVEN;

15. Node 1
Slice 0 Slice 1
Node 2
Slice 2 Slice 3
Data distribution, KEY example #1
INSERT INTO deep_dive VALUES
(1, 'SFO', '2016-09-01'),
(2, 'JFK', '2016-09-14'),
(3, 'SFO', '2017-04-01'),
(4, 'JFK', '2017-05-14');
Table: deep_dive
User Columns System Columns
aid loc dt ins del row
Rows: 2 Rows: 0 Rows: 0Rows: 0Rows: 1
Table: deep_dive
User Columns System Columns
aid loc dt ins del row
Rows: 2Rows: 0Rows: 1
CREATE TABLE deep_dive (
aid INT --audience_id
,loc CHAR(3) --location
,dt DATE --date
) DISTSTYLE KEY DISTKEY (loc);

16. Node 1
Slice 0 Slice 1
Node 2
Slice 2 Slice 3
Data distribution, KEY example #2
INSERT INTO deep_dive VALUES
(1, 'SFO', '2016-09-01'),
(2, 'JFK', '2016-09-14'),
(3, 'SFO', '2017-04-01'),
(4, 'JFK', '2017-05-14');
Table: deep_dive
User Columns System Columns
aid loc dt ins del row
Table: deep_dive
User Columns System Columns
aid loc dt ins del row
Table: deep_dive
User Columns System Columns
aid loc dt ins del row
Table: deep_dive
User Columns System Columns
aid loc dt ins del row
Rows: 0 Rows: 0 Rows: 0 Rows: 0Rows: 1 Rows: 1 Rows: 1 Rows: 1
CREATE TABLE deep_dive (
aid INT --audience_id
,loc CHAR(3) --location
,dt DATE --date
) DISTSTYLE KEY DISTKEY (aid);

17. Node 1
Slice 0 Slice 1
Node 2
Slice 2 Slice 3
Data distribution, ALL example
INSERT INTO deep_dive VALUES
(1, 'SFO', '2016-09-01'),
(2, 'JFK', '2016-09-14'),
(3, 'SFO', '2017-04-01'),
(4, 'JFK', '2017-05-14');
Rows: 0 Rows: 0
Table: deep_dive
User Columns System Columns
aid loc dt ins del row
Rows: 0Rows: 1Rows: 2Rows: 4Rows: 3
Table: deep_dive
User Columns System Columns
aid loc dt ins del row
Rows: 0Rows: 1Rows: 2Rows: 4Rows: 3
CREATE TABLE deep_dive (
aid INT --audience_id
,loc CHAR(3) --location
,dt DATE --date
) DISTSTYLE ALL;

18. ...
1 2 3 4 N
Amazon
Redshift
Spectrum
Load
Query
Amazon Redshift architecture
Massively parallel, shared
nothing columnar architecture
Leader node
SQL endpoint
Stores metadata
Coordinates parallel SQL processing
Compute nodes
Local, columnar storage
Executes queries in parallel
Load, unload, backup, restore
Amazon Redshift Spectrum nodes
Execute queries directly against Amazon S3
SQL clients/BI tools
JDBC/ODBC
Compute
node
Compute
node
Compute
node
Leader
node
Amazon S3

19. Query
SELECT COUNT(*)
FROM S3.EXT_TABLE
GROUP BY…
Life Of A Query
Amazon
Redshift
JDBC/ODBC
...
1 2 3 4 N
Amazon S3
Exabyte-scale object storage
Data Catalog
Apache Hive
Compatible Metastore
1

20. Life Of A Query
Amazon
Redshift
JDBC/ODBC
...
1 2 3 4 N
Amazon S3
Exabyte-scale object storage
Data Catalog
Apache Hive
Compatible Metastore
Query is optimised and compiled at
the leader node. Determine what gets
run locally and what goes to Amazon
Redshift Spectrum
2

21. Life Of A Query
Amazon
Redshift
JDBC/ODBC
...
1 2 3 4 N
Amazon S3
Exabyte-scale object storage
Data Catalog
Apache Hive
Compatible Metastore
Query plan is sent to
all compute nodes3

22. Life Of A Query
Amazon
Redshift
JDBC/ODBC
...
1 2 3 4 N
Amazon S3
Exabyte-scale object storage
Data Catalog
Apache Hive
Compatible Metastore
Compute nodes obtain partition info from
Data Catalog; dynamically prune partitions4

23. Life Of A Query
Amazon
Redshift
JDBC/ODBC
...
1 2 3 4 N
Amazon S3
Exabyte-scale object storage
Data Catalog
Apache Hive
Compatible Metastore
Each compute node issues multiple
requests to the Amazon Redshift
Spectrum layer
5

24. Life Of A Query
Amazon
Redshift
JDBC/ODBC
...
1 2 3 4 N
Amazon S3
Exabyte-scale object storage
Data Catalog
Apache Hive
Compatible Metastore
Amazon Redshift Spectrum nodes
scan your S3 data
6

25. Life Of A Query
Amazon
Redshift
JDBC/ODBC
...
1 2 3 4 N
Amazon S3
Exabyte-scale object storage
Data Catalog
Apache Hive
Compatible Metastore
7
Amazon Redshift
Spectrum projects,
filters, and aggregates

26. Life Of A Query
Amazon
Redshift
JDBC/ODBC
...
1 2 3 4 N
Amazon S3
Exabyte-scale object storage
Data Catalog
Apache Hive
Compatible Metastore
Final aggregations and joins with
local Amazon Redshift tables
done in-cluster
8

27. Life Of A Query
Amazon
Redshift
JDBC/ODBC
...
1 2 3 4 N
Amazon S3
Exabyte-scale object storage
Data Catalog
Apache Hive
Compatible Metastore
Result is sent back to client9

28. Data warehouse
(business data)
Data lake
(event data)
Amazon
Redshift
Amazon Redshift enables you to have a lake house approach
Customers moving to data lake architectures

29. © 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved.

30. Sources
Targets
Relational NoSQL Analytics Data Warehouse*
Amazon Aurora
Amazon Aurora
SAP ASE
Amazon DynamoDB
Amazon DocumentDB (with
MongoDB compatibility)
Db2 LUW SQL Azure
SAP ASE
MongoDB Cassandra
Amazon Redshift
Amazon S3
AWS SnowballAmazon S3
Amazon Elasticsearch
Service
Amazon Kinesis
Data Streams
Oracle SQL Server Netezza
Greenplum Teradata Vertica
* Supported via SCT data extractors
Supported DMS source and targets

31. https://aws.training
https://aws.amazon.com/training/path-databases
https://aws.amazon.com/training/path-advanced-networking
https://bit.ly/aws-office-hours
https://bit.ly/africa-virtual-day
https://bit.ly/emea-summit
https://bit.ly/cobus-youtube
Useful links

32. Thank you!
© 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Cobus Bernard
Sr Developer Advocate
Amazon Web Services
@cobusbernard
cobusbernard
cobusbernard