"In this session, you will learn how to easily access your data on S3, and how to visualize and generate insights from Amazon Athena and other data sources through Amazon QuickSight. In addition we will share some tips & best practices for using Athena & QuickSight. Amazon Athena is an interactive query service that makes it easy to analyze data in Amazon S3 using standard SQL. Amazon QuickSight is a fast, cloud-powered business analytics service that makes it easy to build visualizations, perform ad-hoc analysis, and quickly get business insights from various data sources (Amazon Redshift, Amazon Athena, Amazon EMR, Amazon RDS and more)."

1. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Daniel Haviv, Big Data Specialist Solutions Architect
June 2017
Interactive Analytics on AWS

2. Challenges Customers Faced
• Significant amount of work required to analyze data in
Amazon S3
• Users often only have access to aggregated data sets
• Managing a Hadoop cluster or data warehouse requires
expertise

3. Introducing Amazon Athena
Amazon Athena is an interactive query service
that makes it easy to analyze data directly from
Amazon S3 using Standard SQL

4. A Sample Pipeline

5. A Sample Pipeline
Ad-hoc access to raw data using SQL

6. A Sample Pipeline
Ad-hoc access to data using Athena
Athena can query
aggregated datasets as well

7. Athena is Serverless
• No Infrastructure or
administration
• Zero Spin up time
• Transparent upgrades
• Highly Available
• You connect to a service
endpoint or log into the
console

8. Amazon Athena is Easy To Use
• Log into the Console
• Create a table
• Type in a Hive DDL Statement
• Use the console Add Table wizard
• Start querying

9. Query Data Directly from Amazon S3
• No loading of data
• Query data in its raw format
• Text, CSV, JSON, weblogs, AWS service logs
• Convert to an optimized form like ORC or Parquet for the best
performance and lowest cost
• No ETL required
• Stream data from directly from Amazon S3
• Take advantage of Amazon S3 durability and availability

10. Use ANSI SQL
• Start writing ANSI SQL
• Support for complex joins, nested
queries & window functions
• Support for complex data types
(arrays, structs)
• Support for partitioning of data by
any key
• (date, time, custom keys)
• e.g., Year, Month, Day, Hour or
Customer Key, Date

11. Familiar Technologies Under the Covers
Used for SQL Queries
In-memory distributed query engine
ANSI-SQL compatible with extensions
Used for DDL functionality
Complex data types
Multitude of formats
Supports data partitioning

12. Amazon Athena is Cost Effective
• Pay per query
• $5 per TB scanned from S3
• DDL Queries and failed queries are free
• Save by using compression, columnar formats, partitions

13. • Anyone looking to process data stored in Amazon S3
• Data coming IOT Devices, Apache Logs, Omniture logs, CF logs,
Application Logs
• Anyone who knows SQL
• Both developers or Analysts
• Ad-hoc exploration of data and data discovery
• Customers looking to build a data lake on Amazon S3
Who is Athena for?

14. Accessing Amazon Athena
• Through the console
• Via the AWS API
• Using a JDBC client (either plain SQL client or BI tools)

15. Creating Tables and Querying Data

16. Example
CREATE EXTERNAL TABLE access_logs
(
ip_address String,
request_time Timestamp,
request_method String,
request_path String,
request_protocol String,
response_code String,
response_size String,
referrer_host String,
user_agent String
)
PARTITIONED BY (year STRING,month STRING, day STRING)
ROW FORMAT DELIMITED FIELDS TERMINATED BY 't'
STORED AS TEXTFILE
LOCATION 's3://YOUR-S3-BUCKET-NAME/access-log-processed/'
External = creates a view of this data.
When you delete the table, the data is
not deleted

17. Example
CREATE EXTERNAL TABLE access_logs
(
ip_address String,
request_time Timestamp,
request_method String,
request_path String,
request_protocol String,
response_code String,
response_size String,
referrer_host String,
user_agent String
)
PARTITIONED BY (year STRING,month STRING, day STRING)
ROW FORMAT DELIMITED FIELDS TERMINATED BY 't'
STORED AS TEXTFILE
LOCATION 's3://YOUR-S3-BUCKET-NAME/access-log-processed/'
Location = where data is stored.
In Athena this is mandated to be
in Amazon S3

18. Partitioning in Athena
- Follows Hive Semantics – i.e. Partitioned by any column
- Partition Column is not part of the data itself, but it still
queriable.
- Creates subdirectories/prefixes for each partition
- (alter table add partition date=XXX year =XXX)
- MSCK Repair Table

19. Example
CREATE EXTERNAL TABLE access_logs
(
ip_address String,
request_time Timestamp,
request_method String,
request_path String,
request_protocol String,
response_code String,
response_size String,
referrer_host String,
user_agent String
)
PARTITIONED BY (year STRING,month STRING, day STRING)
ROW FORMAT DELIMITED FIELDS TERMINATED BY 't'
STORED AS TEXTFILE
LOCATION 's3://YOUR-S3-BUCKET-NAME/access-log-processed/'
Partitioning allows you to limit what your
query runs on

20. Example
CREATE EXTERNAL TABLE access_logs
(
ip_address String,
request_time Timestamp,
request_method String,
request_path String,
request_protocol String,
response_code String,
response_size String,
referrer_host String,
user_agent String
)
PARTITIONED BY (year STRING,month STRING, day STRING)
ROW FORMAT DELIMITED FIELDS TERMINATED BY 't'
STORED AS TEXTFILE
LOCATION 's3://YOUR-S3-BUCKET-NAME/access-log-processed/'
Defines how your files are stored

21. Columnar Formats

22. Flat File Sample Layout
First Name Last Name Age Gender
Tootsie Label 34 Female
Miriam Le Fleming 25 Female
Blakeley Lisciandro 45 Female
Ernst Minghi 63 Male
Brew Jime 22 Male

23. Tootsie
Miriam
Blakeley
Ernst
Brew
Label
Le Fleming
Lisciandro
Minghi
Jime
34
25
45
63
22
Female
Female
Female
Male
Male
First Name Last Name Age Gender
MIN: Blakeley
MAX: Tootsie
MIN: Jime
MAX: Minghi
MIN: 22
MAX: 63
MIN: Female
MAX: Male
• Projection pushdown
• Predicate pushdown
• Improved compression ratios
Columnar Formats Layout (Parquet & ORC)

24. Converting to ORC and PARQUET
• You can use Hive CTAS to convert data
• CREATE TABLE new_key_value_store
• STORED AS PARQUET
• AS
• SELECT col_1, col2, col3 FROM noncolumartable
• SORT BY new_key, key_value_pair;
• You can also use Spark to convert the file into PARQUET / ORC
• 20 lines of Pyspark code, running on EMR
• Converts 1TB of text data into 130 GB of Parquet with snappy conversion
• Total cost $5
https://github.com/awslabs/aws-big-data-blog/tree/master/aws-blog-spark-parquet-conversion

25. Pay By the Query - $5/TB Scanned
• Pay by the amount of data scanned per query
• Ways to save costs
• Compress
• Convert to Columnar format
• Use partitioning
Dataset Size on Amazon S3 Query Run time Data Scanned Cost
Logs stored as Text
files
1 TB 237 seconds 1.15TB $5.75
Logs stored in
Apache Parquet
format*
130 GB 5.13 seconds 2.69 GB $0.013
Savings 87% less with Parquet 34x faster 99% less data scanned 99.7% cheaper

26. Amazon QuickSight is a Business Analytics Service that lets business users
quickly and easily visualize, explore, and share insights from their data.

27. QuickSight is designed to make it
easy and cost effective to enable
users across your organization to
consume data and perform ad-hoc
analytics.
Who Is QuickSight For?
Business Professionals
Data Consumers
Data
Professionals

28. QuickSight is deeply integrated
with AWS data sources like
Redshift, RDS, S3, Athena, EMR
(Spark & Presto) as well as third-
party sources like Excel,
Salesforce and more.
Deep Integration with
AWS Data Sources
Amazon RDS,
Aurora
Amazon
Redshift
Amazon
Athena
Amazon S3
Flat Files
Amazon
EMR

29. Demo

30. Thank you!