遊戲數據解決方案 - AWS Athena 使用案例 - DynamoDB 中國客戶使用案例分享 講師: Candy Wu, Business Development Manager, Amazon Web Services Zhao Fei, Solutions Architect, Amazon Web Services

1. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Candy Wu 吳薇庭 Business Developement Manager
2018/3/8
Amazon Athena
無伺服器互動式查詢服務

2. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Agenda
• Amazon Athena 概述
• 主要特點
• 使用案例

3. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
使用者面臨的挑戰
• 在 Amazon S3 中分析資料時需要的工作量很大
• 需要具備專業知識才能管理 Hadoop 群集或資料
倉庫

4. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Amazon Athena 簡介
Amazon Athena 是一種互動式查詢服務，可
使用此服務通過標準 SQL ，直接對 Amazon
S3 中的資料進行分析

5. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
特點1. Athena 是Serveless的
• 無基礎設施或管理
• 零啟動時間
• 透明升級

6. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
特點2. Amazon Athena 易於使用
• 登錄到控制台
• 創建表
• 鍵入 Hive DDL 語句
• 使用控制台的添加表嚮導
• 開始查詢

7. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
底層使用大家熟知的技術
用於 SQL 查詢
In-memory 分散式查詢引擎
相容ANSI-SQL的查詢語句
用於 DDL 功能
複雜data type
多種資料格式
支援資料分區

8. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
特點3. Amazon Athena 高度可用
• Athena 跨多個可用區使用計算池
• 您的資料位於 Amazon S3 中，而 Amazon S3 也具有較
高的可用性，為實現 99.999999999% 的持久性而設計

9. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
特點4. 直接從 Amazon S3 查詢資料
• 直接從s3中提取資料, 不需要載入
• 按資料的原始格式查詢資料
• 文本、CSV、JSON、Web 日誌、AWS 服務日誌
• 轉換為經優化的格式（如 ORC 或 Parquet），以實
現最佳性能和最低成本
• 利用 Amazon S3 持久性和可用性

10. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
特點5. 使用 ANSI SQL
• 支援複雜聯接、嵌套查詢、
window function
• 支援複雜資料類型（array、
structs）
• 支援按任意鍵對資料進行分區
• 例如，年、月、日、小時或自
訂鍵、日期

11. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
特點6. Amazon Athena 支援多種資料格式
• 文字檔，例如 CSV、原始日誌
• Apache Web 日誌、TSV 文件
• JSON（簡單，嵌套）
• 壓縮檔
• 列式格式，例如 Apache Parquet 和 Apache ORC
• AVRO 支持 - 即將推出

12. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
特點7. Amazon Athena 的運行速度非常快
• 調優性能
• 自動並行處理查詢
• 結果將資料流到控制台
• 結果也存儲在 S3 中
• 提升查詢性能
• 壓縮數據
• 使用列式格式

13. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Presto vs. Athena
25
330
276
9.42 14.33 14.1
0
50
100
150
200
250
300
350
Query 1 Query 2 Query 3
Presto VS. Athena
Presto Athena
select sum(lo_extendedprice*lo_discount) as revenue
from lineorder, dwdate
where lo_orderdate = d_datekey and d_year = 1997
and lo_discount between 1 and 3 and lo_quantity < 24;
select sum(lo_revenue), d_year, p_brand1
from lineorder, dwdate, part, supplier
where lo_orderdate = d_datekey
and lo_partkey = p_partkey and lo_suppkey = s_suppkey and p_category = 'MFGR#12'
and s_region = 'AMERICA'
group by d_year, p_brand1
order by d_year, p_brand1;
select c_city, s_city, d_year, sum(lo_revenue) as revenue
from customer, lineorder, supplier, dwdate
where lo_custkey = c_custkey and lo_suppkey = s_suppkey
and lo_orderdate = d_datekey and (c_city='UNITED KI1' or c_city='UNITED KI5')
and (s_city='UNITED KI1' or s_city='UNITED KI5') and d_yearmonth = 'Dec1997'
group by c_city, s_city, d_year
order by d_year asc, revenue desc;
Query 1
Query 2
Query 3
各表條目數：lineorder 6億，part 140萬，customer 300萬，supplier 100萬，dwdate 2556
Presto集群使用3台 r3.xlarge.

14. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
特點8. Amazon Athena 是經濟高效的
• 按查詢掃描過的資料量付費
• 從 S3 掃描每 TB 資料的成本為 5 美元
• DDL 查詢和失敗查詢是免費的
• 使用壓縮、列式格式、分區來節省成本

15. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
• 尋求處理存儲在 Amazon S3 中的資料的任何人
• 數據來自 IOT 設備、Apache 日誌、Omniture 日誌、CDN 日
誌、應用程式日誌
• 瞭解 SQL 的任何人
• 開發人員或分析人員
• 臨時探究資料和資料發現
• 尋求在 Amazon S3 上構建數據湖的客戶
Athena 的目標使用者

16. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
示例場景

17. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
示例場景
使用 Athena 臨時訪問原始資料

18. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
示例場景
使用Athena 查詢聚合資料集使用 Athena 臨時訪問原始資料

19. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
回顧一下挑戰
在 Amazon S3 中分析資料時需要的工作量很大
不需要 ETL。不載入數據。在資料所在位置查詢資料
需要具備專業知識才能管理 Hadoop 群集或資料倉庫
沒有要管理的基礎設施

20. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
按查詢付費 — 每掃描 1TB 數據支付 5 美元
• 根據查詢掃描的資料量付費
• 節省成本的方法
• 壓縮
• 轉換為列式格式
• 使用分區
• 免費：DDL 查詢，失敗的查詢
資料集 Amazon S3 上的大小 查詢執行時間 掃描的資料量 費用
存儲為文字檔的日誌 1TB 237 秒 1.15TB 5.75 美元
以 Apache Parquet 格
式存儲的日誌*
130GB 5.13 秒 2.69GB 0.013 美元
節省 使用 Parquet 可節省
87% 的成本
速度快 34 倍 掃描的資料量少 99% 成本低 99.7%

21. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
使用案例

22. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

23. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Athena 是 Amazon Redshift 和 Amazon EMR 的補充
Amazon S3
EMR Athena
QuickSight
Redshift

24. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
© 2015, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
案例分享
S3
Athena
即時遊戲event logs 上傳到S3 直接查詢
Fluentd
使用情境：
針對單筆客訴，查出該玩家所有活動。
每日做遊戲KPI分析，如：不重複玩家登入狀況、DAU
分析等。
效益
用每個月不到$100美金的低成本，輕鬆處理個案，進行
遊戲關鍵數據分析！
未來可進一步作為遊戲資料庫資料歸檔、查詢的應用。

25. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
回顧一下
特點1. 無伺服器的
特點2. 易於使用, 控制台/JDBC/BI
特點3. 高度可用
特點4. 直接從 Amazon S3 查詢資料
特點5. 使用 ANSI SQL
特點6. 支持多種資料格式
特點7. 運行速度快
特點8. 經濟高效

26. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Athena
雅典娜
智慧與技藝的女神

27. Zhao Fei Solution Architect
2018-3-8
How to use DynamoDB in
Gaming

28. Fully managed
Fast, consistent performance
Highly scalable
Flexible
Event-driven programming
AWS GAMING
WHY DYNAMODB?

29. Table
Items
Attributes
Partition
key
Sort
key
Mandatory
Key-value access pattern
Determines data distribution
Optional
Model 1:N relationships
Enables rich query capabilities
All items for key
==, <, >, >=, <=
“begins with”
“between”
“contains”
“in”
sorted results
counts
top/bottom N values
AWS GAMING
DYNAMODB TABLES

30. 00 55 A954 FFAA
• Partition Key uniquely identifies an item
• Partition Key is used for building an unordered hash index
• Allows table to be partitioned for scale
Id = 1
Name = Jim
Hash (1) = 7B
Id = 2
Name = Andy
Dept = Eng
Hash (2) = 48
Id = 3
Name = Kim
Dept = Ops
Hash (3) = CD
Key Space
AWS GAMING
PARTITION KEYS

31. • Partition: Sort Key uses two attributes together to uniquely identify an Item
• Within unordered hash index, data is arranged by the sort key
• No limit on the number of items (∞) per partition key
• Except if you have local secondary indexes
00:0 FF:∞
Hash (2) = 48
Customer# = 2
Order# = 10
Item = Pen
Customer# = 2
Order# = 11
Item = Shoes
Customer# = 1
Order# = 10
Item = Toy
Customer# = 1
Order# = 11
Item = Boots
Hash (1) = 7B
Customer# = 3
Order# = 10
Item = Book
Customer# = 3
Order# = 11
Item = Paper
Hash (3) = CD
55 A9:∞54:∞ AA
Partition 1 Partition 2 Partition 3
AWS GAMING
PARTITION SORT KEYS

32. Id = 2
Name = Andy
Dept = Engg
Id = 3
Name = Kim
Dept = Ops
Id = 1
Name = Jim
Id = 2
Name = Andy
Dept = Engg
Id = 3
Name = Kim
Dept = Ops
Id = 1
Name = Jim
Id = 2
Name = Andy
Dept = Engg
Id = 3
Name = Kim
Dept = Ops
Id = 1
Name = Jim
Replica 1
Replica 2
Replica 3
Partition 1 Partition 2 Partition N
AWS GAMING
PARTITION ARE 3-WAY REPLICATED

33. A1
(partition)
A3
(sort)
A2
(item key)
A1
(partition)
A2
(sort)
A3 A4 A5
LSIs A1
(partition)
A4
(sort)
A2
(item key)
A3
(projected)
Table
KEYS_ONLY
INCLUDE A3
A1
(partition)
A5
(sort)
A2
(item key)
A3
(projected)
A4
(projected)
ALL
10 GB max per partition key,
i.e. LSIs limit the # of range
keys!
LOCAL SECONDARY INDEX
AWS GAMING

34. A1
(partition)
A2 A3 A4 A5
GSIs A5
(partition)
A4
(sort)
A1
(item key)
A3
(projected)
Table
INCLUDE A3
A4
(partition)
A5
(sort)
A1
(item key)
A2
(projected)
A3
(projected) ALL
A2
(partition)
A1
(itemkey)
KEYS_ONLY
RCUs/WCUs provisioned
separately for GSIs
Online indexingGLOBAL SECONDARY INDEX
AWS GAMING

35. • LSI can be modeled as a GSI
• If data size in an item collection > 10 GB, use GSI
• If eventual consistency is okay for your scenario, use GSI!
AWS GAMING
LSI or GSI

36. Partition
Time
Heat
WHAT BAD NOSQL LOOKS LIKE
AWS GAMING

37. • “To get the most out of DynamoDB
throughput, create tables where the
hash key element has a large
number of distinct values, and
values are requested fairly uniformly,
as randomly as possible.”
• —DynamoDB Developer Guide
• Space: access is evenly spread
over the key-space
• Time: requests arrive evenly
spaced in time
GETTING THE MOST OF DYNAMODB THROUGHPUT
AWS GAMING

38. Partition
Heat
MUCH BETTER PICTURE
AWS GAMING
Time

39. • Use a table or GSI with an alternate partition key
• Use GetItem or BatchGetItem API
• Example: Given a Unique PlayerID or Email, get attributes
Users Table
Partiton key Attributes
PID= 12346789 Email = johndoe@nowhere.com, GamesOwned = SKU889,SKU778
PID = 98765432 Email = maryfowler@somewhere.com, GamesOwned = SKU988,SKU778
Users-Email-GSI
Partition key Attributes
Email=
johndoe@nowhere.com
PID= 12346789, GamesOwned = SKU889,SKU778
Email =
maryfowler@somewhere.
com
PID= 98765432, GamesOwned = SKU988,SKU778
1:1 RELATIONSHIPS OR KEY-VALUES
AWS GAMING

40. Games Played
Partition Key Sort key Attributes
PID = 1 ts = 1493661529447 MapId= 30, Duration = 42
PID = 1 ts = 1503561528088 MapId= 30, Duration = 36
• Use a table or GSI with partition and sort key
• Use Query API
• Example:
• Given a player, find all games played between timestamp X, Y
AWS GAMING
1:N RELATIONSHIPS OR PARENT-CHILDREN

41. User-Games-Table
Partition Key Sort key
UserId = bob GameId = Game1
UserId = fred GameId = Game2
UserId = bob GameId = Game3
Game-Users-GSI
Partition Key Sort key
GameId = Game1 UserId = bob
GameId = Game2 UserId = fred
GameId = Game3 UserId = bob
AWS GAMING
• Use a table and GSI with partition and sort key elements switched
• Use Query API
• Example: Given a user, find all games. Or given a game, find all users.
N:M RELATIONS

42. GameId Date Host Opponent Status
d9bl3 2014-10-02 David Alice DONE
72f49 2014-09-30 Alice Bob PENDING
o2pnb 2014-10-08 Bob Carol IN_PROGRESS
b932s 2014-10-03 Carol Bob PENDING
ef9ca 2014-10-03 David Bob IN_PROGRESS
Games Table Partition key
AWS GAMING
HIERARCHICAL DATA STRUCTURES

43. • DynamoDB indexes provide partiton and sort
• What about queries for two equalities and a sort?
SELECT * FROM Game
WHERE Opponent='Bob‘
AND Status=‘PENDING'
ORDER BY Date DESC
(hash)
(range)
(?)
AWS GAMING
QUERY FOR INCOMING GAME REQUESTS

44. Secondary Index
Opponent Date GameId Status Host
Alice 2014-10-02 d9bl3 DONE David
Carol 2014-10-08 o2pnb IN_PROGRESS Bob
Bob 2014-09-30 72f49 PENDING Alice
Bob 2014-10-03 b932s PENDING Carol
Bob 2014-10-03 ef9ca IN_PROGRESS David
Bob
Partition key Sort key
AWS GAMING
APPROACH 1: QUERY FILTER

45. Secondary Index
Bob
Opponent Date GameId Status Host
Alice 2014-10-02 d9bl3 DONE David
Carol 2014-10-08 o2pnb IN_PROGRESS Bob
Bob 2014-09-30 72f49 PENDING Alice
Bob 2014-10-03 b932s PENDING Carol
Bob 2014-10-03 ef9ca IN_PROGRESS David
SELECT * FROM Game
WHERE Opponent='Bob'
ORDER BY Date DESC
FILTER ON Status='PENDING'
(filtered out)
AWS GAMING
APPROACH 1: QUERY FILTER

46. Bob
AWS GAMING
NEEDLE IN A HAYSTACK

47. • Send back less data “on the wire”
• Simplify application code
• Simple SQL-like expressions
• AND, OR, NOT, ()
Important When: Your index isn’t entirely selective
AWS GAMING
USE QUERY FILTER

48. StatusDate
DONE_2014-10-02
IN_PROGRESS_2014-10-08
IN_PROGRESS_2014-10-03
PENDING_2014-09-30
PENDING_2014-10-03
Status
DONE
IN_PROGRESS
IN_PROGRESS
PENDING
PENDING
Date
2014-10-02
2014-10-08
2014-10-03
2014-10-03
2014-09-30
+ =
AWS GAMING
APPROACH 2: COMPOSITE KEY

49. Secondary Index
Opponent StatusDate GameId Host
Alice DONE_2014-10-02 d9bl3 David
Carol IN_PROGRESS_2014-10-08 o2pnb Bob
Bob IN_PROGRESS_2014-10-03 ef9ca David
Bob PENDING_2014-09-30 72f49 Alice
Bob PENDING_2014-10-03 b932s Carol
Partition key Sort key
AWS GAMING
APPROACH 2: COMPOSITE KEY

50. Opponent StatusDate GameId Host
Alice DONE_2014-10-02 d9bl3 David
Carol IN_PROGRESS_2014-10-08 o2pnb Bob
Bob IN_PROGRESS_2014-10-03 ef9ca David
Bob PENDING_2014-09-30 72f49 Alice
Bob PENDING_2014-10-03 b932s Carol
Secondary Index
Bob
SELECT * FROM Game
WHERE Opponent='Bob'
AND StatusDate BEGINS_WITH 'PENDING'
AWS GAMING
APPROACH 2: COMPOSITE KEY

51. Bob
AWS GAMING
NEEDLE IN A SORTED HAYSTACK

52. •
Id
(Partition)
User Game Score Date Award
1 Bob G1 1300 2012-12-23
2 Bob G1 1450 2012-12-23
3 Jay G1 1600 2012-12-24
4 Mary G1 2000 2012-10-24 Champ
5 Ryan G2 123 2012-03-10
6 Jones G2 345 2012-03-20
Game-scores-table
Award
(Partition)
Id User Score
Champ 4 Mary 2000
Award-GSI
Scan sparse GSIs
AWS GAMING
SPARSE INDEXES

53. • Stream of updates to a table
• Asynchronous
• Exactly once
• Strictly ordered
• Per item
• Highly durable
• Scale with table
• 24-hour lifetime
• Sub-second latency
AWS GAMING
DYNAMODB STREAMS

54. View Type Destination
Old image—before update Name = John, Destination = Mars
New image—after update Name = John, Destination = Pluto
Old and new images Name = John, Destination = Mars
Name = John, Destination = Pluto
Keys only Name = John
UpdateItem (Name = John, Destination = Pluto)
AWS GAMING
VIEW TYPES

55. Stream
Table
Partition 1
Partition 2
Partition 3
Partition 4
Partition 5
Table
Shard 1
Shard 2
Shard 3
Shard 4
KCL
Worker
KCL
Worker
KCL
Worker
KCL
Worker
Kinesis Client Library
Application
DynamoDB
Client Application
Updates
AWS GAMING
STREAMS AND KCL

56. DynamoDB Streams
Open Source Cross-Region
Replication Library
Asia Pacific (Sydney) EU (Ireland) Replica
US East (N. Virginia)
AWS GAMING
CROSS REGION REPLICATION

57. Lambda function
Notify change
Derivative Tables
CloudSearch
ElastiCache
AWS GAMING
TRIGGERS

58. • Fully managed, highly available: handles all software management, fault
tolerant, replication across multi-AZs within a region
• DynamoDB API compatible: seamlessly caches DynamoDB API calls, no
application re-writes required
• Write-through: DAX handles caching for writes
• Flexible: Configure DAX for one table or many
• Scalable: scales-out to any workload with up to 10 read replicas
• Manageability: fully integrated AWS service: Amazon CloudWatch, Tagging for
DynamoDB, AWS Console
• Security: Amazon VPC, AWS IAM, AWS CloudTrail, AWS Organizations
DynamoDB
Your Application
DynamoDB Accelerator
Table #1
DYNAMODB ACCELERATOR-DAX
AWS GAMING

59. Q: How easy is it to add in-
memory caching with DAX?
A: Comment out the this code and
add this code
AWS GAMING

60. QUESTIONS?
<name>@amazon.com

61. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.