Application Development and Data Modeling on Amazon DynamoDB
1. Basics Web Sessions Social Gaming Tagging Leaderboards Architecture
Application Development and Data Modeling on
Amazon DynamoDB
David Yanacek, Sr. Software Development Engineer,
Amazon DynamoDB
4. Basics Web Sessions Social Gaming Tagging Leaderboards Architecture
High performance databases
throughput elasticity
availability
latency
query flexibility
storage capacity
operational complexity
5. Basics Web Sessions Social Gaming Tagging Leaderboards Architecture
Scale
Performance
Predictable, consistent
Degraded performance
with scale
6. Basics Web Sessions Social Gaming Tagging Leaderboards Architecture
= more problems
Data sharding
Data caching
Provisioning
Cluster management Fault management
7. Basics Web Sessions Social Gaming Tagging Leaderboards Architecture
Undifferentiated
heavy lifting
8. Basics Web Sessions Social Gaming Tagging Leaderboards Architecture
Amazon DynamoDB
9. Basics Web Sessions Social Gaming Tagging Leaderboards Architecture
Highlights
Flexible
Durable storage
Predictable performance
Zero admin
Low latency
Large scale
Seamless scaling
Excellent availability
10. Basics Web Sessions Social Gaming Tagging Leaderboards Architecture
Plan
• What is DynamoDB?
• In-depth application modeling
– Website Session Data (basic data model)
– Tic Tac Toe (conditional writes)
– Image Tagging (indexes and querying)
– Gaming Leaderboard (materialized views)
• High-level use cases
– Application architecture, data models
11. Basics Web Sessions Social Gaming Tagging Leaderboards Architecture
Website Session Data
basic data model and APIs
Web Sessions
12. Basics Web Sessions Social Gaming Tagging Leaderboards Architecture
Website Session Data
Web Sessions
13. Basics Web Sessions Social Gaming Tagging Leaderboards Architecture
Website Session Data
EC2 Instance
(web server)
Users
(first hit to website)
set-cookie: {
“SessionId”: “af872e”,
“History”: [“index.html”]
}
Web Sessions
14. Basics Web Sessions Social Gaming Tagging Leaderboards Architecture
Website Session Data: ElastiCache
Elastic Load
Balancing
EC2 Instances
(web server)
Users
ElastiCache
• ElastiCache distributed
cache
• Fast, scales
• Node failures cause
loss of session data
• Adding and Removing
nodes is tricky
Web Sessions
15. Basics Web Sessions Social Gaming Tagging Leaderboards Architecture
Website Session Data: RDBMS
Elastic Load
Balancing
Users
RDS
• Durable
• Adding and Removing
nodes is tricky
• Database blips impact
site availability
• Traffic can interfere
with other DB
workloads
Web Sessions
EC2 Instances
(web server)
16. Basics Web Sessions Social Gaming Tagging Leaderboards Architecture
Website Session Data: DynamoDB
Elastic Load
Balancing
Users
• Durable
• Transparently scalable
• Low latency
• High availability
• Read consistency
• Simple to develop
• Simple to operate
DynamoDB
Web Sessions
EC2 Instances
(web server)
17. Basics Web Sessions Social Gaming Tagging Leaderboards Architecture
DynamoDB Session Handling
Sessions Table
SessionId Last Accessed Shopping Cart
dc32f 2014-06-10T12:... {“Products”: [“Harry Potter DVD”]}
2d231 2014-06-10T12:... {“Products”: [“Amazon Fire Phone”]}
e0da3 2014-06-10T12:... {“Products”: [“Kindle Fire”]}
13f47 2014-06-10T12:...
Web Sessions
18. Basics Web Sessions Social Gaming Tagging Leaderboards Architecture
GetItem: Getting the current session
Web Sessions
19. Basics Web Sessions Social Gaming Tagging Leaderboards Architecture
PutItem: Updating the session state
Web Sessions
20. Basics Web Sessions Social Gaming Tagging Leaderboards Architecture
Using higher-level SDKs
Web Sessions
22. Basics Web Sessions Social Gaming Tagging Leaderboards Architecture
Offline development
• DynamoDB Local: Offline DynamoDB for testing,
development
Web Sessions
23. Basics Web Sessions Social Gaming Tagging Leaderboards Architecture
Recap: Website Session Data
• DynamoDB’s scalability, performance, availability,
and durability, and consistency make it a great fit
for website session data
• Very easy to use for your website – already
implemented open source
• Other key/value lookup use cases are natural fits
too
Web Sessions
24. Basics Web Sessions Social Gaming Tagging Leaderboards Architecture
Game State
conditional writes
Social Gaming
25. Basics Web Sessions Social Gaming Tagging Leaderboards Architecture
Tic Tac Toe
Social Gaming
26. Basics Web Sessions Social Gaming Tagging Leaderboards Architecture
Tic Tac Toe
{
Id : abecd,
Players : [ Alice, Bob ],
State : STARTED,
Turn : Bob,
Top-Right : O
}
Game Item
Social Gaming
27. Basics Web Sessions Social Gaming Tagging Leaderboards Architecture
Tic Tac Toe
Amazon
DynamoDB
Alice Bob
Social Gaming
28. Basics Web Sessions Social Gaming Tagging Leaderboards Architecture
Tic Tac Toe
Amazon
DynamoDB
Alice Bob
Update:
Top-Right : O
Turn : Bob
Social Gaming
29. Basics Web Sessions Social Gaming Tagging Leaderboards Architecture
Tic Tac Toe
Amazon
DynamoDB
Alice Bob
Update:
Top-Left : X
Turn : Alice
Social Gaming
30. Basics Web Sessions Social Gaming Tagging Leaderboards Architecture
Gaming the System
Alice Bob (1)
Amazon
DynamoDB
Bob (2) Bob (3)
Social Gaming
31. Basics Web Sessions Social Gaming Tagging Leaderboards Architecture
Gaming the System
Alice Bob (1)
Amazon
DynamoDB
Bob (2) Bob (3)
Social Gaming
32. Basics Web Sessions Social Gaming Tagging Leaderboards Architecture
Gaming the System
Alice Bob (1)
Amazon
DynamoDB
Bob (2) Bob (3)
Social Gaming
33. Basics Web Sessions Social Gaming Tagging Leaderboards Architecture
Gaming the System
Bob (1)
Amazon
DynamoDB
Bob (2)
Bob (3)
State : STARTED,
Turn : Bob,
Top-Right : O
Social Gaming
34. Basics Web Sessions Social Gaming Tagging Leaderboards Architecture
Gaming the System
Bob (1)
Amazon
DynamoDB
Bob (2)
Bob (3)
Update:
Turn : Alice
Top-Left : X
Update:
Turn : Alice
Mid : X
State : STARTED,
Turn : Bob,
Top-Right : O
Update:
Turn : Alice
Low-Right : X
Social Gaming
35. Basics Web Sessions Social Gaming Tagging Leaderboards Architecture
Gaming the System
Bob (1)
Amazon
DynamoDB
Bob (2)
Bob (3)
Update:
Turn : Alice
Top-Left : X
Update:
Turn : Alice
Mid : X
State : STARTED,
Turn : Alice,
Top-Right : O,
Top-Left : X,
Mid: X,
Low-Right: X
Update:
Turn : Alice
Low-Right : X
Social Gaming
36. Basics Web Sessions Social Gaming Tagging Leaderboards Architecture
Conditional Writes
• Accept a write only if values are as expected
• Otherwise reject the write
• Performance like normal writes
Social Gaming
37. Basics Web Sessions Social Gaming Tagging Leaderboards Architecture
Tic Tac Toe (Fixed)
Bob (1)
Amazon
DynamoDB
Bob (2)
Bob (3)
Update:
Turn : Alice
Top-Left : X
Expect:
Turn : Bob
Top-Left : null
State : STARTED,
Turn : Bob,
Top-Right : O
Update:
Turn : Alice
Mid : X
Expect:
Turn : Bob
Mid : null
Update:
Turn : Alice
Low-Right : X
Expect:
Turn : Bob
Low-Right : null
Social Gaming
38. Basics Web Sessions Social Gaming Tagging Leaderboards Architecture
Tic Tac Toe (Fixed)
Bob (1)
Amazon
DynamoDB
Bob (2)
Bob (3)
State : STARTED,
Turn : Bob,
Top-Right : O
Update:
Turn : Alice
Top-Left : X
Expect:
Turn : Bob
Top-Left : null
Update:
Turn : Alice
Low-Right : X
Expect:
Turn : Bob
Low-Right : null
Update:
Turn : Alice
Mid : X
Expect:
Turn : Bob
Mid : null
Social Gaming
39. Basics Web Sessions Social Gaming Tagging Leaderboards Architecture
Tic Tac Toe (Fixed)
Bob (1)
Amazon
DynamoDB
Bob (2)
Bob (3)
State : STARTED,
Turn : Alice,
Top-Right : O,
Top-Left : X
Update:
Turn : Alice
Top-Left : X
Expect:
Turn : Bob
Top-Left : null
Update:
Turn : Alice
Mid : X
Expect:
Turn : Bob
Mid : null
Update:
Turn : Alice
Low-Right : X
Expect:
Turn : Bob
Low-Right : null
Social Gaming
40. Basics Web Sessions Social Gaming Tagging Leaderboards Architecture
Recap: Game State
• UpdateItem can update specific attributes
• PutItem replaces the entire item
• Conditional writes synchronize state transitions
Social Gaming
41. Basics Web Sessions Social Gaming Tagging Leaderboards Architecture
Image Tagging
querying tables, secondary indexes
Tagging
42. Basics Web Sessions Social Gaming Tagging Leaderboards Architecture
Image Tagging: Requirements
• Query a user’s images
• Query a user’s images by date
• Tag other users in images
• Query images a user is tagged in
Tagging
43. Basics Web Sessions Social Gaming Tagging Leaderboards Architecture
Image Tagging
Images Table
Tagging
44. Basics Web Sessions Social Gaming Tagging Leaderboards Architecture
Image Tagging
Images Table
User Image Date Link
Bob aed4c 2013-10-01 s3://…
Bob cf2e2 2013-09-05 s3://…
Bob f93bae 2013-10-08 s3://…
Alice ca61a 2013-09-12 s3://…
Tagging
45. Basics Web Sessions Social Gaming Tagging Leaderboards Architecture
Image Tagging
Images Table
User Image Date Link
Bob aed4c 2013-10-01 s3://…
Bob cf2e2 2013-09-05 s3://…
Bob f93bae 2013-10-08 s3://…
Alice ca61a 2013-09-12 s3://…
Hash and Range primary key schema (composite key)
Range: Index is sorted by range key, can do range queries
Hash: Equality queries only
Tagging
46. Basics Web Sessions Social Gaming Tagging Leaderboards Architecture
Image Tagging
Images Table
User Image Date Link
Bob aed4c 2013-10-01 s3://…
Bob cf2e2 2013-09-05 s3://…
Bob f93bae 2013-10-08 s3://…
Alice ca61a 2013-09-12 s3://…
Bob
Query for Bob’s Images
Query
FROM Images
WHERE User=Bob
Tagging
47. Basics Web Sessions Social Gaming Tagging Leaderboards Architecture
Image Tagging
• Query a user’s images
• Query a user’s images by date
• Tag other users in images
• Query images a user is tagged in
Tagging
48. Basics Web Sessions Social Gaming Tagging Leaderboards Architecture
Local Secondary Indexes
• Alternate Range Key for your table
• More flexible Query patterns
• Local to the Hash Key
– (more query options within “Bob’s” images)
Tagging
49. Basics Web Sessions Social Gaming Tagging Leaderboards Architecture
Image Tagging
Images Table
User Image Date Link
Bob aed4c 2013-10-01 s3://…
Bob cf2e2 2013-09-05 s3://…
Bob f93bae 2013-10-08 s3://…
Alice ca61a 2013-09-12 s3://…
User Date Image
Bob 2013-09-05 cf2e2
Bob 2013-10-01 aed4c
Bob 2013-10-08 f93bae
Alice 2013-09-12 ca61a
Table ByDate Local Secondary Index
Local Secondary Index on Date
Tagging
50. Basics Web Sessions Social Gaming Tagging Leaderboards Architecture
Image Tagging
Images Table
User Image Date Link
Bob aed4c 2013-10-01 s3://…
Bob cf2e2 2013-09-05 s3://…
Bob f93bae 2013-10-08 s3://…
Alice ca61a 2013-09-12 s3://…
User Date Image
Bob 2013-09-05 cf2e2
Bob 2013-10-01 aed4c
Bob 2013-10-08 f93bae
Alice 2013-09-12 ca61a
Table ByDate Local Secondary Index
Query for Bob’s
two most recent imagesQuery
FROM Image-Index
WHERE User=Bob, DESC, LIMIT 2
Tagging
51. Basics Web Sessions Social Gaming Tagging Leaderboards Architecture
Image Tagging
• Query a user’s images
• Query a user’s images by date
• Tag other users in images
• Query images a user is tagged in
Tagging
52. Basics Web Sessions Social Gaming Tagging Leaderboards Architecture
Image Tagging
Images
Table
ImageTags
Table
Tagging
53. Basics Web Sessions Social Gaming Tagging Leaderboards Architecture
Image Tagging
ImageTags Table
Image User
aed4c Alice
aed4c Bob
f93bae Bob
Tagging
54. Basics Web Sessions Social Gaming Tagging Leaderboards Architecture
Image Tagging
ImageTags Table
Image User
aed4c Alice
aed4c Bob
f93bae Bob
Hash and Range Primary Key Schema
Tagging
55. Basics Web Sessions Social Gaming Tagging Leaderboards Architecture
Image User
aed4c Alice
aed4c Bob
f93bae Bob
Image Tagging
ImageTags Table
Bob
Query
FROM ImageTags
WHERE Image=aed4c
Tagging
56. Basics Web Sessions Social Gaming Tagging Leaderboards Architecture
Image Tagging
ImageTags Table
Image User
aed4c Alice
aed4c Bob
f93bae Alice
f93bae Bob
Bob
Tag Alice in Image f93bae
Tagging
57. Basics Web Sessions Social Gaming Tagging Leaderboards Architecture
Image Tagging
• Query a user’s images
• Query a user’s images by date
• Tag other users in images
• Query images a user is tagged in
Tagging
58. Basics Web Sessions Social Gaming Tagging Leaderboards Architecture
Global Secondary Indexes
• Alternate Hash and/or Range Key for your table
• Even more flexible Query patterns
Tagging
59. Basics Web Sessions Social Gaming Tagging Leaderboards Architecture
Image Tagging
ImageTags Table
Image User
aed4c Alice
aed4c Bob
f93bae Alice
f93bae Bob
Table
Tagging
60. Basics Web Sessions Social Gaming Tagging Leaderboards Architecture
Image Tagging
ImageTags Table
Global Secondary Index on User, Image
User Image
Bob aed4c
Bob f93bae
Alice aed4c
Alice f93bae
ByUser Global Secondary Index
Image User
aed4c Alice
aed4c Bob
f93bae Alice
f93bae Bob
Table
Alternate Hash and Range Keys
Tagging
61. Basics Web Sessions Social Gaming Tagging Leaderboards Architecture
Image Tagging
ImageTags Table
Alice
Query for images tagged Alice
User Image
Bob aed4c
Bob f93bae
Alice aed4c
Alice f93bae
ByUser Global Secondary Index
Image User
aed4c Alice
aed4c Bob
f93bae Alice
f93bae Bob
Table
Query
FROM ImageTags-Index
WHERE User=Alice
Tagging
62. Basics Web Sessions Social Gaming Tagging Leaderboards Architecture
Recap: Image Tagging
• Schema design on DynamoDB is all about the
indexes
• Local Secondary Indexes support flexible queries
• Global Secondary Indexes unlock even more
flexible queries
• Many other use cases for indexing metadata on
DynamoDB
Tagging
63. Basics Web Sessions Social Gaming Tagging Leaderboards Architecture
Social Leaderboard
materialized views
Leaderboards
64. Basics Web Sessions Social Gaming Tagging Leaderboards Architecture
Friends Leaderboard
• Find friends playing the same game
• See where you stack up against your friends
Leaderboards
65. Basics Web Sessions Social Gaming Tagging Leaderboards Architecture
Friends Leaderboard
User HighScore
Chuck 20
Alice 1000
Bob 470
Dave 850
Erin 580
Friends Table HighScores Table
User Friend
Bob Alice
Bob Chuck
Alice Bob
Alice Chuck
Alice Erin
Leaderboards
66. Basics Web Sessions Social Gaming Tagging Leaderboards Architecture
Friends Leaderboard
User HighScore
Chuck 20
Alice 1000
Bob 470
Dave 850
Erin 580
Friends Table HighScores Table
User Friend
Bob Alice
Bob Chuck
Alice Bob
Alice Chuck
Alice Erin
Alice
Leaderboards
67. Basics Web Sessions Social Gaming Tagging Leaderboards Architecture
Friends Leaderboard
User HighScore
Chuck 20
Alice 1000
Bob 470
Dave 850
Erin 580
Friends Table HighScores Table
User Friend
Bob Alice
Bob Chuck
Alice Bob
Alice Chuck
Alice Erin
1. Query 2. BatchGetItemAlice
Leaderboards
68. Basics Web Sessions Social Gaming Tagging Leaderboards Architecture
Inefficiencies in friends leaderboard
• Always retrieve all friends
– Some friends don’t play
– Might care about top 10 scores only
• Single-item operations for each friend
– Retrieving 10 items individually is more expensive
than 10 items in a row
Leaderboards
69. Basics Web Sessions Social Gaming Tagging Leaderboards Architecture
Improved Friends Leaderboard: Reads
User Friend H.S.
Bob Alice 1000
Bob Chuck 20
Alice Bob 470
Alice Chuck 20
Alice Erin 580
User HighScore
Chuck 20
Alice 1000
Bob 470
Dave 850
Erin 580
FriendScores Table HighScores Table
Leaderboards
70. Basics Web Sessions Social Gaming Tagging Leaderboards Architecture
Improved Friends Leaderboard: Reads
User Friend H.S.
Bob Alice 1000
Bob Chuck 20
Alice Bob 470
Alice Chuck 20
Alice Erin 580
User HighScore
Chuck 20
Alice 1000
Bob 470
Dave 850
Erin 580
User H.S. Friend
Bob 20 Chuck
Bob 1000 Alice
Alice 20 Chuck
Alice 470 Bob
Alice 580 Erin
Table Local Secondary Index
FriendScores Table HighScores Table
Leaderboards
71. Basics Web Sessions Social Gaming Tagging Leaderboards Architecture
Improved Friends Leaderboard: Reads
User Friend H.S.
Bob Alice 1000
Bob Chuck 20
Alice Bob 470
Alice Chuck 20
Alice Erin 580
User HighScore
Chuck 20
Alice 1000
Bob 470
Dave 850
Erin 580
User H.S. Friend
Bob 20 Chuck
Bob 1000 Alice
Alice 20 Chuck
Alice 470 Bob
Alice 580 Erin
Table Local Secondary Index
FriendScores Table HighScores Table
Query
FROM FriendScores-Index
WHERE User=Alice, DESC, LIMIT 10Alice
Leaderboards
72. Basics Web Sessions Social Gaming Tagging Leaderboards Architecture
User HighScore
Chuck 30
Alice 1000
Bob 470
Dave 850
Erin 580
Improved Friends Leaderboard: Writes
User Friend H.S.
Bob Alice 1000
Bob Chuck 30
Alice Bob 470
Alice Chuck 30
Alice Erin 580
FriendScores Table HighScores Table
Chuck
1. UpdateItem
2. UpdateItem(s)
Leaderboards
73. Basics Web Sessions Social Gaming Tagging Leaderboards Architecture
Recap: Friends Leaderboard
• BatchGetItem helps retrieve many items
• Query is cheaper than retrieving items
individually
• Trade off: reads can be made faster, cheaper, by
doing more writes
Leaderboards
74. Basics Web Sessions Social Gaming Tagging Leaderboards Architecture
High-level use cases
application architecture
Architecture
75. Basics Web Sessions Social Gaming Tagging Leaderboards Architecture
High-level use cases
• Online game coordination
• Real-time ad impression bids
• Indexed event logging
• Social messaging
• Real-time voting
• Real-time analytics
Architecture
76. Basics Web Sessions Social Gaming Tagging Leaderboards Architecture
Online Game Coordination
Availability Zone #1
DynamoDB
region
EC2
Availability Zone #1
EC2
EC2
CloudFron
t
Amazon S3
(content)
(client app)
(game servers)
(lobby, stats service)
Architecture
77. Basics Web Sessions Social Gaming Tagging Leaderboards Architecture
Online Game Coordination
Availability Zone #1
DynamoDB
region
EC2
Availability Zone #1
EC2
EC2
1) Login
2) Request matchmaking
3) Get game server IP
CloudFron
t
Amazon S3
Architecture
78. Basics Web Sessions Social Gaming Tagging Leaderboards Architecture
Online Game Coordination
Availability Zone #1
DynamoDB
region
EC2
Availability Zone #1
EC2
EC2
1) Login
2) Request matchmaking
3) Get game server IP
4) Connect to server
5) Pull down assets
6) Other players join
CloudFron
t
Amazon S3
Architecture
79. Basics Web Sessions Social Gaming Tagging Leaderboards Architecture
Online Game Coordination
Availability Zone #1
DynamoDB
region
EC2
Availability Zone #1
EC2
EC2
1) Login
2) Request matchmaking
3) Get game server IP
4) Connect to server
5) Pull down assets
6) Other players join
7) Game ends
8) Update stats
CloudFron
t
Amazon S3
Architecture
80. Basics Web Sessions Social Gaming Tagging Leaderboards Architecture
Indexed event logging
User DateTime Product
David 2014-06-22T21:01 Fire TV
David 2014-06-22T21:02 Fire Phone
David 2014-06-23T08:32 Fire Game Controller
Alice 2014-06-23… …
… … …
Index webpage clicks by user
Architecture
81. Basics Web Sessions Social Gaming Tagging Leaderboards Architecture
Real-time ad impression bids
Architecture
82. Basics Web Sessions Social Gaming Tagging Leaderboards Architecture
Social networks, messaging
Thread MessageId User Message
Fri Night 21:01.1 David Hi!
Fri Night 21:01.2 Steve What’s up?
Fri Night 21:02.1 David Happy Hour?
Participants Table,
Secondary Index
Messages Table
Thread User
Fri Night David
Fri Night Steve
DynamoDB? David
User Thread
David DynamoDB?
David Fri Night
Steve Fri Night
Architecture