Diese Präsentation wurde erfolgreich gemeldet.
Wir verwenden Ihre LinkedIn Profilangaben und Informationen zu Ihren Aktivitäten, um Anzeigen zu personalisieren und Ihnen relevantere Inhalte anzuzeigen. Sie können Ihre Anzeigeneinstellungen jederzeit ändern.

(BDT203) From Zero to NoSQL Hero: Amazon DynamoDB Tutorial | AWS re:Invent 2014

6.304 Aufrufe

Veröffentlicht am

Got data? Interested in learning about NoSQL? In this session, we take you from not knowing anything about Amazon DynamoDB to being able to build an advanced application on top of DynamoDB. We start with an overview of the service, basic fundamental concepts, and then dive right in to a hands-on follow along tutorial in which you: create your own table, make queries, add secondary indexes to existing tables, query against the secondary indexes, modify your indexes, as well as detect changes to your data in DynamoDB to build all kinds of analytics and complex event processing apps. You can walk in a novice with DynamoDB, but rest assured, you will walk out as a NoSQL expert ready to tackle large distributed systems problems with your database problems addressed with DynamoDB.

Veröffentlicht in: Technologie
  • Als Erste(r) kommentieren

(BDT203) From Zero to NoSQL Hero: Amazon DynamoDB Tutorial | AWS re:Invent 2014

  1. 1. November 12, 2014 | Las Vegas, NV David Yanacek, Amazon DynamoDB Jason Lambert, Here by Nokia
  2. 2. App/Web Tier Client TierRDBMS
  3. 3. App/Web Tier Client TierRDBMS
  4. 4. App/Web Tier Client TierData Tier Search Cache Blob Store RDBMS NoSQL Data Warehouse
  5. 5. Data Tier Search Cache Blob Store RDBMS NoSQL Data Warehouse logging analytics key/value simple query rich search hot reads complex queries and transactions
  6. 6. Data Tier Amazon DynamoDB Amazon RDS Amazon ElastiCache Amazon S3 Amazon Redshift Amazon CloudSearch logging analytics key/value simple query rich search hot reads complex queries and transactions
  7. 7. “DynamoDB has scaled effortlessly to match our company's explosive growth, doesn't burden our operations staff, and integrates beautifully with our other AWS assets”. “I love how DynamoDB enables us to provision our desired throughput, and achieve low latency and seamless scale, even with our constantly growing workloads.”
  8. 8. RDBMS Poor Availability Limited Scalability High Cost
  9. 9. lack of strong operational consistency every engineer needs to learn distributed systems complexity
  10. 10. Non-Relational Fast & Predictable Performance Seamless Scalability Easy Administration
  11. 11. database service automated operationspredictable performance fast development always durable low latencycost effective =
  12. 12. table
  13. 13. WRITES Continuously replicated to 3 zones Quorum acknowledgment Persisted to disk (custom SSD) READS Strongly or eventually consistent No trade-off in latency
  14. 14. “Our previous NoSQL database required almost a full time administrator to run. Now AWS takes care of it.” “Being optimized at AdRollmeans we spend more every month on snacks than we do on DynamoDB–and almost nothing on an ops team” Save MoneyReduce Effort
  15. 15. Weatherbugmobile app Lightning detection & alerting for 40M users/month Developed and tested in weeks, at “1/20th of the cost of the traditional DB approach” Super Bowl promotion Millions of interactions over a relatively short period of time Built the app in 3 days, from design to production-ready
  16. 16. database service automated operationspredictable performance fast development always durable low latencycost effective =
  17. 17. https://www.example.com/maps/Pike+Place+Market,+Seattle,+ WA/47.6206924,-122.3492979/@47.6151852,- 122.3516502,16z/data=!3m1!4b1!4m8!4m7!1m5!1m1!1s0x54906ab2c151d6cb:0x60441d5f633e9f1c!2m2!1d- 122.3420567!2d47.6101359!1m0
  18. 18. https://__________________________Place+Market,+Seattle,+ WA/47.6206924,-122.3492979/@47.6151852,- 122.3516502,16z/data=!3m1!4b1!4m8!4m7!1m5!1m1!1s0x54906ab2c151d6cb:0x60441d5f633e9f1c!2m2!1d- 122.3420567!2d47.6101359!1m0 https://short.example.com/c3b87de1
  19. 19. client Amazon DynamoDB Web Service Elastic Load Balancing URL Shortener App Target Website example.com website
  20. 20. client example.com website Web Service Elastic Load Balancing GET /u/d83 HTTP 1.1 301 Moved Permanently Location: http://www.example.com/maps/... Amazon DynamoDB
  21. 21. client example.com website Web Service Elastic Load Balancing GET /maps/Pike+Place... Amazon DynamoDB
  22. 22. ShortUrl RedirectTo CreatedBy CreateDate example.com/u/d83 <long url> David 2014-09-09 example.com/u/abc <long url> Alice 2013-08-17 example.com/u/123 <long url> 2013-12-21 example.com/u/ea2 <long url> Bob 2014-03-30 Url Table Table
  23. 23. ShortUrl RedirectTo CreatedBy CreateDate example.com/u/d83 <long url> David 2014-09-09 example.com/u/abc <long url> Alice 2013-08-17 example.com/u/123 <long url> 2013-12-21 example.com/u/ea2 <long url> Bob 2014-03-30 Url Table Item
  24. 24. ShortUrl RedirectTo CreatedBy CreateDate example.com/u/d83 <long url> David 2014-09-09 example.com/u/abc <long url> Alice 2013-08-17 example.com/u/123 <long url> 2013-12-21 example.com/u/ea2 <long url> Bob 2014-03-30 Url Table Attributes (schema-less)
  25. 25. ShortUrl RedirectTo CreatedBy CreateDate example.com/u/d83 <long url> David 2014-09-09 example.com/u/abc <long url> Alice 2013-08-17 example.com/u/123 <long url> 2013-12-21 example.com/u/ea2 <long url> Bob 2014-03-30 Url Table Primary Key (Hash Index)
  26. 26. Item (JSON Document)
  27. 27. String Set String Number Map Not pictured: •Binary •List •Null •Number Set •Binary Set Boolean
  28. 28. 64 KB 400 KB
  29. 29. dynamodb.putItem({ TableName:'Url', Item:{ ShortUrl:"short.example.com/4ua93", RedirectTo:"long.example.com/maps/Pike+Place...", IsLive:true, Tags:["map","directions"], Metadata:{ CreatedBy:"David", CreateDate:"2014-19-21T08:49", TargetViews:100 }, Stats:{ LastVisited:"2014-10-30T10:31", PageViews:326 } } },function(err,data){ if(err)print(err);// an error occurred elseprint(data);// successful response }); Request Response (empty)
  30. 30. dynamodb.getItem({ TableName:'Url', Key:{ "ShortUrl":"short.example.com/4ua93" }, ConsistentRead:true },function(err,data){ if(err)print(err); elseprint(data); }); Optional read-your-writes consistency
  31. 31. dynamodb.scan({ TableName:'Url', ProjectionExpression:'ShortUrl, ' +'RedirectTo, Stats.LastVisited', Limit:2 }).eachPage(function(err,data){ if(err)print(err); elseif(data)print(data); }); SQL-like expressions Paginated (max 1 MB per page)
  32. 32. dynamodb.scan({ TableName:'Url', FilterExpression: 'begins_with(RedirectTo, :domain) '+ 'AND contains(Tags, :tag)'+ 'AND ('+ ' Stats.PageViews> Metadata.TargetViews'+ ' OR '+ ' Stats.StarCount> Metadata.TargetStars'+ ')', ExpressionAttributeValues:{ ':domain':'long.example.com/', ':tag':'directions' } }).eachPage(function(err,data){ if(err)print(err); elseif(data)print(data); }); Nested conditions Compare existing attributes Filter Expressions and Improved Conditions!
  33. 33. Update when viewed
  34. 34. dynamodb.updateItem({ TableName:'Url', Key:{ "ShortUrl":"short.example.com/4ua93" }, UpdateExpression: 'SET Usage.Views= Usage.Views+ :increment_by, ' +'Usage.LastVisited= :now', ExpressionAttributeValues:{ ':increment_by':1, ':now':newDate().toISOString() }, ReturnValues:'ALL_NEW', ConditionExpression:'attribute_exists(ShortUrl)' },function(err,data){ if(err)print(err); elseprint(data); }); Increment views by 1
  35. 35. dynamodb.updateItem({ TableName:'Url', Key:{ "ShortUrl":"short.example.com/4ua93" }, UpdateExpression: 'SET Usage.Views= Usage.Views+ :increment_by, ' +'Usage.LastVisited= :now', ExpressionAttributeValues:{ ':increment_by':1, ':now':newDate().toISOString() }, ReturnValues:'ALL_NEW', ConditionExpression:'attribute_exists(ShortUrl)' },function(err,data){ if(err)print(err); elseprint(data); }); Writes are also free reads!
  36. 36. Images Table
  37. 37. Images Table User Image Date Link Bob aed4c 2013-10-01 s3://… Bob cf2e2 2013-09-05 s3://… Bob f93bae 2013-10-08 s3://… Alice ca61a 2013-09-12 s3://… Hash and range primary key schema (composite key)
  38. 38. Images Table User Image Date Link Bob aed4c 2013-10-01 s3://… Bob cf2e2 2013-09-05 s3://… Bob f93bae 2013-10-08 s3://… Alice ca61a 2013-09-12 s3://… Range: Index is sorted by range key, can do range queries Hash: Equality queries only
  39. 39. Images Table User Image Date Link Bob aed4c 2013-10-01 s3://… Bob cf2e2 2013-09-05 s3://… Bob f93bae 2013-10-08 s3://… Alice ca61a 2013-09-12 s3://… Bob Query for Bob’s Images Query FROM Images WHERE User=Bob
  40. 40. Images Table User Image Date Link Bob aed4c 2013-10-01 s3://… Bob cf2e2 2013-09-05 s3://… Bob f93bae 2013-10-08 s3://… Alice ca61a 2013-09-12 s3://… User Date Image Bob 2013-09-05 cf2e2 Bob 2013-10-01 aed4c Bob 2013-10-08 f93bae Alice 2013-09-12 ca61a Table ByDate Local Secondary Index Local secondary index on Date
  41. 41. Images Table User Image Date Link Bob aed4c 2013-10-01 s3://… Bob cf2e2 2013-09-05 s3://… Bob f93bae 2013-10-08 s3://… Alice ca61a 2013-09-12 s3://… User Date Image Bob 2013-09-05 cf2e2 Bob 2013-10-01 aed4c Bob 2013-10-08 f93bae Alice 2013-09-12 ca61a Table ByDate Local Secondary Index Query for Bob’s two most recent images Query FROM Image-Index WHERE User=Bob, DESC, LIMIT 2
  42. 42. Images Table ImageTags Table
  43. 43. ImageTags Table Image User aed4c Alice aed4c Bob f93bae Bob
  44. 44. ImageTags Table Image User aed4c Alice aed4c Bob f93bae Bob Hash and range primary key schema
  45. 45. Image User aed4c Alice aed4c Bob f93bae Bob ImageTags Table Bob Query FROM ImageTags WHERE Image=aed4c
  46. 46. ImageTags Table Image User aed4c Alice aed4c Bob f93bae Alice f93bae Bob Bob Tag Alice in Image f93bae
  47. 47. and/or range keyfor your table •Even moreflexible query patterns Distributed transaction managed for you by DynamoDB
  48. 48. ImageTags Table Image User aed4c Alice aed4c Bob f93bae Alice f93bae Bob Table
  49. 49. ImageTags Table Global secondary index on User, Image User Image Bob aed4c Bob f93bae Alice aed4c Alice f93bae ByUser Global Secondary Index Image User aed4c Alice aed4c Bob f93bae Alice f93bae Bob Table Alternate hash and range keys
  50. 50. ImageTags Table Alice Query for images tagged Alice User Image Bob aed4c Bob f93bae Alice aed4c Alice f93bae ByUser Global Secondary Index Image User aed4c Alice aed4c Bob f93bae Alice f93bae Bob Table Query FROM ImageTags-Index WHERE User=Alice
  51. 51. { Id: abecd, Players : [ Alice, Bob ], State : STARTED, Turn : Bob, Top-Right : O } Game Item
  52. 52. Amazon DynamoDB Alice Bob
  53. 53. Amazon DynamoDB Alice Bob Update: Top-Right : O Turn : Bob
  54. 54. Amazon DynamoDB Alice Bob Update: Top-Left : X Turn : Alice
  55. 55. Alice Bob (1) Amazon DynamoDB Bob (2) Bob (3)
  56. 56. Alice Bob (1) Amazon DynamoDB Bob (2) Bob (3)
  57. 57. Alice Bob (1) Amazon DynamoDB Bob (2) Bob (3)
  58. 58. Bob (1) Amazon DynamoDB Bob (2) Bob (3) State : STARTED, Turn : Bob, Top-Right : O
  59. 59. Bob (1) Amazon DynamoDB Bob (2) Bob (3) Update: Turn : Alice Top-Left : X Update: Turn : Alice Mid : X State : STARTED, Turn : Bob, Top-Right : O Update: Turn : Alice Low-Right : X
  60. 60. Bob (1) Amazon DynamoDB Bob (2) Bob (3) Update: Turn : Alice Top-Left : X Update: Turn : Alice Mid : X State : STARTED, Turn : Alice, Top-Right : O, Top-Left : X, Mid: X, Low-Right: X Update: Turn : Alice Low-Right : X
  61. 61. Update: Turn : Alice Low-Right : X Expect: Turn : Bob Low-Right : null Bob (1) Amazon DynamoDB Bob (2) Bob (3) Update: Turn : Alice Top-Left : X Expect: Turn : Bob Top-Left : null State : STARTED, Turn : Bob, Top-Right : O Update: Turn : Alice Mid : X Expect: Turn : Bob Mid : null
  62. 62. Bob (1) Amazon DynamoDB Bob (2) Bob (3) State : STARTED, Turn : Bob, Top-Right : O Update: Turn : Alice Top-Left : X Expect: Turn : Bob Top-Left : null Update: Turn : Alice Low-Right : X Expect: Turn : Bob Low-Right : null Update: Turn : Alice Mid : X Expect: Turn : Bob Mid : null
  63. 63. Bob (1) Amazon DynamoDB Bob (2) Bob (3) State : STARTED, Turn : Alice, Top-Right : O, Top-Left : X Update: Turn : Alice Top-Left : X Expect: Turn : Bob Top-Left : null Update: Turn : Alice Mid : X Expect: Turn : Bob Mid : null Update: Turn : Alice Low-Right : X Expect: Turn : Bob Low-Right : null
  64. 64. Amazon DynamoDB Bob Alice Update: Turn : Alice Top-Right : X Expect: Turn : Bob Top-Right : null Update: Turn : Bob Top-Left : X Expect: Turn : Alice Top-Left : null
  65. 65. { Id : abecd, Players : [ Alice, Bob ], State : STARTED, Turn : Bob, Top-Right: O, Version : 2 }
  66. 66. Bob (1) Bob (2) Time
  67. 67. Bob (1) Bob (2) Read Version: 1 Turn: Bob Version: 1 Turn: Bob Time
  68. 68. Bob (1) Bob (2) Read Read Version: 1 Turn: Bob Version: 1 Turn: Bob Version: 1 Turn: Bob Time
  69. 69. Bob (1) Bob (2) Time Read Read Version: 1 Turn: Bob Version: 1 Turn: Bob Version: 1 Turn: Bob Version: 2 Turn: Alice Top-Right: X Version: 2 Turn: Alice Top-Right: X
  70. 70. Bob (1) Bob (2) Read Read Version: 1 Turn: Bob Version: 1 Turn: Bob Version: 1 Turn: Bob Version: 2 Turn: Alice Top-Right: X Version: 2 Turn: Alice Top-Right: X Version: 2 Turn: Alice Top-Right: X Put If V=1 Time
  71. 71. Bob (1) Bob (2) Read Read Version: 1 Turn: Bob Version: 1 Turn: Bob Version: 1 Turn: Bob Version: 2 Turn: Alice Top-Right: X Version: 2 Turn: Alice Top-Right: X Version: 2 Turn: Alice Top-Right: X Put If V=1 Put If V=1 Time
  72. 72. @DynamoDBTable(tableName="TicTacToeGame") publicclassGame { privateString id; // ... other attributes privateLong version; @DynamoDBHashKey publicString getId(){...} @DynamoDBVersionAttribute publicLong getVersion(){...} }
  73. 73. @DynamoDBTable(tableName="TicTacToeGame") publicclassGame { privateString id; // ... other attributes privateLong version; @DynamoDBHashKey publicString getId(){...} @DynamoDBVersionAttribute publicLong getVersion(){...} }
  74. 74. // 1. Load Game Game game=dynamoDBMapper.load(Game.class,"cde3f"); // 2. See if the move is valid // 3. Update the Game object // 4. Save the game dynamoDBMapper.save(game);
  75. 75. Amazon Redshift Amazon EMR DynamoDB Amazon RDS Amazon CloudSearch dynamoDBMapper.save(game); transaction.commit();
  76. 76. Addresses to Geolocations
  77. 77. records, tracking and analytics
  78. 78. high performance run time data retrieval
  79. 79. –Preparing a new wave of data to hit the service, tripling our coverage and data count
  80. 80. •Chop the big data stream files into small files •Store those billions of files on a big CDN file system
  81. 81. http://navteq-maps.ovi.com.edgesuite.net/144/0206/23/87/53/206238753_r10_f5_x0_y2.jpg We split the world into chunks –lots of those chunks have no streets We couldn’t implement our own tracking and analytics $$$$$$$$$$$$
  82. 82. Future flexible API
  83. 83. Keeps an index of all large files Stores large files Origin Service
  84. 84. RDS
  85. 85. 406311934325785de138c33-fa48-4418- 8ffa-54cc05e72adabigFile8098.big A server can look up a DynamoDB record by key (~10ms), find the s3 location and use the Amazon S3 API to read file chunks (~60ms)
  86. 86. ~50% 80%
  87. 87. http://bit.ly/awsevals

×