Diese Präsentation wurde erfolgreich gemeldet.
Die SlideShare-Präsentation wird heruntergeladen. ×

MongoDB World 2018: Using Change Streams to Keep Up with Your Data

Anzeige
Anzeige
Anzeige
Anzeige
Anzeige
Anzeige
Anzeige
Anzeige
Anzeige
Anzeige
Anzeige
Anzeige

Hier ansehen

1 von 51 Anzeige

Weitere Verwandte Inhalte

Diashows für Sie (20)

Ähnlich wie MongoDB World 2018: Using Change Streams to Keep Up with Your Data (20)

Anzeige

Weitere von MongoDB (20)

Aktuellste (20)

Anzeige

MongoDB World 2018: Using Change Streams to Keep Up with Your Data

  1. 1. Using Change Streams to Keep Up With Your Data Kevin Albertson, Software Engineer II
  2. 2. Kevin Albertson Software Engineer II, MongoDB
 @kevinAlbs
  3. 3. Why Real-Time Data?
  4. 4. March 8th, 2018
  5. 5. March 8th, 2018 Point A Point B
  6. 6. March 8th, 2018 6:33 6:45 6:54 Tw eet:everything ok Iarrive atstation Train scheduled 7:00 Stillw aiting 7:15 Stillw aiting
  7. 7. March 8th, 2018 6:33 6:45 6:54 Tw eet:everything ok Iarrive atstation Train scheduled 7:15 Stillw aiting 7:00 Stillw aiting
  8. 8. 6:33 6:45 6:54 Tw eet:everything ok Iarrive atstation Train scheduled 7:30 Stillw aiting 7:15 Stillw aiting 7:00 Stillw aiting
  9. 9. 6:33 6:45 6:54 Tw eet:everything ok Iarrive atstation Train scheduled 7:30 Stillw aiting 7:15 Stillw aiting 7:00 Stillw aiting 7:45 Stillw aiting
  10. 10. March 8th, 2018 6:33 6:45 6:54 Tw eet:everything ok Iarrive atstation Train scheduled 8:00 A nnouncem ent 7:30 Stillw aiting 7:15 Stillw aiting 7:00 Stillw aiting 7:45 Stillw aiting
  11. 11. March 8th, 2018
  12. 12. Unnecessary 6:33 6:45 6:54 Tw eet:everything ok Iarrive atstation Train scheduled 8:00 A nnouncem ent 7:30 Stillw aiting 7:15 Stillw aiting 7:00 Train passes 7:45 Stillw aiting Arrive in N YC 9:30 Take train in otherdirection 9:30 Take train to N YC 11:00
  13. 13. Real-time is an Expectation
  14. 14. Change Streams Enable Existing Apps to Provide Real-time Data
  15. 15. const changeStream = db.collection('train').watch(); c h a n g e S t r e a m . o n ( ' c h a n g e ' , ( c h a n g e ) = > { c o n s o l e . l o g ( c h a n g e ) } ) ; Added in 3.6 Improved in 4.0 { "_id" : (resumeToken), "operationType" : "insert", "ns" : { "db" : "test", "coll" : "train" }, "documentKey" : { "_id" : 123 }, "fullDocument" : { "_id" : 123, "text": "hello" }, }
  16. 16. Change Stream Characteristics
  17. 17. 1. Use Collection Access Controls 2. Present a Defined API 3. Scale Across Nodes Change Streams
  18. 18. Return 5 Operation Types 1. Insert 2. Update 3. Replace 4. Delete 5. Invalidate
  19. 19. mongos 3 2 1 Shard 1 Shard 2 Shard 3 Total Ordering of Changes Across Shards
  20. 20. Documents Uniquely Identified Shard key combined with _id to uniquely identify documents mongos Shard 2 Shard 3Shard 1
  21. 21. Changes are Durable S S collection.watch() P
  22. 22. Change Streams are Resumable collection.watch() PP { _id: <resumeToken>, operationType: 'update' ... } 
 SP S
  23. 23. Change Streams are Resumable changeStream.on ('change', (change) => { console.log(change); cachedResumeToken = change["_id"]; }); changeStream.on ('error', () => { if (cachedResumeToken) { establishChangeStream (cachedResumeToken); } });
  24. 24. Change Streams Utilize the Power of the Aggregation Framework $match $project $addFields $replaceRoot $redact var changeStream = coll.watch([{ $match: {operationType: {$in: ['delete', 'replace']}}}]);
  25. 25. 1. Collection Access Controls 2. Defined API 3. Enable Scaling 4. Total Ordering 5. Durable 6. Resumable 7. Power of Aggregation
  26. 26. Types of Changes
  27. 27. Insert > db.collection.watch([], {maxAwaitTimeMS: 30000}).pretty() > db.collection.insert({ _id: 1, text: "hello" }) Shell 1 Shell 2 { "_id" : (resumeToken), "operationType" : "insert", "ns" : { "db" : "test", "coll" : "test" }, "documentKey" : { "_id" : 1 }, "fullDocument" : { "_id" : 1, "text": "hello" }, }
  28. 28. Update > db.collection.watch([], {maxAwaitTimeMS: 30000}).pretty() > db.collection.updateOne({_id: 1}, { $set: {“text”: “updated”} }) Shell 1 Shell 2 { "_id" : (resumeToken), "operationType" : "update", "ns" : { "db" : "test", "coll" : "collection" }, "documentKey" : { "_id" : 1 }, "updateDescription" : { "updatedFields" : { "text" : "updated" }, "removedFields" : [ ] } }
  29. 29. Update with fullDocument: updateLookup > db.collection.watch([], { fullDocument: “updateLookup”, maxAwaitTimeMS: 30000 }).pretty() > db.collection.updateOne({_id: 1}, { $set: {“number”: “5”} }) Shell 1 Shell 2 { "_id" : (resumeToken), "operationType" : "update", "fullDocument" : { "_id" : 1, "text" : "updated", "number" : 5 }, "ns": {"db": "test", "coll": "collection"}, "documentKey" : { "_id" : 1 }, "updateDescription" : { "updatedFields" : { "number" : 5 }, "removedFields" : [ ] } }
  30. 30. Replace > db.collection.watch([], {maxAwaitTimeMS: 30000}).pretty() > db.collection.replaceOne({_id: 1}, {text: "replaced"}) Shell 1 Shell 2 { "_id" : (resumeToken), "operationType" : "replace", "ns" : { "db" : "test", "coll" : "collection" }, "documentKey" : { "_id" : 1 }, "fullDocument" : { "_id" : 1, "text" : "replaced" } }
  31. 31. Delete > db.collection.watch([], {maxAwaitTimeMS: 30000}).pretty() > db.collection.remove({_id: 1}) Shell 1 Shell 2 { "_id" : (resumeToken), "operationType" : "delete", "ns" : { "db" : "test", "coll" : "collection" }, "documentKey" : { "_id" : 1 } }
  32. 32. Invalidate > db.collection.watch([], {maxAwaitTimeMS: 30000}).pretty() > db.collection.drop() Shell 1 Shell 2 { "_id" : (token), "operationType" : "invalidate" }
  33. 33. New in 4.0
  34. 34. 1. startAtOperationTime 2. Change stream on db or client 3. Resumable before first change
  35. 35. Use Case
  36. 36. Real-time Train Arrival Estimates
  37. 37. Collection > db.train.find().pretty() { "_id" : 11, "position" : { "lat" : 40.04714, "lon" : -74.0351 } }
  38. 38. App Caches State of All Trains App Cache
  39. 39. _id: 11 (40.0, -74.0) db.train.find().readConcern("majority") App Cache Populate the Cache
  40. 40. _id: 11 (40.0, -74.0) db.train.watch([]) _id: (resumeToken), operationType: 'update', updateDescription: { updatedFields: { lat: 40.41169 } }, documentKey: {_id: 11} Handled _id: (resumeToken), operationType: 'update', updateDescription: { updatedFields: { passengerCount: 5 } } documentKey: {_id: 11} Handled? App Cache _id: 11 (40.4, -74.0) Watch for changes
  41. 41. $match to Filter Irrelevant Changes
  42. 42. db.train.watch([{ $match: { $or: [ {'updateDescription.updatedFields.lat': {$exists: 1}}, {'updateDescription.updatedFields.lon': {$exists: 1}} ] } }]) _id: (resumeToken), operationType: 'insert', fullDocument: { _id: 16, lat: 40.5889, lon: -76.1938 }, documentKey: {_id: 16} Ignored? _id: (resumeToken), operationType: 'update', updateDescription: { updatedFields: { passengerCount: 5 } } documentKey: {_id: 11} Ignored App Cache _id: 11 (40.4, -74.0)
  43. 43. Handle All Document Changes If Caching State of Documents:
  44. 44. db.train.watch([{ $match: { $or: [ { operationType: 'update', $or: [ {'updateDescription.updatedFields.lat': {$exists: 1}}, {'updateDescription.updatedFields.lon': {$exists: 1}} ] }, { operationType: { $in: ['delete', 'insert', 'replace'] } } ] } }]) App Cache _id: 11 (40.4, -74.0) _id: (resumeToken), operationType: 'insert', fullDocument: { _id: 16, lat: 40.5889, lon: -76.1938 }, documentKey: {_id: 16} Handled _id: 16 (40.4, -74.0) _id: (resumeToken), operationType: 'delete', documentKey: {_id: 11}Handled
  45. 45. Race Between Populating and Listening! db.train.find().readConcern("majority") . . . db.train.watch() update occurs!
  46. 46. find Client Server [ {_id: 11}…], $clusterTime: X watch, startAtOperationTime: X start session startAtOperationTime
  47. 47. startAtOperationTime const session = client.startSession(); const cursor = db.collection('train').find({}, { session: session }); populateApplicationCache(cursor); const changeStream = db.collection('train').watch([], { session: session, startAtOperationTime: session.operationTime }); changeStream.on('change', updateApplicationCache);
  48. 48. App Complete! 1. Doesn't miss updates 2. Handles network blips 3. Reduces bandwidth 4. Easy to prototype

×