MongoDB .local Toronto 2019: Using Change Streams to Keep Up with Your Data

#MDBlocal
Using Change Streams to Keep Up With Your Data
TORONTO
Kevin Albertson

21 22 23 24
25 26 27 28
28
8 days until Black Friday

mongodb-kitchen.com
The Shard Web Scale SauceThe Recipe Collection

BSON Pot
for your unstructured meals™

MongoDBKitchen.com
Mobile App
Third Party API
Catalog
db.orders
Read the source of truth

every day at 5pm
for each new order
if in inventory… ship
otherwise… manufacture, then ship

BSON Pot
• expensive to manufacture
• low margin
• can't anticipate demand
Solution: no inventory

BSON Pot
• long time to manufacture
• want quick delivery
Solution: manufacture
on demand

when customer orders BSON Pot
immediately manufacture BSON Pot

MongoDBKitchen.com
Mobile App
Third Party API
Catalog
db.orders
Listen to the source of truthRead the source of truth
pull —> push

21 22 23 24
25 26 27 28
28
3 days to implement
QA QA QA QA
QA

Tail the Oplog?
Not document or supported, but….

P
SS
Oplog
Oplog: special capped collection
of operations "operation log"
Secondaries tail the oplog
of the sync source

> use local
> db.oplog.findOne()
{
"ts": Timestamp(1573359145, 2),
"t": NumberLong(1),
"h": NumberLong(0),
"v": 2,
"op": "i",
"ns": "mdbkitchen.orders",
"ui": UUID("5de76b13-cb71-4fd1-b2da-3e9f44400162"),
"wall": ISODate("2019-11-10T04:12:25.747Z"),
"o": {
"_id": ObjectId("5dc78e29cd45383e19bbfed1"),
"sku": 7318166,
"name": "BSON Pot",
"user_id": ObjectId("5dc78e29cd45383e19bbfed0")
"status": "Not Shipped"
}
}

listener.py
db = client.local
cursor = db.oplog.rs.find(
{ "op": "i", "o.sku": sku },
cursor_type=CursorType.TAILABLE_AWAIT
)
while cursor.alive:
for doc in cursor:
manufacture_order (doc)

listener.py
last_saved_match = loads (open("saved.json", "r").read())
filter["ts"] = { "$gt": last_saved_match["ts"] }
open ("saved.json", "w").write (dumps (doc))
db = client.local
{ "op": "i", "o.sku": sku },
)
while cursor.alive:
for doc in cursor:

listener.py
while True:
time.sleep (1)
db = client.local
{ "op": "i", "o.sku": sku },
)
while cursor.alive:
for doc in cursor:

QA: "After failover it sends
wrong events… sometimes"

S PPPlistener.py S
A
B
C
cursor replication
A
B
C
A
B
Can't undo…
replication
Solution: only use majority committed events
Rollback

while True:
time.sleep (1)
db = client.local
{ "op": "i", "o.sku": sku },
)
while cursor.alive:
for doc in cursor:
listener.py

while True:
time.sleep (1)
db = client.local
cursor = coll.find(
{ "op": "i", "o.sku": sku },
)
while cursor.alive:
for doc in cursor:
listener.py
rc = ReadConcern(level="majority")
coll = db.get_collection("oplog.rs", read_concern=rc)

QA: "Restarting after long time
with no orders hurts perf"

Plistener.py
A
cursor
A
saved.json
match
non-match
B
Z
...
A
Restart
Solution: save all events

if doc["op"] == "i" and doc["o"]["sku"] == 7318166:
while True:
time.sleep (1)
db = client.local
cursor = coll.find(
{},
)
while cursor.alive:
for doc in cursor:
rc = ReadConcern(level="majority")
coll = db.get_collection("oplog.rs", read_concern=rc)
listener.py

QA: "Doesn't work when orders
is sharded"

Change Streams
Data event listeners

change_stream = db.orders.watch()
Added in MongoDB 3.6

Change Streams:
8 Characteristics

1. Present a Defined API
NodeJS let changestream = db.collection("orders").watch()
changestream.on("change", (event) => { console.log(event) })
Python changestream = db.orders.watch()
for event in changestream:
print event
C++ auto changestream = db["orders"].watch();
for (auto& event : changestream) {
cout << to_json(event) << endl;
}

2. Use Access Controls
db.createRole({
role: "analyzer",
privileges: [
{
resource: { db: "test", collection: "example" },
actions: [ "find", "changeStream" ]
},
],
roles: []
})

3. Use on any Data-Bearing Node
P
Oplog
db.coll.watch()
S
Oplog
db.coll.watch()

4. Total Ordering Across Shards
Shard Shard Shard
3 1 2
mongos
1 2 3

5. Documents Uniquely Identified
Sharded Cluster
{
operationType: 'update'
documentKey: {
_id: 123,
shardKey: 456
}
...
}
{
documentKey: {
_id: 123
}
...
}
Replica Set

6. Changes are Durable
S PPP S
Oplog Oplog
db.coll.watch()
PS
Oplog

P
7. Change Streams are Resumable
{
_id: <resumeToken>,
...
}
PP S
Oplog Oplog
db.coll.watch()
PS
Oplog
No duplicates
No missed events

8. Change Streams Use Aggregation
$match $project $addFields $replaceRoot $redact
coll.watch([{
$match: {
operationType: {$in: ['insert', 'update'] }
}
}]);

1. Defined API
2. Access Controls
3. Use on any Data Bearing Node
4. Total Ordering
5. Uniquely Identifies Documents
6. Durable
7. Resumable
8. Aggregation

change_stream =
… client.watch()
… db.watch()
… coll.watch()

Filter with pipelines
change_stream = coll.watch ([{$match: {…}}])

Iterate to get events (blocking)
doc = change_stream.next()
for doc in change_stream.next()
Set polling frequency
change_stream = coll.watch (maxAwaitTimeMS=500)

change_stream.on("change", callback)
Or listen in async (in NodeJS, Java async, C#)

Track with a "resume token"
token = change_stream.resume_token
And restart
change_stream = coll.watch (resume_after=token)

Other options
fullDocument
include entire document in update events
startAfter
like resumeAfter, except fails if stream invalidated

Returns 8 operation types
• Insert
• Update
• Replace
• Delete
• Rename
• Drop
• DropDatabase
• Invalidate

collection.watch()
Insert, Update, Replace, Delete on the collection.
Drop/Rename on the collection. Always followed by an Invalidate.
Invalidate on when collection dropped or renamed. Closes connection.

database.watch()
Insert, Update, Replace, Delete on all nested collections.
Drop/Rename on all nested collections. Not followed by an invalidate.
Invalidate on when database dropped. Closes connection.
DropDatabase when database dropped. Always followed by an invalidate.

client.watch()
Insert, Update, Replace, Delete on all databases and collections.
Drop/Rename on all collections.
Invalidate on all databases and collections. Does not close connection.
DropDatabase on all databases.

db = client.mdbkitchen
pipeline = [{ "$match": { "operationType": "insert", "fullDocument.sku": sku } }]
listener.py
def save_token (change_stream):
open ("token.json", "w").write (dumps (change_stream.resume_token))
def load_token ():
loads (open ("token.json", "r").read())
change_stream = db.orders.watch(
pipeline,
start_after=load_token()
)
while change_stream.alive:
for doc in change_stream:
save_token (change_stream)
save_token (change_stream)

• MongoDB supported API
• Has retry logic
• Events aren't rolled back
• No perf hit for sparse events
• Works if sharded too

P
listener.py
One change stream

P
listener.py
~1000 per server
listener.py listener.py

P
listener.py
Can scale with secondaries…
listener.py listener.py
S
listener.pylistener.py listener.py

P
but middleware is better
listener.py
Middleware
listener.py
listener.py
listener.py
listener.py

P
e.g. Apache Kafka
listener.py
Apache Kafka
listener.py
listener.py
listener.py
listener.py

Apache Kafka is
a stream processing service
that can read from and write to external systems
Any Source Any Sink
Apache Kafka

Apache Kafka is
a stream processing service
that can read from and write to external systems
Apache Kafka

MongoDB provides
confluent-hub install mongodb/kafka-connect-mongodb:0.2
a Kafka source and sink connector
verified by Confluent

BSON Pot
for your unstructured meals™
Want to track all BSON Pot
order state changes

orders = orders.with_options(read_concern="majority")
cursor = db.orders.find({"sku": 7318166 })
cache = list (cursor)
change_stream = orders.watch (
[{ "$match": { "sku": 7318166 }}])
for event in change_stream:
update_cache (cache, update)
update occurs, missed!

cursor = db.orders.find({"sku": 7318166 })
[{ "$match": { "sku": 7318166 }}])
But what if find selects old secondary?

P PP S2
Oplog Oplog
PS1
Oplog
db.coll.watch() db.coll.find()
First replicate up to S1
Session

cursor = db.orders.find(
{"sku": 7318166 }, session=session)
[{ "$match": { "sku": 7318166 }}], session=session)
with client.start_session(causal_consistency=True) as session:

MongoDB .local Toronto 2019: Using Change Streams to Keep Up with Your Data

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to MongoDB .local Toronto 2019: Using Change Streams to Keep Up with Your Data

Similar to MongoDB .local Toronto 2019: Using Change Streams to Keep Up with Your Data (20)

More from MongoDB

More from MongoDB (20)

Recently uploaded

Recently uploaded (20)

MongoDB .local Toronto 2019: Using Change Streams to Keep Up with Your Data