Speaker: Leigha Mitchell, Software Engineer, Hubba | Edward Robinson, Hubba

1. Time for a
Change Stream
By Leigha Mitchell & Edward Robinson
@LeighaNotLeia @earobinson

2. Change streams and
using them to version
your data

4. The hubba stack
● MEANR - Mongo, Express, Angular, Node, React
● Many Services (Payments, Products, Users, etc)
● Three Engineering teams (one Python, two JS)
● AWS, GCP, MongoDB Atlas, RabbitMQ, Redis, etc
● Mongoose, Mongo Native Driver

5. What is a change stream?
Change streams allow applications to access real-
time data changes without the complexity and risk
of tailing the oplog. Applications can use change
streams to subscribe to all data changes on a
collection and immediately react to them. --
https://docs.mongodb.com/manual/changeStrea
ms/

6. Why are we here?
Hubba
● 6 year old company
● Networking for Brands and Buyers
● Microservices
● Decided to launch ordering

7. The exact product you ordered must be
delivered to you
Ordering must haves
This Not this

8. What exactly is a version?
A particular form of something differing in certain
respects from an earlier form or other forms of the
same type of thing. --
https://www.google.com?q=define+version

9. 1. Options
➔ Not versioning
Previously wasn’t needed, continue to
not us this
➔ Make copy
How do we know what to copy?
➔ In-app versioning
Lean into Mongoose and version the
app
➔ Oplog versioning
Write new service to consume the
oplog

10. Not versioning
Would have locked
orderable products

11. Make a copy
Knowing what to copy is
hard

12. In-app Versioning
Works great, if you’re
building from scratch

13. model.save()
vs
db.getCollection('users').update({},
{$set: {name: ‘frodo’}})

14. Denormalization:
Denormalization allows you to avoid some application-
level joins, at the expense of having more complex and
expensive updates. Denormalizing one or more fields
makes sense if those fields are read much more often
than they are updated. --
https://www.mongodb.com/blog/post/6-rules-of-
thumb-for-mongodb-schema-design-part-3

15. Denormalization Example - Pre bearer of the ring
Users / Hobbits
{
_id : 1,
name : Frodo,
occupation: unemployed
}, {
_id : 2,
name : Sam,
occupation: unemployed
},
Messages
{
_id : 93,
from : 2,
to: 1,
fromOccupation: unemployed,
toOccupation: unemployed,
message: What do you call a hobbit party?
}, {
_id : 94,
from : 1,
to: 2,
fromOccupation: unemployed,
toOccupation: unemployed,
message: A little get together.
}

16. Denormalization Example - Post bearer of the ring
Users / Hobbits
{
_id : 1,
name : Frodo,
occupation: Bearer of the ring
}, {
_id : 2,
name : Sam,
occupation: Protector of Frodo
},
Messages
{
_id : 93,
from : 2,
to: 1,
fromOccupation: Protector of Frodo,
toOccupation: Bearer of the ring,
message: What do you call a hobbit party?
}, {
_id : 94,
from : 1,
to: 2,
fromOccupation: Bearer of the ring,
toOccupation: Protector of Frodo,
message: A little get together.
}
message.update({to: ObjectId: 1}, {$set: {toOccupation: Bearer of the ring}}, {multi: true})

17. Oplog Versioning

18. So what did we choose?

19. So what did we choose?
None of them!

20. Along comes:
MongoDB 3.6
Now with
Change Streams!

21. 2. Why Mongo 3.6
➔ Easy Versioning
Ability to do versioning without
significant architecture changes
➔ Raw Queries
Allowed us to use existing raw queries
without altering to support versioning
➔ Many Sources
No front-end data consumers did not
need to be altered

22. MongoDBWebsite
Product
Service
Orders
Service
History
Service
Hubba Services

23. Creating a Product
Website
Products
Service
MongoDB
History
Service
Create
Version 0
Create
Version 0
Create
Version 0
Created
Version 0
Created
Version 0

24. Ordering a Product
Website
Orders
Service
History
Service
MongoDB
Order
Version 0
Order
Version 0
Order
Version 0
Yes You
Can!
Ordered
Version 0

25. But what about the raw
queries??

26. But what about the raw
queries??
The version numbers are
Human Readable : SHA
42:c3f42aeb1c3d85b5a1594a5d0a727fcdf58a33ac

27. But what about the raw
queries??
The version numbers are
Human Readable : SHA
42:c3f42aeb1c3d85b5a1594a5d0a727fcdf58a33ac

28. But what about the raw
queries??
The version numbers are
Human Readable : SHA
42:c3f42aeb1c3d85b5a1594a5d0a727fcdf58a33ac

29. Autonomy of a Change Stream
{
_id : { <BSON Object> },
“operationType” : “<operation>“,
“fullDocument” : { <document> },
“ns” : {
“db” : “<database>“,
“coll” : “<collection”
},
“documentKey” : { “_id” : <ObjectId> },
“updateDescription” : {
“updatedFields” : { <document> },
“removedFields” : [ “<field>“, ... ]
}
}

30. Autonomy of a Change Stream
{
_id : { <BSON Object> },
“operationType” : “<operation>“,
“fullDocument” : { <document> },
“ns” : {
“db” : “<database>“,
“coll” : “<collection”
},
“documentKey” : { “_id” : <ObjectId> },
“updateDescription” : {
“updatedFields” : { <document> },
“removedFields” : [ “<field>“, ... ]
}
}
Insert
Delete
Update
Replace
Invalidate

31. Autonomy of a Change Stream
{
_id : { <BSON Object> },
“operationType” : “<operation>“,
“fullDocument” : { <document> },
“ns” : {
“db” : “<database>“,
“coll” : “<collection”
},
“documentKey” : { “_id” : <ObjectId> },
“updateDescription” : {
“updatedFields” : { <document> },
“removedFields” : [ “<field>“, ... ]
}
}

32. Autonomy of a Change Stream
{
_id : { <BSON Object> },
“operationType” : “<operation>“,
“fullDocument” : { <document> },
“ns” : {
“db” : “<database>“,
“coll” : “<collection>”
},
“documentKey” : { “_id” : <ObjectId> },
“updateDescription” : {
“updatedFields” : { <document> },
“removedFields” : [ “<field>“, ... ]
}
}

33. Autonomy of a Change Stream
{
_id : { <BSON Object> },
“operationType” : “<operation>“,
“fullDocument” : { <document> },
“ns” : {
“db” : “<database>“,
“coll” : “<collection”
},
“documentKey” : { “_id” : <ObjectId> },
“updateDescription” : {
“updatedFields” : { <document> },
“removedFields” : [ “<field>“, ... ]
}
}

34. Autonomy of a Change Stream
{
_id : { <BSON Object> },
“operationType” : “<operation>“,
“fullDocument” : { <document> },
“ns” : {
“db” : “<database>“,
“coll” : “<collection”
},
“documentKey” : { “_id” : <ObjectId> },
“updateDescription” : {
“updatedFields” : { <document> },
“removedFields” : [ “<field>“, ... ]
}
}

35. Change Streams in Action

36. Insert a message
Mongo Query
db.getCollection('messages').insert({from: 2, to: 1, fromOccupation: 'unemployed', toOccupation:
'unemployed', message: 'What do you call a habbit party?', version: 0})
Document
{
"_id" :
ObjectId("5b0b6c7adce137d2655f7efe"),
"from" : 2,
"to" : 1,
"fromOccupation" : "unemployed",
"toOccupation" : "unemployed",
"message" : "What do you call a habbit
party?",
"version" : 0
}
Change Stream
{
"_id":{
"_data":"glsLbHoAAAABRmRfaWQAZFsLbHrc
4TfSZV9+/gBaEASgyrHmfadIaLKbJfTNB8BgB
A=="
},
"operationType":"insert",
"fullDocument":{...},
"ns":{ "db":"hubba", "coll":"messages" },
"documentKey":{
"_id":"5b0b6c7adce137d2655f7efe"
}

37. Insert History Document

38. Update a message
Mongo Query
db.getCollection('messages').update({_id: ObjectId("5b0b6c7adce137d2655f7efe")}, {$set: {version: 1,
message : "What do you call a hobbit party?"}})
Document
{
"_id" :
ObjectId("5b0b6c7adce137d2655f7efe"),
"from" : 2,
"to" : 1,
"fromOccupation" : "unemployed",
"toOccupation" : "unemployed",
"message" : "What do you call a hobbit
party?",
"version" : 1
}
Change Stream
{
"_id": {…}, "operationType":"update",
"fullDocument":{...},
"Ns":{"db":"hubba","coll":"messages"}, "documentKey":
{... },
"updateDescription":{
"updatedFields":{
"message":"What do you call a hobbit party?",
"version":1
},
"removedFields":{}
}
}

39. Update History Document

40. Implementation
- Proof of concept
- Used full document to consume all changes
from Mongo and write them to our DB
- Successfully mirrored actual documents

41. Gotcha #1
Principle of eventual consistency
- Full document wouldn’t always represent the
document that existed in the DB
Lesson: If you’re using change streams for event sourcing,
you can source attributes out of the update description, but
not out of the full document

42. Solution:
Be more like a database
Change Streams are guaranteed to be delivered in
the order that the change happens. Source the
events as they happen, and apply update
description to the previous version

43. Gotcha #2
How do we Bootstrap versions into DB
- With lack of insert events for all records, how do we get
them into the DB?

44. Solution:
Use full document
If we can’t find a previous event, just use full
document

45. Gotcha #3
How do you ensure you record each write only once?

46. Solution:
Change streams guarantee that each
change will be delivered in order once
and only once.

47. Gotcha #4
Large documents can cause issues

48. Solution:
Don’t do it.
Seriously we don’t. 16Mb is a lot of data!

49. Gotcha #5
We fell off the Oplog
If your oplog can hold 100 documents, and I fill it up with
messages, you will not be able to resume the products
change stream

50. Solution:
We got a bigger oplog and update our
versioned collections
We build a script to run every hour and randomly
update one of the documents in every versioned
collection

51. Cons
➔ Histories has no context
Histories is unable to validate the data
it gets, its just a blind data store
➔ Spinning up a whole service
We could have solved this with in app
versioning, now we maintain an extra
service
➔ Issues upgrading
We had a few road bumps with
performance when we first released
mongo 3.6 + the new drivers

52. Benefits
➔ Histories is isolated
As long as our data is persisted we
have a history of it
➔ We get to keep our raw queries
Our denormalization strategies
continue to work
➔ Language support
We do not need to implement a history
support for every language we use, just
a version generator, if we want access.

53. Metrics

54. Conclusion:
1. Change streams are a great way to follow updates to
your documents
2. Using change streams for event sourcing would be
amazing
3. If you are versioning data in a legacy app, change
streams may be for you

55. Demo

56. Thank you!