Slides of my talk on MongoDB Database Replication At Sydney MongoDB User Group.

1. Database Replication
valikhani@sirangweb.com
@mehdivk
Mehdi Valikhani

2. Replication
Process of keeping IDENTICAL COPY
of your data on different servers.
Replication offers
Minimum Downtime
in case of hardware failure, software
failure or maintenance

3. Replication is highly
recommended for all
databases in
production from the
very first day.

4. Replica Set
MongoDB’s Terminology for Replication
A replica set is a group of servers with one primary,
the server taking client requests, and multiple
secondaries, servers that keep copies of the primary’s
data.

5. Give A Go To
MongoDB’s Replica Set
In
60 Seconds!
#ToBeTriedOnlyAtHome

6. Step 1: Setup Replica Set
$: mongo --nodb
$: myReplica = new ReplSetTest({nodes: 3});
$: myReplica.startSet();
$: myReplica.initiate();

7. Congratulations!
3 member of replica set running on ports 31000, 31001 and 31002

8. Step 2: Connect To Primary
$: mongo --port 31000
testReplSet:PRIMARY> use surryhills_db
testReplSet:PRIMARY> db.offices.insert({name: ‘MongoDB’});
testReplSet:PRIMARY> show dbs
testReplSet:PRIMARY> show collections
testReplSet:PRIMARY> db.offices.find().pretty()

9. Pro Tip :]
Auto Format Query Response
Add
DBQuery.prototype._prettyShell = true;
to your ~/.mongorc.js

10. Step 3: Connect To A Secondary
$: mongo --port 31001
testReplSet:SECONDARY> db.slaveOk()
testReplSet:SECONDARY> show dbs
testReplSet:SECONDARY> use surryhills_db
testReplSet:SECONDARY> show collections
testReplSet:SECONDARY> db.offices.find()

11. Step 4: Shutdown Replica Set
$: myReplica.stopSet()

12. Setup Your Replica Set
Part 1: Run replica members
$: mongod --port 31000 --fork --logpath ./path/to/log --dbpath
/path/to/srv1 --replSet myReplica
$: mongod --port 31001 --fork --logpath ./path/to/log --dbpath
/path/to/srv2 --replSet myReplica
$: mongod --port 31002 --fork --logpath ./path/to/log --dbpath
/path/to/srv3 --replSet myReplica

13. Setup Your Replica Set
Part 2: Configure the set
$: mongo --port 31000
$: rs.initiate();
$: rs.add(‘my-host:31001’);
$: rs.add(‘my-host:31002’);

14. Setup Your Replica Set
Part 2: Example Configuration
$: rs.config();
{
"_id" : "myReplica",
"version": 3,
"members" : [
{
"_id" : 0,
"host" : "my-host:31000"
},
{
"_id" : 1,
"host" : "my-host:31001"
},
{
"_id" : 2,
"host" : "my-host:31002"
}
]
}

15. Auto Connect To Current Primary
$: mongo --host myReplica/my-host:31000,my-host:31001,my-host:31002

16. Replica Set
Good To Knows
1. Always start your instance with `replSet`
2. You can’t initiate a replica set with more than
one member with existing data on it.
3. If one of set members has data, you must run
rs.initiate() against that member
4. `system.replset` collection on `local` database
of all members contains set configuration.
5. Replica sets can have up to 50 members, 7 voting
6. By default, MongoDB reads from primary

17. Replica Set Reconfiguration
Add new member
$: mongo --port 31000 # current primary
$: rs.add(‘my-host:31003’) # accepts object as well.

18. Replica Set Reconfiguration
Remove existing member
$: mongo --port 31000 # current primary
$: rs.remove(‘my-host:31003’)

19. Replica Set Reconfiguration
Update existing member
$: mongo --port 31000 # current primary
$: config = rs.config()
$: config.members[2].priority = 0;
$: rs.reconfig(config);

20. How Replication Works
MongoDB keeps a log of every write operations that
primary does in database
Logs are saved in “oplog.rs” collection of “local” db
Secondaries apply changes by reading from oplog.rs and
applieing to their data
Secondaries maintain their own “oplog.rs” as well.

21. “oplog.rs” Facts
“oplog.rs” is a capped collection which means it can
save limited amount of operation logs.
Bulk operations fill “oplog” quickly. A bulk removal of
1K documents adds 1K documents to “oplog”
It’s important to have a “oplog.rs” to keep logs for
atleast last 24 hrs.

22. “Stale” Secondary :(
Secondary misses a copy of last operation it applied in
syncing source.
Secondary queries other members to find another
source to continue syncing.
Secondary missed some operations permanently.
Admin must resyncs from scratch.

23. Heartbeat
Members PING each other every 2 seconds.
Members should
reply within 10 seconds
otherwise they’re counted
as unreachable.

24. Change in replica set configuration
Current primary steps down
Replica Set Election
When It Happens?
Current primary is unreachable

25. Majority
The Single Most
Important Fact About
Replica Set
More than half of All Voting Members in the set must be reachable.

26. You need a majority of voting
members to elect a primary.
A primary can only stay
primary so long as it can reach
a majority of voting members.
Majority

27. Replica Set with two members.
Three Common
Replica Set
Design Mistakes :(
Even number of members in two data center.
A set with more than one arbiter

28. Keep a majority of members in preferred
data center and rest in the other one.
Two Common
Replica Set
Design Patterns :)
Keep even number of members in two data
center and a tie breaker in third one.

29. Meet Arbiter
Budget Friendly Member Of Replica Set
Does not hold data
It’s a lightweight process
Can vote on elections

30. Start An Arbiter Process
$: mongod --port 31005 --replSet myReplica --
nojournal --smallfiles
Arbiter does not save data, no need for journaling
Don’t waste disk space and use --smallfiles

31. Add An Arbiter To Replica Set
$: rs.addArb(‘arbiter-host-name:port’);

32. Arbiter
Good To Know
You can’t change an arbiter to non-arbiter member
If possible, use normal members instead of arbiters
At most you need one arbiter in your set
Do not add an arbiter to a set with odd members

33. Replica Set
Useful Commands
$: rs.config()
$: rs.reconfig()
$: rs.isMaster()
$: rs.status()
$: rs.stepDown()

34. Rollback :(
Primary does a write and
goes down before at least
one secondary has a chance
to replicate the write.

35. Rollback
Main Reasons
Network Partitions
Different write rates between
primary and secondary

36. Rollback Process
1. Former primary reverts write operations
2. Saves affected documents into /rollback directory
3. Admin browses documents using mongorestore
4. Admin applies changes to the current primary

37. Durability
Write Concerns And Replica Set
1. Unacknowledged
2. Acknowledged
3. Journaled
4. Replica Acknowledged
5. Custom Writer Concern

38. Improve Durability
db.runCommand({getLastError:1}) options
Returns the error status of the preceding write
operation on the current connection.
j: boolean
w: number | majority | custom role
wtimeout: number (milliseconds)
fsync: boolean

39. “wtimeout”
GOOD TO KNOW
MongoDB does not “rollback” or ”
undo modifications” made before
the “wtimeout” interval expired.

40. Custom Write Concerns
Step 1: Tags
$: config = rs.config()
$: config.members[0].tags = {dc: ‘dc-1’};
$: config.members[3].tags = {dc: ‘dc-2’};
$: rs.reconfig(config);

41. Custom Write Concerns
Step 2: Custom Last Error Mode
$: config = rs.config()
$: config.settings.lastErrorModes = [{
“each-dc”: {dc: 2}
}];
$: rs.reconfig(config);

42. Custom Write Concerns
Step 3: Use it!
$: db.runCommand({getLastError:1, w: ‘each-dc’})

43. Replica Set &
Mixed Storage Engines
You can have both
WiredTiger & MMAPv1
● Due to migration
● Due to different use cases

44. Keen To Go Further?
Register for M102: MONGODB FOR DBAS course!
Starts tomorrow!
It’s awesome!
https://university.mongodb.com/courses/M102/about

45. Keen To Go Further?
docs.mongodb.org/manual/replication
MongoDB
The Definite Guide

46. Stay In Touch
● Follow me on Twitter
@mehdivk
● Add me to your Linkedin network https://au.
linkedin.com/in/valikhani
● Subscribe to my blog
blog.mehdivk.net
● Send me an email
valikhani@sirangweb.com

47. Thank
You!