A short presentation with some discussion about mongoDB, why I use it and how to set up Replica Sets for a fully HA/failover enabled system.

1. High Availability & Replica Sets
with
Gareth Davies
@ShaolinTiger
www.shaolintiger.com

2. Who am I?
- Blogger (shaolintiger.com/darknet.org.uk)
- Community Starter (security-forums.com/shutterasia.com)
- Geek/Sys-admin
- WordPress/Web Publishing Scaling Expert
- Recent MongoDB user
- Currently working at Mindvalley

3. Why I <3 MongoDB
- It's FAST
- It's relatively easy to setup
- It's a LOT easier to scale than say..MySQL
- Does anyone know about scaling MySQL?

4. Scaling MySQL Is Like...

5. Basic Concepts – Master Slave

6. The Next Level – Replica Set

7. Master Slave vs Replica Set

8. Replica Sets – Things to Grok
- The primary AKA Master is auto-elected
- Drivers and mongos can detect the primary
- Replica sets provide you:
- Data Redundancy
- Automated Failover AKA High-availaiblity
- Distributed Read Load/Read Scaling
- Disaster Recovery

9. Replica Sets - Caveats
- You must have an odd number of mongos at all
times to avoid vote locking & primary becoming
read only
- You can have a maximum of 12 nodes in a set,
if you need more read capacity – look to
sharding
- If you are reading from the secondary servers
you need to understand Eventual Consistency

10. Getting Started
- I <3 Linode!
- Easy scaling/Nodebalancers/Cloning/Fast Roll-
up/Fastest IO in the industry/Ubuntu 12.04LTS
etc
– Examples are done on Linode

11. Add 3 new Nodes
- Chose your location, put 2 in your primary DC and 1 in
a different geographical DC
- In my case this would be 2 servers in Atlanta, GA and
1 in Dallas TX
– This gives you a replica set that works if a whole
datacenter goes down

12. Select The OS
- Ubuntu 12.04LTS 64-bit – this gives you
package support for the next 5 years
- Also gives you the ability to grow your
MongoDB instance above 2GB safely

13. Do your basic shizzles
- Set the hostname/local IP address etc
- Disable Swap
- Change SSH port
- Remove password based login
- Block root SSH acess
- aptitude update; aptitude safe-upgrade
- Install base packages (munin/iotop/sysstat etc)
- Configure unattended security updates

14. Install MongoDB
- I don't recommend installing from the regular
repo as it will be out of date after some time
- Install direct from the 10gen repo
sudo apt-key adv --keyserver keyserver.ubuntu.com --recv 7F0CEB10
sudo nano /etc/apt/sources.list – add this:
deb http://downloads-distro.mongodb.org/repo/ubuntu-upstart dist 10gen
aptitude update; aptitude install mongodb-10gen
- That's it – it's installed!

15. Clone that bad boy!
- Bear in mind you only have to do all of that stuff
once! When it's done – just clone it over to your
two new nodes.
- Remember to delete the config & disk images from
your target first & power down the initial machine.
* Do note when copying to the remote DC it will
take quite a long time

16. Get the Replica Set Started
Do:
sudo nano /etc/mongodb.conf
Find the line like so:
# in replica set configuration, specify the name of the replica set
# replSet = setname
Change it to:
# in replica set configuration, specify the name of the replica set
replSet = yoursetname
Do this on all your Mongos and then restart them
sudo service mongodb restart

17. Configure the Replica Set
- After restarting if you check the logs you'll see
something like this:
- This basically means the Replica set is running,
but it's not yet aware of the other nodes

18. Add the member nodes
- You'll need to run 'mongo' to get the mongo
shell then:
rs.initiate({_id: 'yoursetname', members: [
{_id: 0, host: '192.168.1.1:27017'},
{_id: 1, host: '192.168.1.2:27017'},
{_id: 2, host: '192.168.1.3:27017'}]
})
- This will spin up the set

19. Check that it worked
- I suggest running tail -f on the logs on one of
the other nodes, you'll see a bunch of
messages about replSet & rsStart (hopefully)
- If you see all that in /var/log/mongodb/mongodb.log
– you're good!

20. That's It!
- Yah I know, too easy right?
- That's how hard it is to set up a fully scalable,
high availability database cluster with
MongoDB

21. Things to Consider
- ALWAYS monitor, make decisions made on
statistics and numbers not on assumptions
- I like (and very actively use) munin
- munin works well with MongoDB and a myriad
of other software

22. Further Learnings
- Think about security (Bind
address/IPTables/Authentication/Cluster Keys)
- If you have a write heavy application and you
need to scale writes – look to sharding
- Sharding and replica sets work well together
(but each shard needs a replica set)
- Try and give your MongoDB instance enough
RAM to keep the hot index in memory

23. THE END!
Questions?
This presentation will be available at http://slideshare.net/shaolintiger