1. High Performance,
Scalable MongoDB
in a Bare Metal Cloud
Harold Hannon, Sr. Software Architect

2. 100kservers
24kcustomers
23million domains

3. 13 data centers
16 network POPs
20Gb fiber interconnects
Global Footprint

4. On the agenda today…..
• Big Data considerations
• Some deployment options
• Performance Testing with JS
Benchmarking Harness
• Review some internal product research
performed
• Discuss the impact of those findings on
our product development

5. “Build me a Big Data
Solution”

6. Product Use Case
• MongoDB deployed for customers on purchase
• Complex configurations including sharding and
replication
• Configurable via Portal interface
• Performance tuned to 3 „t-shirt size‟
deployments

7. Big Data Requirements
• High Performance
• Reliable, Predictable Performance
• Rapidly Scalable
• Easy to Deploy

8. Requirements Reviewed
Cloud Provider Bare Metal Instance
High Performance
Reliable, Predictable
Performance
Rapidly Scalable
X
Easy to Deploy
X
I’ve got nothing……

9. The “Marc-O-Meter”
I’M NOT
HAPPY 

10. Marc… Angry

11. Thinking about Big Data

12. The 3 V’s

13. Physical Deployment

15. Public Cloud

16. Public Cloud
• Speed of deployment
• Great for bursting use case
• Imaging and cloning make POC/Dev work easy
• Shared I/O
• Great for POC/DEV
• Excellent for App level applications
• Not consistent enough for disk intensive applications
• Must have application developed for “cloud”

17. Physical Servers

18. Bare Metal
• Build to your specs
• Robust, quickly scaled environment
• Management of all aspects of environment
• Image Based
• No Hypervisor
• Single Tenant
• Great for Big Data Solutions

19. The Proof is in the Pudding

20. Beware The “Best Case Test Case”
1 8 5 8 1 7 . 6
1 9 0 5 2 5 . 4
1 8 7 8 8 2 . 2
1 9 1 1 0 1 . 8
1 8 4 4 0 8 . 8
1 8 8 1 3 5 . 4
1 8 7 0 8 0 . 6
1 8 6 3 4 3 . 4
1 9 1 8 9 9 . 6
1 8 7 7 3 6 . 6
1 8 8 9 7 8 . 8
1 8 7 4 4 0
1 8 6 9 5 0 . 4
1 8 7 6 2 3
1 8 7 7 8 3 . 8
1 8 7 7 7 5 . 8
1 9 2 8 0 6 . 8
1 8 6 6 4 3 . 2


21. Do It Yourself
• Data Set Sizing
• Document/Object Sizes
• Platform
• Controlled client or AFAIC
• Concurrency
• Local or Remote Client
• Read/Write Tests

22. JS Benchmarking Harness
• Data Set Sizing
• Document/Object Sizes
• Platform
• Controlled client or AFAIC
• Concurrency
• Local or Remote Client
• Read/Write Tests

23. db.foo.drop();
db.foo.insert( { _id : 1 } )
ops = [{op: "findOne", ns: "test.foo", query: {_id: 1}},
{op: "update", ns: "test.foo", query: {_id: 1}, update: {$inc: {x: 1}}}]
for ( var x = 1; x <= 128; x *= 2) {
res = benchRun( {
parallel : x ,
seconds : 5 ,
ops : ops
} );
print( "threads: " + x + "t queries/sec: " + res.query );
}
Quick Example

24. host
The hostname of the machine mongod is running on (defaults to localhost).
username
The username to use when authenticating to mongod (only use if running with auth).
password
The password to use when authenticating to mongod (only use if running with auth).
db
The database to authenticate to (only necessary if running with auth).
ops
A list of objects describing the operations to run (documented below).
parallel
The number of threads to run (defaults to single thread).
seconds
The amount of time to run the tests for (defaults to one second).
Options

25. ns
The namespace of the collection you are running the operation on, should be of the form
"db.collection".
op
The type of operation can be "findOne", "insert", "update", "remove", "createIndex",
"dropIndex" or "command".
query
The query object to use when querying or updating documents.
update
The update object (same as 2nd argument of update() function).
doc
The document to insert into the database (only for insert and remove).
safe
boolean specifying whether to use safe writes (only for update and insert).
Options

26. { "#RAND_INT" : [ min , max , <multiplier> ] }
[ 0 , 10 , 4 ] would produce random numbers between 0 and 10 and then multiply by 4.
{ "#RAND_STRING" : [ length ] }
[ 3 ] would produce a string of 3 random characters.
var complexDoc3 = { info: "#RAND_STRING": [30] } }
var complexDoc3 = { info: { inner_field: { "#RAND_STRING": [30] } } }
Dynamic Values

27. Lots of them here:
https://github.com/mongodb/mongo/tree/master/jstests
Example Scripts

28. Read Only Test
• Random document size < 4k (mostly 1k)
• 6GB Working Data Set Size
• Random read only
• 10 second per query set execution
• Exponentially increasing concurrent clients from 1-128
• 48 Hour Test Run
• RAID10 4 SSD drives
• Local Client
• “Pre-warmed cache”

29. The Results
Concurrent Clients Avg Read OPS/Sec
1 38288.527
2 72103.35796
4 127451.8867
8 180798.4396
16 191817.3361
32 186429.4517
64 187011.7824
128 188187.0704

30. Some Tougher Tests
• Small MongoDB Bare Metal Cloud vs Public
Cloud Instance
• Medium MongoDB Bare Metal Cloud vs
Public Cloud Instance
• SSD and 15K SAS
• Large MongoDB Bare Metal Cloud vs Public
Cloud Instance
• SSD and 15K SAS

31. Pre-configurations
• Set SSD Read Ahead Defaults to 16 Blocks – SSD drives have
excellent seek times allowing for shrinking the Read Ahead to
16 blocks. Spinning disks might require slight buffering so these
have been set to 32 blocks.
• noatime – Adding the noatime option eliminates the need for
the system to make writes to the file system for files which are
simply being read — or in other words: Faster file access and
less disk wear.

32. • Turn NUMA Off in BIOS – Linux, NUMA and MongoDB
tend not to work well together. If you are running
MongoDB on NUMA hardware, we recommend turning it
off (running with an interleave memory policy). If you
don’t, problems will manifest in strange ways like
massive slow downs for periods of time or high system
CPU time.
• Set ulimit – We have set the ulimit to 64000 for open files
and 32000 for user processes to prevent failures due to a
loss of available file handles or user processes.

33. Use ext4 – We have selected ext4 over ext3. We found ext3
to be very slow in allocating files (or removing them).
Additionally, access within large files is poor with ext3.

34. Private Network
JMETER
SERVER
JMETER
SERVER
JMETER
SERVER
JMETER
SERVER
RMI
Jmeter Master Client
RDP
Tester’s Local Machine
Test Environment

35. var ops = [];
while (low_rand < high_id) {
if(readTest){
ops.push({
op : "findOne",
ns : "test.foo",
query : {
incrementing_id : {
"#RAND_INT" : [ low_rand, low_rand + RAND_STEP ]
}
}
});
}
if(updateTest){
ops.push({ op: "update", ns: "test.foo",
query: { incrementing_id: { "#RAND_INT" : [0,high_id]}},
update: { $inc: { counter: 1 }},
safe: true });
}
low_rand += RAND_STEP;
}
function printLine(tokens, columns, width) {
line = "";
column_width = width / columns;
for (var i=0;i<tokens.length;i++) {
line += tokens[i];
// token_width = tokens[token].toString().length;
// pad = column_width - token_width;
// while (pad--) {
if(i != tokens.length-1)
line += " , ";
// }
}

36. Small Test
Small MongoDB Server
Single 4-core Intel 1270 CPU
64-bit CentOS
8GB RAM
2 x 500GB SATAII – RAID1
1Gb Network
Virtual Provider Instance
4 Virtual Compute Units
64-bit CentOS
7.5GB RAM
2 x 500GB Network Storage – RAID1
1Gb Network
Tests Performed
Small Data Set (8GB of .5mb
documents)
200 iterations of 6:1 query-to-update
operations
Concurrent client connections
exponentially increased from 1 to 32
Test duration spanned 48 hours

37. Small Test
Small Bare Metal Cloud Instance
• 64-bit CentOS
• 8GB RAM
• 2 x 500GB SATAII – RAID1
• 1Gb Network
Public Cloud Instance
• 4 Virtual Compute Units
• 64-bit CentOS
• 7.5GB RAM
• 2 x 500GB Network Storage – RAID1
• 1Gb Network

38. Small Public Cloud
122
193 201 271
480
835
0
200
400
600
800
1000
1200
1400
1 2 4 8 16 32
Ops/Second
Concurrent Clients

39. Small Bare Metal
237
337
413 524
597
1112
0
200
400
600
800
1000
1200
1400
1600
1 2 4 8 16 32
Ops/Second
Concurrent Clients

40. Medium Test
Medium MongoDB Server
Dual 6-core Intel 5670 CPUs
64-bit CentOS
36GB RAM
2 x 64GB SSD – RAID1 (Journal Mount)
4 x 300GB 15K SAS – RAID10 (Data Mount)
1Gb Network – Bonded
Virtual Provider Instance
26 Virtual Compute Units
64-bit CentOS
30GB RAM
2 x 64GB Network Storage – RAID1 (Journal
Mount)
4 x 300GB Network Storage – RAID10 (Data
Mount)
1Gb Network
Tests Performed
Small Data Set (32GB of .5mb
documents)
200 iterations of 6:1 query-to-update
operations
Concurrent client connections
exponentially increased from 1 to 128
Test duration spanned 48 hours

41. Medium Test
Bare Metal Cloud Instance
• Dual 6-core Intel 5670 CPUs
• 64-bit CentOS
• 36GB RAM
• 2 x 64GB SSD – RAID1 (Journal Mount)
• 4 x 300GB 15K SAS – RAID10 (Data Mount)
• 1Gb Network – Bonded
Public Cloud Instance
• 26 Virtual Compute Units
• 64-bit CentOS
• 30GB RAM
• 2 x 64GB Network Storage – RAID1 (Journal Mount)
• 4 x 300GB Network Storage – RAID10 (Data Mount)
• 1Gb Network

42. Medium Test
Bare Metal Cloud Instance
• Dual 6-core Intel 5670 CPUs
• 64-bit CentOS
• 36GB RAM
• 2 x 64GB SSD – RAID1 (Journal Mount)
• 4 x 400GB SSD– RAID10 (Data Mount)
• 1Gb Network – Bonded
Public Cloud Instance
• 26 Virtual Compute Units
• 64-bit CentOS
• 30GB RAM
• 2 x 64GB Network Storage – RAID1 (Journal Mount)
• 4 x 400GB Network Storage – RAID10 (Data Mount)
• 1Gb Network

43. Medium Test
Tests Performed
• Data Set (32GB of .5mb documents)
• 200 iterations of 6:1 query-to-update operations
• Concurrent client connections exponentially
increased from 1 to 128
• Test duration spanned 48 hours

44. Medium Public Cloud
219 326 477 716
1298
1554
1483
1594
0
500
1000
1500
2000
2500
3000
3500
4000
4500
5000
1 2 4 8 16 32 64 128
Ops/Second
Concurrent Clients

45. Medium Bare Metal
15k SAS
542 818 1042 1260
1643
3392
4120
5443
0
1000
2000
3000
4000
5000
6000
7000
8000
1 2 4 8 16 32 64 128
Ops/Second
Concurrent Clients

46. Medium Bare Metal
SSD
1389
2115
2637
2995
3047 3161
3742 3846
0
500
1000
1500
2000
2500
3000
3500
4000
4500
1 2 4 8 16 32 64 128
Ops/Second
Concurrent Clients

47. Large Test
Large MongoDB Server
Dual 8-core Intel E5-2620 CPUs
64-bit CentOS
128GB RAM
2 x 64GB SSD – RAID1 (Journal Mount)
6 x 600GB 15K SAS – RAID10 (Data Mount)
1Gb Network – Bonded
Virtual Provider Instance
26 Virtual Compute Units
64-bit CentOS
64GB RAM (Maximum available on this
provider)
2 x 64GB Network Storage – RAID1 (Journal
Mount)
6 x 600GB Network Storage – RAID10 (Data
Mount)
1Gb Network
Tests Performed
Small Data Set (64GB of .5mb
documents)
200 iterations of 6:1 query-to-update
operations
Concurrent client connections
exponentially increased from 1 to 128
Test duration spanned 48 hours

48. Large Test
Bare Metal Cloud Instance
• Dual 8-core Intel E5-2620 CPUs
• 64-bit CentOS
• 128GB RAM
• 2 x 64GB SSD – RAID1 (Journal Mount)
• 6 x 600GB 15K SAS – RAID10 (Data Mount)
• 1Gb Network – Bonded
Public Cloud Instance
• 26 Virtual Compute Units
• 64-bit CentOS
• 64GB RAM (Maximum available on this provider)
• 2 x 64GB Network Storage – RAID1 (Journal Mount)
• 6 x 600GB Network Storage – RAID10 (Data Mount)
• 1Gb Network

49. Large Test
Bare Metal Cloud Instance
• Dual 8-core Intel E5-2620 CPUs
• 64-bit CentOS
• 128GB RAM
• 2 x 64GB SSD – RAID1 (Journal Mount)
• 6 x 400GB SSD – RAID10 (Data Mount)
• 1Gb Network – Bonded
Public Cloud Instance
• 26 Virtual Compute Units
• 64-bit CentOS
• 64GB RAM (Maximum available on this provider)
• 2 x 64GB Network Storage – RAID1 (Journal Mount)
• 6 x 400GB Network Storage – RAID10 (Data Mount)
• 1Gb Network

50. Large Test
Tests Performed
• Data Set (64GB of .5mb documents)
• 200 iterations of 6:1 query-to-update operations
• Concurrent client connections exponentially
increased from 1 to 128
• Test duration spanned 48 hours

51. Large Public Cloud
105
409
943
636
1252
1733
1902
2044
0
1000
2000
3000
4000
5000
6000
1 2 4 8 16 32 64 128
Ops/Second
Concurrent Clients

52. Large Bare Metal
15k SAS
412
686 946 1123
1373
2353
5097
5572
0
1000
2000
3000
4000
5000
6000
7000
1 2 4 8 16 32 64 128
Ops/Second
Concurrent Clients

53. Large Bare Metal
SSD
1898
2919
3672
4351 3961
3629 3737 3864
0
1000
2000
3000
4000
5000
6000
1 2 4 8 16 32 64 128
Ops/Second
Concurrent Clients

54. Superior Performance
Deployment Size Bare Metal Drive
Type
Bare Metal Average
Performance Advantage
over Virtual
Small SATA II 70%
Medium 15k SAS 133%
Medium SSD 297%
Large 15k SAS 111%
Large SSD 446%

55. Consistent Performance
Virtual Instance Bare Metal Instance
Small 6-36% 1-9%
Medium 8-43% 1-8%
Large 8-93% 1-9%
RSD (Relative Standard Deviation) by Platform

56. Requirements Reviewed
Cloud Provider Bare Metal Instance
High Performance
X
Reliable, Predictable
Performance X
Rapidly Scalable
X
Easy to Deploy
X
Not Quite There Yet……

57. The “Marc-O-Meter”
NOT SURE IF
WANT

58. The Dream

59. The Reality
Virtual Instance
Striped Network
Attached Virtual
Volumes

60. Cluster
Deployment Complexity
Virtual Instance
Striped Network
Attached Virtual
Volumes
Virtual Instance
Striped Network
Attached Virtual
Volumes
Virtual Instance
Striped Network
Attached Virtual
Volumes

61. Deployment Serenity:
The Solution Designer

62. MongoDB Solutions
• Preconfigured
• Performance Tuned
• Bare Metal Single Tenant
• Complex Environment Configurations

63. Requirements Reviewed
Cloud Provider Bare Metal Instance
High Performance
X
Reliable, Predictable
Performance X
Rapidly Scalable
X X
Easy to Deploy
X X

64. The “Marc-O-Meter”
B+ FOR
EFFORT

65. Customer Feedback
“We have over two terabytes of raw event
data coming in every day ... Struq has
been able to process over 95 percent of
requests in fewer than 30 milliseconds”
- Aaron McKee
CTO, Struq

66. The “Marc-O-Meter”
WIN!!

67. Summary
• Bare Metal Cloud can be leveraged to
simplify deployments
• Bare Metal has a significant
performance superiority/consistency
over Public Cloud
• Public Cloud is best suited for Dev/POC
or when running data sets in memory
only

68. More information:
www.softlayer.com
blog@ http://sftlyr.com/bdperf