Optimizing MongoDB: Lessons Learned at Localytics

Optimizing MongoDB: Lessons Learned at Localytics Benjamin Darfler MongoBoston - September 2011

Introduction ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

MongoDB at Localytics ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Disclaimer ,[object Object],[object Object],[object Object]

Quick Poll ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Optimizing Our Data Documents and Indexes

Shorten Names ,[object Object],[object Object],[object Object],[object Object],[object Object]

Use BinData for uuids/hashes ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Override _id ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Pre-aggregate ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Prefix Indexes ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Sparse Indexes ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Upgrade to {v:1} indexes ,[object Object],[object Object],[object Object],[object Object]

Optimizing Our Queries Reading and Writing

You are using an index right? ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Only as much as you need ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Only what you need ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Covering Indexes ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Prefetch ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Optimizing Our Disk Fragmentation

Inserts doc1 doc2 doc3 doc4 doc5

Deletes doc1 doc2 doc3 doc4 doc5 doc1 doc2 doc3 doc4 doc5

Updates doc1 doc2 doc3 doc4 doc5 doc1 doc2 doc3 doc4 doc5 doc3 Updates can be in place if the document doesn't grow

Reclaiming Freespace doc1 doc2 doc6 doc4 doc5 doc1 doc2 doc3 doc4 doc5

Memory Mapped Files doc1 doc2 doc6 doc4 doc5 } } page page Data is mapped into memory a full page at a time

Fragmentation ,[object Object],[object Object],[object Object],[object Object]

Fragmentation Mitigation ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Fragmentation Fixes ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Optimizing Our Keys Index and Shard

B-Tree Indexes - hash/uuid key Hashes/UUIDs randomly distribute across the whole b-tree

B-Tree Indexes - temporal key Keys with a temporal prefix (i.e. ObjectId) are right aligned

Migrations - hash/uuid shard key Chunk 1 k: 1 to 5 Chunk 2 k: 6 to 9 Shard 1 Shard 2 Chunk 1 k: 1 to 5 {k: 4, …} {k: 8, …} {k: 3, …} {k: 7, …} {k: 5, …} {k: 6, …} {k: 4, …} {k: 3, …} {k: 5, …}

Hash/uuid shard key ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Migrations - temporal shard key Chunk 1 k: 1 to 5 Chunk 2 k: 6 to 9 Shard 1 Shard 2 Chunk 1 k: 1 to 5 {k: 3, …} {k: 4, …} {k: 5, …} {k: 6, …} {k: 7, …} {k: 8, …} {k: 3, …} {k: 4, …} {k: 5, …}

Temporal shard key ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Optimizing Our Deployment Hardware and Configuration

Elastic Compute Cloud ,[object Object],[object Object],[object Object],[object Object],[object Object]

Elastic Block Storage ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Pathological Test ,[object Object],[object Object],[object Object]

One Mongod Index out of RAM Index in RAM ,[object Object],[object Object]

Many Mongods Index out of RAM Index in RAM ,[object Object],[object Object],[object Object]

Sharding within a node ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Reminder ,[object Object],[object Object],[object Object]

Questions? @bdarfler http://bdarfler.com

Optimizing MongoDB: Lessons Learned at Localytics

Empfohlen

Empfohlen

Weitere ähnliche Inhalte

Was ist angesagt?

Was ist angesagt? (20)

Ähnlich wie Optimizing MongoDB: Lessons Learned at Localytics

Ähnlich wie Optimizing MongoDB: Lessons Learned at Localytics (20)

Kürzlich hochgeladen

Kürzlich hochgeladen (20)

Optimizing MongoDB: Lessons Learned at Localytics