SlideShare ist ein Scribd-Unternehmen logo
1 von 45
Downloaden Sie, um offline zu lesen
#MongoMelbourne




Indexing and Query
Optimisation
Stephen Steneker
Support Engineer, 10gen Australia
Agenda
• What are indexes?
• Why do I need them?
• Working with indexes in MongoDB
• Optimise your queries
• Avoiding common mistakes
What are indexes?
What are indexes?
Imagine you’re looking for a recipe in a cookbook
ordered by recipe name. Looking up a recipe by
name is quick and easy.
What are indexes?
• How would you find a recipe using chicken?
• How about a 250-350 calorie recipe using
 chicken?
KRISTINE TO INSERT IMAGE OF COOKBOOK




Consult the index!
1   2   3    4    5   6   7




        Linked List
Finding 7 in Linked List
4


    2                       6


1          3        5           7


        Finding 7 in Tree
Indexes in MongoDB are B-trees
Queries, inserts and deletes:
       O(log(n)) time
Indexes are the single
biggest tuneable
performance factor in
MongoDB
Absent or suboptimal
indexes are the most
common avoidable
MongoDB performance
problem.
Why do I need indexes?
A brief story
Working with Indexes in
MongoDB
How do I create indexes?
// Create an index if one does not exist
db.recipes.createIndex({ main_ingredient: 1 })



// The client remembers the index and raises no errors
db.recipes.ensureIndex({ main_ingredient: 1 })




* 1 means ascending, -1 descending
What can be indexed?
// Multiple fields (compound key indexes)
db.recipes.ensureIndex({
   main_ingredient: 1,
   calories: -1
})

// Arrays of values (multikey indexes)
{
   name: 'Chicken Noodle Soup’,
   ingredients : ['chicken', 'noodles']
}

db.recipes.ensureIndex({ ingredients: 1 })
What can be indexed?
// Subdocuments
{
   name : 'Pavlova',
   contributor: {
     name: 'Ima Aussie',
     id: 'ima123'
   }
}

db.recipes.ensureIndex({ 'contributor.id': 1 })

db.recipes.ensureIndex({ 'contributor': 1 })
How do I manage indexes?
// List a collection's indexes
db.recipes.getIndexes()
db.recipes.getIndexKeys()


// Drop a specific index
db.recipes.dropIndex({ ingredients: 1 })


// Drop all indexes and recreate them
db.recipes.reIndex()


// Default (unique) index on _id
Background Index Builds
// Index creation is a blocking operation that can take a long time
// Background creation yields to other operations
db.recipes.ensureIndex(
    { ingredients: 1 },
    { background: true }
)
Options
• Uniqueness constraints (unique, dropDups)
• Sparse Indexes
• Geospatial (2d) Indexes
• TTL Collections (expireAfterSeconds)
Uniqueness Constraints
// Only one recipe can have a given value for name
db.recipes.ensureIndex( { name: 1 }, { unique: true } )


// Force index on collection with duplicate recipe names – drop the
duplicates
db.recipes.ensureIndex(
    { name: 1 },
    { unique: true, dropDups: true }
)


* dropDups is probably never what you want
Sparse Indexes
// Only documents with field calories will be indexed
db.recipes.ensureIndex(
    { calories: -1 },
    { sparse: true }
)
// Allow multiple documents to not have calories field
db.recipes.ensureIndex(
    { name: 1 , calories: -1 },
    { unique: true, sparse: true }
)
* Missing fields are stored as null(s) in the index
Geospatial Indexes
// Add latitude, longitude coordinates
{
     name: '10gen Sydney’,
     loc: [ 151.21037, -33.88456 ]
}
// Index the coordinates
db.locations.ensureIndex( { loc : '2d' } )


// Query for locations 'near' a particular coordinate
db.locations.find({
     loc: { $near: [ 151.21, -33.88 ] }
})
TTL Collections
// Documents must have a BSON UTC Date field
{ 'status' : ISODate('2012-11-09T11:44:07.211Z'), … }


// Documents are removed after 'expireAfterSeconds' seconds
db.recipes.ensureIndex(
    { submitted_date: 1 },
    { expireAfterSeconds: 3600 }
)
Limitations
• Collections can not have > 64 indexes.

• Index keys can not be > 1024 bytes (1K).

• The name of an index, including the namespace, must be <
  128 characters.
• Queries can only use 1 index*

• Indexes have storage requirements, and impact the
  performance of writes.
• In memory sort (no-index) limited to 32mb of return data.
Optimise Your Queries
Profiling Slow Ops
db.setProfilingLevel( n , slowms=100ms )


n=0 profiler off
n=1 record operations longer than slowms
n=2 record all queries


db.system.profile.find()




* The profile collection is a capped collection, and fixed in size
The Explain Plan (Pre Index)
db.recipes.find( { calories:
    { $lt : 40 } }
).explain( )
{
    "cursor" : "BasicCursor" ,
    "n" : 42,
    "nscannedObjects” : 12345
    "nscanned" : 12345,
    ...
    "millis" : 356,
    ...
}
* Doesn’t use cached plans, re-evals and resets cache
The Explain Plan (Post Index)
db.recipes.find( { calories:
    { $lt : 40 } }
).explain( )
{
    "cursor" : "BtreeCursor calories_-1" ,
    "n" : 42,
    "nscannedObjects": 42
    "nscanned" : 42,
    ...
    "millis" : 0,
    ...
}
* Doesn’t use cached plans, re-evals and resets cache
The Query Optimiser
The Query Optimiser
• For each "type" of query, MongoDB
  periodically tries all useful indexes
• Aborts the rest as soon as one plan wins
• The winning plan is temporarily cached for
  each “type” of query
Manually Select Index to Use
// Tell the database what index to use
db.recipes.find({
  calories: { $lt: 1000 } }
).hint({ _id: 1 })


// Tell the database to NOT use an index
db.recipes.find(
  { calories: { $lt: 1000 } }
).hint({ $natural: 1 })
Use Indexes to Sort Query
Results
// Given the following index
db.collection.ensureIndex({ a:1, b:1 , c:1, d:1 })

// The following query and sort operations can use the index
db.collection.find( ).sort({ a:1 })
db.collection.find( ).sort({ a:1, b:1 })

db.collection.find({ a:4 }).sort({ a:1, b:1 })
db.collection.find({ b:5 }).sort({ a:1, b:1 })
Indexes that won’t work for
sorting query results
// Given the following index
db.collection.ensureIndex({ a:1, b:1, c:1, d:1 })


// These can not sort using the index
db.collection.find( ).sort({ b: 1 })
db.collection.find({ b: 5 }).sort({ b: 1 })
Index Covered Queries
// MongoDB can return data from just the index
db.recipes.ensureIndex({ main_ingredient: 1, name: 1 })

// Return only the ingredients field
db.recipes.find(
   { main_ingredient: 'chicken’ },
   { _id: 0, name: 1 }
)

// indexOnly will be true in the explain plan
db.recipes.find(
    { main_ingredient: 'chicken' },
    { _id: 0, name: 1 }
).explain()
{
    "indexOnly": true,
}
Absent or suboptimal
indexes are the most
common avoidable
MongoDB performance
problem.
Avoiding Common
Mistakes
Trying to Use Multiple
Indexes
// MongoDB can only use one index for a query
db.collection.ensureIndex({ a: 1 })
db.collection.ensureIndex({ b: 1 })


// Only one of the above indexes is used
db.collection.find({ a: 3, b: 4 })
Compound Key Mistakes
// Compound key indexes are very effective
db.collection.ensureIndex({ a: 1, b: 1, c: 1 })


// But only if the query is a prefix of the index


// This query can't effectively use the index
db.collection.find({ c: 2 })


// …but this query can
db.collection.find({ a: 3, b: 5 })
Low Selectivity Indexes
db.collection.distinct('status’)
[ 'new', 'processed' ]


db.collection.ensureIndex({ status: 1 })


// Low selectivity indexes provide little benefit
db.collection.find({ status: 'new' })


// Better
db.collection.ensureIndex({ status: 1, created_at: -1 })
db.collection.find(
  { status: 'new' }
).sort({ created_at: -1 })
Regular Expressions
db.users.ensureIndex({ username: 1 })


// Left anchored regex queries can use the index
db.users.find({ username: /^joe smith/ })


// But not generic regexes
db.users.find({username: /smith/ })


// Or case insensitive queries
db.users.find({ username: /Joe/i })
Negation
// Indexes aren't helpful with negations
db.things.ensureIndex({ x: 1 })

// e.g. "not equal" queries
db.things.find({ x: { $ne: 3 } })

// …or "not in" queries
db.things.find({ x: { $nin: [2, 3, 4 ] } })

// …or the $not operator
db.people.find({ name: { $not: 'John Doe' } })
Choosing the right
indexes is one of the
most important things
you can do as a
MongoDB developer so
take the time to get your
indexes right!
#MongoMelbourne




Thank you
Stephen Steneker
Support Engineer, 10gen

Más contenido relacionado

Was ist angesagt?

C* Summit EU 2013: Cassandra Made Simple with CQL Drivers and DevCenter
C* Summit EU 2013: Cassandra Made Simple with CQL Drivers and DevCenter C* Summit EU 2013: Cassandra Made Simple with CQL Drivers and DevCenter
C* Summit EU 2013: Cassandra Made Simple with CQL Drivers and DevCenter DataStax Academy
 
Connecting NodeJS & MongoDB
Connecting NodeJS & MongoDBConnecting NodeJS & MongoDB
Connecting NodeJS & MongoDBEnoch Joshua
 
Mongo db pefrormance optimization strategies
Mongo db pefrormance optimization strategiesMongo db pefrormance optimization strategies
Mongo db pefrormance optimization strategiesronwarshawsky
 
MongoDB for Beginners
MongoDB for BeginnersMongoDB for Beginners
MongoDB for BeginnersEnoch Joshua
 
An introduction to CouchDB
An introduction to CouchDBAn introduction to CouchDB
An introduction to CouchDBDavid Coallier
 
Introduction to mongo db
Introduction to mongo dbIntroduction to mongo db
Introduction to mongo dbRohit Bishnoi
 
MongoDB Roadmap
MongoDB RoadmapMongoDB Roadmap
MongoDB RoadmapMongoDB
 
MongoDB basics & Introduction
MongoDB basics & IntroductionMongoDB basics & Introduction
MongoDB basics & IntroductionJerwin Roy
 
Back to Basics Spanish Webinar 3 - Introducción a los replica sets
Back to Basics Spanish Webinar 3 - Introducción a los replica setsBack to Basics Spanish Webinar 3 - Introducción a los replica sets
Back to Basics Spanish Webinar 3 - Introducción a los replica setsMongoDB
 
Optimizing Slow Queries with Indexes and Creativity
Optimizing Slow Queries with Indexes and CreativityOptimizing Slow Queries with Indexes and Creativity
Optimizing Slow Queries with Indexes and CreativityMongoDB
 
Mongodb in-anger-boston-rb-2011
Mongodb in-anger-boston-rb-2011Mongodb in-anger-boston-rb-2011
Mongodb in-anger-boston-rb-2011bostonrb
 
[2C6]SQLite DB 의 입출력 특성분석 : Android 와 Tizen 사례
[2C6]SQLite DB 의 입출력 특성분석 : Android 와 Tizen 사례[2C6]SQLite DB 의 입출력 특성분석 : Android 와 Tizen 사례
[2C6]SQLite DB 의 입출력 특성분석 : Android 와 Tizen 사례NAVER D2
 
DMDW Extra Lesson - NoSql and MongoDB
DMDW  Extra Lesson - NoSql and MongoDBDMDW  Extra Lesson - NoSql and MongoDB
DMDW Extra Lesson - NoSql and MongoDBJohannes Hoppe
 
Cassandra Day Chicago 2015: Advanced Data Modeling
Cassandra Day Chicago 2015: Advanced Data ModelingCassandra Day Chicago 2015: Advanced Data Modeling
Cassandra Day Chicago 2015: Advanced Data ModelingDataStax Academy
 
Better Data Persistence on Android
Better Data Persistence on AndroidBetter Data Persistence on Android
Better Data Persistence on AndroidEric Maxwell
 
MongoDB Shell Tips & Tricks
MongoDB Shell Tips & TricksMongoDB Shell Tips & Tricks
MongoDB Shell Tips & TricksMongoDB
 
How Clean is your database? Data scrubbing for all skills sets
How Clean is your database? Data scrubbing for all skills setsHow Clean is your database? Data scrubbing for all skills sets
How Clean is your database? Data scrubbing for all skills setsChad Petrovay
 
MongoDB + Java - Everything you need to know
MongoDB + Java - Everything you need to know MongoDB + Java - Everything you need to know
MongoDB + Java - Everything you need to know Norberto Leite
 

Was ist angesagt? (20)

C* Summit EU 2013: Cassandra Made Simple with CQL Drivers and DevCenter
C* Summit EU 2013: Cassandra Made Simple with CQL Drivers and DevCenter C* Summit EU 2013: Cassandra Made Simple with CQL Drivers and DevCenter
C* Summit EU 2013: Cassandra Made Simple with CQL Drivers and DevCenter
 
Connecting NodeJS & MongoDB
Connecting NodeJS & MongoDBConnecting NodeJS & MongoDB
Connecting NodeJS & MongoDB
 
Mongo db pefrormance optimization strategies
Mongo db pefrormance optimization strategiesMongo db pefrormance optimization strategies
Mongo db pefrormance optimization strategies
 
Intro To Couch Db
Intro To Couch DbIntro To Couch Db
Intro To Couch Db
 
MongoDB for Beginners
MongoDB for BeginnersMongoDB for Beginners
MongoDB for Beginners
 
An introduction to CouchDB
An introduction to CouchDBAn introduction to CouchDB
An introduction to CouchDB
 
Introduction to mongo db
Introduction to mongo dbIntroduction to mongo db
Introduction to mongo db
 
MongoDB Roadmap
MongoDB RoadmapMongoDB Roadmap
MongoDB Roadmap
 
MongoDB basics & Introduction
MongoDB basics & IntroductionMongoDB basics & Introduction
MongoDB basics & Introduction
 
Back to Basics Spanish Webinar 3 - Introducción a los replica sets
Back to Basics Spanish Webinar 3 - Introducción a los replica setsBack to Basics Spanish Webinar 3 - Introducción a los replica sets
Back to Basics Spanish Webinar 3 - Introducción a los replica sets
 
Optimizing Slow Queries with Indexes and Creativity
Optimizing Slow Queries with Indexes and CreativityOptimizing Slow Queries with Indexes and Creativity
Optimizing Slow Queries with Indexes and Creativity
 
Mongodb in-anger-boston-rb-2011
Mongodb in-anger-boston-rb-2011Mongodb in-anger-boston-rb-2011
Mongodb in-anger-boston-rb-2011
 
[2C6]SQLite DB 의 입출력 특성분석 : Android 와 Tizen 사례
[2C6]SQLite DB 의 입출력 특성분석 : Android 와 Tizen 사례[2C6]SQLite DB 의 입출력 특성분석 : Android 와 Tizen 사례
[2C6]SQLite DB 의 입출력 특성분석 : Android 와 Tizen 사례
 
DMDW Extra Lesson - NoSql and MongoDB
DMDW  Extra Lesson - NoSql and MongoDBDMDW  Extra Lesson - NoSql and MongoDB
DMDW Extra Lesson - NoSql and MongoDB
 
Cassandra Day Chicago 2015: Advanced Data Modeling
Cassandra Day Chicago 2015: Advanced Data ModelingCassandra Day Chicago 2015: Advanced Data Modeling
Cassandra Day Chicago 2015: Advanced Data Modeling
 
Mongo DB
Mongo DB Mongo DB
Mongo DB
 
Better Data Persistence on Android
Better Data Persistence on AndroidBetter Data Persistence on Android
Better Data Persistence on Android
 
MongoDB Shell Tips & Tricks
MongoDB Shell Tips & TricksMongoDB Shell Tips & Tricks
MongoDB Shell Tips & Tricks
 
How Clean is your database? Data scrubbing for all skills sets
How Clean is your database? Data scrubbing for all skills setsHow Clean is your database? Data scrubbing for all skills sets
How Clean is your database? Data scrubbing for all skills sets
 
MongoDB + Java - Everything you need to know
MongoDB + Java - Everything you need to know MongoDB + Java - Everything you need to know
MongoDB + Java - Everything you need to know
 

Andere mochten auch

Giftivo mongodb
Giftivo mongodbGiftivo mongodb
Giftivo mongodbMongoDB
 
Webinar: Replication and Replica Sets
Webinar: Replication and Replica SetsWebinar: Replication and Replica Sets
Webinar: Replication and Replica SetsMongoDB
 
Building your first java application with MongoDB
Building your first java application with MongoDB Building your first java application with MongoDB
Building your first java application with MongoDB MongoDB
 
A Morning with MongoDB - Helsinki
A Morning with MongoDB - HelsinkiA Morning with MongoDB - Helsinki
A Morning with MongoDB - HelsinkiMongoDB
 
Branf final bringing mongodb into your organization - mongo db-boston2012
Branf final   bringing mongodb into your organization - mongo db-boston2012Branf final   bringing mongodb into your organization - mongo db-boston2012
Branf final bringing mongodb into your organization - mongo db-boston2012MongoDB
 
Discover MongoDB - Israel
Discover MongoDB - IsraelDiscover MongoDB - Israel
Discover MongoDB - IsraelMichael Fiedler
 

Andere mochten auch (6)

Giftivo mongodb
Giftivo mongodbGiftivo mongodb
Giftivo mongodb
 
Webinar: Replication and Replica Sets
Webinar: Replication and Replica SetsWebinar: Replication and Replica Sets
Webinar: Replication and Replica Sets
 
Building your first java application with MongoDB
Building your first java application with MongoDB Building your first java application with MongoDB
Building your first java application with MongoDB
 
A Morning with MongoDB - Helsinki
A Morning with MongoDB - HelsinkiA Morning with MongoDB - Helsinki
A Morning with MongoDB - Helsinki
 
Branf final bringing mongodb into your organization - mongo db-boston2012
Branf final   bringing mongodb into your organization - mongo db-boston2012Branf final   bringing mongodb into your organization - mongo db-boston2012
Branf final bringing mongodb into your organization - mongo db-boston2012
 
Discover MongoDB - Israel
Discover MongoDB - IsraelDiscover MongoDB - Israel
Discover MongoDB - Israel
 

Ähnlich wie Indexing and Query Optimisation

Indexing and Query Optimisation
Indexing and Query OptimisationIndexing and Query Optimisation
Indexing and Query OptimisationMongoDB
 
Indexing and Query Optimization
Indexing and Query OptimizationIndexing and Query Optimization
Indexing and Query OptimizationMongoDB
 
Indexing & Query Optimization
Indexing & Query OptimizationIndexing & Query Optimization
Indexing & Query OptimizationMongoDB
 
Webinar: Indexing and Query Optimization
Webinar: Indexing and Query OptimizationWebinar: Indexing and Query Optimization
Webinar: Indexing and Query OptimizationMongoDB
 
Indexing and Query Optimization
Indexing and Query OptimizationIndexing and Query Optimization
Indexing and Query OptimizationMongoDB
 
Indexing Strategies to Help You Scale
Indexing Strategies to Help You ScaleIndexing Strategies to Help You Scale
Indexing Strategies to Help You ScaleMongoDB
 
Fast querying indexing for performance (4)
Fast querying   indexing for performance (4)Fast querying   indexing for performance (4)
Fast querying indexing for performance (4)MongoDB
 
unit 4,Indexes in database.docx
unit 4,Indexes in database.docxunit 4,Indexes in database.docx
unit 4,Indexes in database.docxRaviRajput416403
 
Mongoseattle indexing-2010-07-27
Mongoseattle indexing-2010-07-27Mongoseattle indexing-2010-07-27
Mongoseattle indexing-2010-07-27MongoDB
 
Indexing documents
Indexing documentsIndexing documents
Indexing documentsMongoDB
 
Indexing and Query Optimizer (Richard Kreuter)
Indexing and Query Optimizer (Richard Kreuter)Indexing and Query Optimizer (Richard Kreuter)
Indexing and Query Optimizer (Richard Kreuter)MongoDB
 
MongoDB.local DC 2018: Tips and Tricks for Avoiding Common Query Pitfalls
MongoDB.local DC 2018: Tips and Tricks for Avoiding Common Query PitfallsMongoDB.local DC 2018: Tips and Tricks for Avoiding Common Query Pitfalls
MongoDB.local DC 2018: Tips and Tricks for Avoiding Common Query PitfallsMongoDB
 
Indexing and Query Optimizer (Mongo Austin)
Indexing and Query Optimizer (Mongo Austin)Indexing and Query Optimizer (Mongo Austin)
Indexing and Query Optimizer (Mongo Austin)MongoDB
 
Mongophilly indexing-2011-04-26
Mongophilly indexing-2011-04-26Mongophilly indexing-2011-04-26
Mongophilly indexing-2011-04-26kreuter
 
Indexing and Query Optimizer
Indexing and Query OptimizerIndexing and Query Optimizer
Indexing and Query OptimizerMongoDB
 
MongoDB and Indexes - MUG Denver - 20160329
MongoDB and Indexes - MUG Denver - 20160329MongoDB and Indexes - MUG Denver - 20160329
MongoDB and Indexes - MUG Denver - 20160329Douglas Duncan
 
Indexing with MongoDB
Indexing with MongoDBIndexing with MongoDB
Indexing with MongoDBMongoDB
 
Automated Slow Query Analysis: Dex the Index Robot
Automated Slow Query Analysis: Dex the Index RobotAutomated Slow Query Analysis: Dex the Index Robot
Automated Slow Query Analysis: Dex the Index RobotMongoDB
 
Introduction to-mongo db-execution-plan-optimizer-final
Introduction to-mongo db-execution-plan-optimizer-finalIntroduction to-mongo db-execution-plan-optimizer-final
Introduction to-mongo db-execution-plan-optimizer-finalM Malai
 

Ähnlich wie Indexing and Query Optimisation (20)

Indexing and Query Optimisation
Indexing and Query OptimisationIndexing and Query Optimisation
Indexing and Query Optimisation
 
Indexing and Query Optimization
Indexing and Query OptimizationIndexing and Query Optimization
Indexing and Query Optimization
 
Indexing & Query Optimization
Indexing & Query OptimizationIndexing & Query Optimization
Indexing & Query Optimization
 
Webinar: Indexing and Query Optimization
Webinar: Indexing and Query OptimizationWebinar: Indexing and Query Optimization
Webinar: Indexing and Query Optimization
 
Indexing and Query Optimization
Indexing and Query OptimizationIndexing and Query Optimization
Indexing and Query Optimization
 
Indexing Strategies to Help You Scale
Indexing Strategies to Help You ScaleIndexing Strategies to Help You Scale
Indexing Strategies to Help You Scale
 
Fast querying indexing for performance (4)
Fast querying   indexing for performance (4)Fast querying   indexing for performance (4)
Fast querying indexing for performance (4)
 
unit 4,Indexes in database.docx
unit 4,Indexes in database.docxunit 4,Indexes in database.docx
unit 4,Indexes in database.docx
 
Mongoseattle indexing-2010-07-27
Mongoseattle indexing-2010-07-27Mongoseattle indexing-2010-07-27
Mongoseattle indexing-2010-07-27
 
Indexing documents
Indexing documentsIndexing documents
Indexing documents
 
Indexing and Query Optimizer (Richard Kreuter)
Indexing and Query Optimizer (Richard Kreuter)Indexing and Query Optimizer (Richard Kreuter)
Indexing and Query Optimizer (Richard Kreuter)
 
MongoDB.local DC 2018: Tips and Tricks for Avoiding Common Query Pitfalls
MongoDB.local DC 2018: Tips and Tricks for Avoiding Common Query PitfallsMongoDB.local DC 2018: Tips and Tricks for Avoiding Common Query Pitfalls
MongoDB.local DC 2018: Tips and Tricks for Avoiding Common Query Pitfalls
 
Indexing and Query Optimizer (Mongo Austin)
Indexing and Query Optimizer (Mongo Austin)Indexing and Query Optimizer (Mongo Austin)
Indexing and Query Optimizer (Mongo Austin)
 
Mongophilly indexing-2011-04-26
Mongophilly indexing-2011-04-26Mongophilly indexing-2011-04-26
Mongophilly indexing-2011-04-26
 
Indexing and Query Optimizer
Indexing and Query OptimizerIndexing and Query Optimizer
Indexing and Query Optimizer
 
MongoDB and Indexes - MUG Denver - 20160329
MongoDB and Indexes - MUG Denver - 20160329MongoDB and Indexes - MUG Denver - 20160329
MongoDB and Indexes - MUG Denver - 20160329
 
Indexing with MongoDB
Indexing with MongoDBIndexing with MongoDB
Indexing with MongoDB
 
Indexing In MongoDB
Indexing In MongoDBIndexing In MongoDB
Indexing In MongoDB
 
Automated Slow Query Analysis: Dex the Index Robot
Automated Slow Query Analysis: Dex the Index RobotAutomated Slow Query Analysis: Dex the Index Robot
Automated Slow Query Analysis: Dex the Index Robot
 
Introduction to-mongo db-execution-plan-optimizer-final
Introduction to-mongo db-execution-plan-optimizer-finalIntroduction to-mongo db-execution-plan-optimizer-final
Introduction to-mongo db-execution-plan-optimizer-final
 

Mehr von MongoDB

MongoDB SoCal 2020: Migrate Anything* to MongoDB Atlas
MongoDB SoCal 2020: Migrate Anything* to MongoDB AtlasMongoDB SoCal 2020: Migrate Anything* to MongoDB Atlas
MongoDB SoCal 2020: Migrate Anything* to MongoDB AtlasMongoDB
 
MongoDB SoCal 2020: Go on a Data Safari with MongoDB Charts!
MongoDB SoCal 2020: Go on a Data Safari with MongoDB Charts!MongoDB SoCal 2020: Go on a Data Safari with MongoDB Charts!
MongoDB SoCal 2020: Go on a Data Safari with MongoDB Charts!MongoDB
 
MongoDB SoCal 2020: Using MongoDB Services in Kubernetes: Any Platform, Devel...
MongoDB SoCal 2020: Using MongoDB Services in Kubernetes: Any Platform, Devel...MongoDB SoCal 2020: Using MongoDB Services in Kubernetes: Any Platform, Devel...
MongoDB SoCal 2020: Using MongoDB Services in Kubernetes: Any Platform, Devel...MongoDB
 
MongoDB SoCal 2020: A Complete Methodology of Data Modeling for MongoDB
MongoDB SoCal 2020: A Complete Methodology of Data Modeling for MongoDBMongoDB SoCal 2020: A Complete Methodology of Data Modeling for MongoDB
MongoDB SoCal 2020: A Complete Methodology of Data Modeling for MongoDBMongoDB
 
MongoDB SoCal 2020: From Pharmacist to Analyst: Leveraging MongoDB for Real-T...
MongoDB SoCal 2020: From Pharmacist to Analyst: Leveraging MongoDB for Real-T...MongoDB SoCal 2020: From Pharmacist to Analyst: Leveraging MongoDB for Real-T...
MongoDB SoCal 2020: From Pharmacist to Analyst: Leveraging MongoDB for Real-T...MongoDB
 
MongoDB SoCal 2020: Best Practices for Working with IoT and Time-series Data
MongoDB SoCal 2020: Best Practices for Working with IoT and Time-series DataMongoDB SoCal 2020: Best Practices for Working with IoT and Time-series Data
MongoDB SoCal 2020: Best Practices for Working with IoT and Time-series DataMongoDB
 
MongoDB SoCal 2020: MongoDB Atlas Jump Start
 MongoDB SoCal 2020: MongoDB Atlas Jump Start MongoDB SoCal 2020: MongoDB Atlas Jump Start
MongoDB SoCal 2020: MongoDB Atlas Jump StartMongoDB
 
MongoDB .local San Francisco 2020: Powering the new age data demands [Infosys]
MongoDB .local San Francisco 2020: Powering the new age data demands [Infosys]MongoDB .local San Francisco 2020: Powering the new age data demands [Infosys]
MongoDB .local San Francisco 2020: Powering the new age data demands [Infosys]MongoDB
 
MongoDB .local San Francisco 2020: Using Client Side Encryption in MongoDB 4.2
MongoDB .local San Francisco 2020: Using Client Side Encryption in MongoDB 4.2MongoDB .local San Francisco 2020: Using Client Side Encryption in MongoDB 4.2
MongoDB .local San Francisco 2020: Using Client Side Encryption in MongoDB 4.2MongoDB
 
MongoDB .local San Francisco 2020: Using MongoDB Services in Kubernetes: any ...
MongoDB .local San Francisco 2020: Using MongoDB Services in Kubernetes: any ...MongoDB .local San Francisco 2020: Using MongoDB Services in Kubernetes: any ...
MongoDB .local San Francisco 2020: Using MongoDB Services in Kubernetes: any ...MongoDB
 
MongoDB .local San Francisco 2020: Go on a Data Safari with MongoDB Charts!
MongoDB .local San Francisco 2020: Go on a Data Safari with MongoDB Charts!MongoDB .local San Francisco 2020: Go on a Data Safari with MongoDB Charts!
MongoDB .local San Francisco 2020: Go on a Data Safari with MongoDB Charts!MongoDB
 
MongoDB .local San Francisco 2020: From SQL to NoSQL -- Changing Your Mindset
MongoDB .local San Francisco 2020: From SQL to NoSQL -- Changing Your MindsetMongoDB .local San Francisco 2020: From SQL to NoSQL -- Changing Your Mindset
MongoDB .local San Francisco 2020: From SQL to NoSQL -- Changing Your MindsetMongoDB
 
MongoDB .local San Francisco 2020: MongoDB Atlas Jumpstart
MongoDB .local San Francisco 2020: MongoDB Atlas JumpstartMongoDB .local San Francisco 2020: MongoDB Atlas Jumpstart
MongoDB .local San Francisco 2020: MongoDB Atlas JumpstartMongoDB
 
MongoDB .local San Francisco 2020: Tips and Tricks++ for Querying and Indexin...
MongoDB .local San Francisco 2020: Tips and Tricks++ for Querying and Indexin...MongoDB .local San Francisco 2020: Tips and Tricks++ for Querying and Indexin...
MongoDB .local San Francisco 2020: Tips and Tricks++ for Querying and Indexin...MongoDB
 
MongoDB .local San Francisco 2020: Aggregation Pipeline Power++
MongoDB .local San Francisco 2020: Aggregation Pipeline Power++MongoDB .local San Francisco 2020: Aggregation Pipeline Power++
MongoDB .local San Francisco 2020: Aggregation Pipeline Power++MongoDB
 
MongoDB .local San Francisco 2020: A Complete Methodology of Data Modeling fo...
MongoDB .local San Francisco 2020: A Complete Methodology of Data Modeling fo...MongoDB .local San Francisco 2020: A Complete Methodology of Data Modeling fo...
MongoDB .local San Francisco 2020: A Complete Methodology of Data Modeling fo...MongoDB
 
MongoDB .local San Francisco 2020: MongoDB Atlas Data Lake Technical Deep Dive
MongoDB .local San Francisco 2020: MongoDB Atlas Data Lake Technical Deep DiveMongoDB .local San Francisco 2020: MongoDB Atlas Data Lake Technical Deep Dive
MongoDB .local San Francisco 2020: MongoDB Atlas Data Lake Technical Deep DiveMongoDB
 
MongoDB .local San Francisco 2020: Developing Alexa Skills with MongoDB & Golang
MongoDB .local San Francisco 2020: Developing Alexa Skills with MongoDB & GolangMongoDB .local San Francisco 2020: Developing Alexa Skills with MongoDB & Golang
MongoDB .local San Francisco 2020: Developing Alexa Skills with MongoDB & GolangMongoDB
 
MongoDB .local Paris 2020: Realm : l'ingrédient secret pour de meilleures app...
MongoDB .local Paris 2020: Realm : l'ingrédient secret pour de meilleures app...MongoDB .local Paris 2020: Realm : l'ingrédient secret pour de meilleures app...
MongoDB .local Paris 2020: Realm : l'ingrédient secret pour de meilleures app...MongoDB
 
MongoDB .local Paris 2020: Upply @MongoDB : Upply : Quand le Machine Learning...
MongoDB .local Paris 2020: Upply @MongoDB : Upply : Quand le Machine Learning...MongoDB .local Paris 2020: Upply @MongoDB : Upply : Quand le Machine Learning...
MongoDB .local Paris 2020: Upply @MongoDB : Upply : Quand le Machine Learning...MongoDB
 

Mehr von MongoDB (20)

MongoDB SoCal 2020: Migrate Anything* to MongoDB Atlas
MongoDB SoCal 2020: Migrate Anything* to MongoDB AtlasMongoDB SoCal 2020: Migrate Anything* to MongoDB Atlas
MongoDB SoCal 2020: Migrate Anything* to MongoDB Atlas
 
MongoDB SoCal 2020: Go on a Data Safari with MongoDB Charts!
MongoDB SoCal 2020: Go on a Data Safari with MongoDB Charts!MongoDB SoCal 2020: Go on a Data Safari with MongoDB Charts!
MongoDB SoCal 2020: Go on a Data Safari with MongoDB Charts!
 
MongoDB SoCal 2020: Using MongoDB Services in Kubernetes: Any Platform, Devel...
MongoDB SoCal 2020: Using MongoDB Services in Kubernetes: Any Platform, Devel...MongoDB SoCal 2020: Using MongoDB Services in Kubernetes: Any Platform, Devel...
MongoDB SoCal 2020: Using MongoDB Services in Kubernetes: Any Platform, Devel...
 
MongoDB SoCal 2020: A Complete Methodology of Data Modeling for MongoDB
MongoDB SoCal 2020: A Complete Methodology of Data Modeling for MongoDBMongoDB SoCal 2020: A Complete Methodology of Data Modeling for MongoDB
MongoDB SoCal 2020: A Complete Methodology of Data Modeling for MongoDB
 
MongoDB SoCal 2020: From Pharmacist to Analyst: Leveraging MongoDB for Real-T...
MongoDB SoCal 2020: From Pharmacist to Analyst: Leveraging MongoDB for Real-T...MongoDB SoCal 2020: From Pharmacist to Analyst: Leveraging MongoDB for Real-T...
MongoDB SoCal 2020: From Pharmacist to Analyst: Leveraging MongoDB for Real-T...
 
MongoDB SoCal 2020: Best Practices for Working with IoT and Time-series Data
MongoDB SoCal 2020: Best Practices for Working with IoT and Time-series DataMongoDB SoCal 2020: Best Practices for Working with IoT and Time-series Data
MongoDB SoCal 2020: Best Practices for Working with IoT and Time-series Data
 
MongoDB SoCal 2020: MongoDB Atlas Jump Start
 MongoDB SoCal 2020: MongoDB Atlas Jump Start MongoDB SoCal 2020: MongoDB Atlas Jump Start
MongoDB SoCal 2020: MongoDB Atlas Jump Start
 
MongoDB .local San Francisco 2020: Powering the new age data demands [Infosys]
MongoDB .local San Francisco 2020: Powering the new age data demands [Infosys]MongoDB .local San Francisco 2020: Powering the new age data demands [Infosys]
MongoDB .local San Francisco 2020: Powering the new age data demands [Infosys]
 
MongoDB .local San Francisco 2020: Using Client Side Encryption in MongoDB 4.2
MongoDB .local San Francisco 2020: Using Client Side Encryption in MongoDB 4.2MongoDB .local San Francisco 2020: Using Client Side Encryption in MongoDB 4.2
MongoDB .local San Francisco 2020: Using Client Side Encryption in MongoDB 4.2
 
MongoDB .local San Francisco 2020: Using MongoDB Services in Kubernetes: any ...
MongoDB .local San Francisco 2020: Using MongoDB Services in Kubernetes: any ...MongoDB .local San Francisco 2020: Using MongoDB Services in Kubernetes: any ...
MongoDB .local San Francisco 2020: Using MongoDB Services in Kubernetes: any ...
 
MongoDB .local San Francisco 2020: Go on a Data Safari with MongoDB Charts!
MongoDB .local San Francisco 2020: Go on a Data Safari with MongoDB Charts!MongoDB .local San Francisco 2020: Go on a Data Safari with MongoDB Charts!
MongoDB .local San Francisco 2020: Go on a Data Safari with MongoDB Charts!
 
MongoDB .local San Francisco 2020: From SQL to NoSQL -- Changing Your Mindset
MongoDB .local San Francisco 2020: From SQL to NoSQL -- Changing Your MindsetMongoDB .local San Francisco 2020: From SQL to NoSQL -- Changing Your Mindset
MongoDB .local San Francisco 2020: From SQL to NoSQL -- Changing Your Mindset
 
MongoDB .local San Francisco 2020: MongoDB Atlas Jumpstart
MongoDB .local San Francisco 2020: MongoDB Atlas JumpstartMongoDB .local San Francisco 2020: MongoDB Atlas Jumpstart
MongoDB .local San Francisco 2020: MongoDB Atlas Jumpstart
 
MongoDB .local San Francisco 2020: Tips and Tricks++ for Querying and Indexin...
MongoDB .local San Francisco 2020: Tips and Tricks++ for Querying and Indexin...MongoDB .local San Francisco 2020: Tips and Tricks++ for Querying and Indexin...
MongoDB .local San Francisco 2020: Tips and Tricks++ for Querying and Indexin...
 
MongoDB .local San Francisco 2020: Aggregation Pipeline Power++
MongoDB .local San Francisco 2020: Aggregation Pipeline Power++MongoDB .local San Francisco 2020: Aggregation Pipeline Power++
MongoDB .local San Francisco 2020: Aggregation Pipeline Power++
 
MongoDB .local San Francisco 2020: A Complete Methodology of Data Modeling fo...
MongoDB .local San Francisco 2020: A Complete Methodology of Data Modeling fo...MongoDB .local San Francisco 2020: A Complete Methodology of Data Modeling fo...
MongoDB .local San Francisco 2020: A Complete Methodology of Data Modeling fo...
 
MongoDB .local San Francisco 2020: MongoDB Atlas Data Lake Technical Deep Dive
MongoDB .local San Francisco 2020: MongoDB Atlas Data Lake Technical Deep DiveMongoDB .local San Francisco 2020: MongoDB Atlas Data Lake Technical Deep Dive
MongoDB .local San Francisco 2020: MongoDB Atlas Data Lake Technical Deep Dive
 
MongoDB .local San Francisco 2020: Developing Alexa Skills with MongoDB & Golang
MongoDB .local San Francisco 2020: Developing Alexa Skills with MongoDB & GolangMongoDB .local San Francisco 2020: Developing Alexa Skills with MongoDB & Golang
MongoDB .local San Francisco 2020: Developing Alexa Skills with MongoDB & Golang
 
MongoDB .local Paris 2020: Realm : l'ingrédient secret pour de meilleures app...
MongoDB .local Paris 2020: Realm : l'ingrédient secret pour de meilleures app...MongoDB .local Paris 2020: Realm : l'ingrédient secret pour de meilleures app...
MongoDB .local Paris 2020: Realm : l'ingrédient secret pour de meilleures app...
 
MongoDB .local Paris 2020: Upply @MongoDB : Upply : Quand le Machine Learning...
MongoDB .local Paris 2020: Upply @MongoDB : Upply : Quand le Machine Learning...MongoDB .local Paris 2020: Upply @MongoDB : Upply : Quand le Machine Learning...
MongoDB .local Paris 2020: Upply @MongoDB : Upply : Quand le Machine Learning...
 

Indexing and Query Optimisation

  • 1. #MongoMelbourne Indexing and Query Optimisation Stephen Steneker Support Engineer, 10gen Australia
  • 2. Agenda • What are indexes? • Why do I need them? • Working with indexes in MongoDB • Optimise your queries • Avoiding common mistakes
  • 4. What are indexes? Imagine you’re looking for a recipe in a cookbook ordered by recipe name. Looking up a recipe by name is quick and easy.
  • 5. What are indexes? • How would you find a recipe using chicken? • How about a 250-350 calorie recipe using chicken?
  • 6. KRISTINE TO INSERT IMAGE OF COOKBOOK Consult the index!
  • 7. 1 2 3 4 5 6 7 Linked List
  • 8. Finding 7 in Linked List
  • 9. 4 2 6 1 3 5 7 Finding 7 in Tree
  • 10. Indexes in MongoDB are B-trees
  • 11. Queries, inserts and deletes: O(log(n)) time
  • 12. Indexes are the single biggest tuneable performance factor in MongoDB
  • 13. Absent or suboptimal indexes are the most common avoidable MongoDB performance problem.
  • 14. Why do I need indexes? A brief story
  • 15. Working with Indexes in MongoDB
  • 16. How do I create indexes? // Create an index if one does not exist db.recipes.createIndex({ main_ingredient: 1 }) // The client remembers the index and raises no errors db.recipes.ensureIndex({ main_ingredient: 1 }) * 1 means ascending, -1 descending
  • 17. What can be indexed? // Multiple fields (compound key indexes) db.recipes.ensureIndex({ main_ingredient: 1, calories: -1 }) // Arrays of values (multikey indexes) { name: 'Chicken Noodle Soup’, ingredients : ['chicken', 'noodles'] } db.recipes.ensureIndex({ ingredients: 1 })
  • 18. What can be indexed? // Subdocuments { name : 'Pavlova', contributor: { name: 'Ima Aussie', id: 'ima123' } } db.recipes.ensureIndex({ 'contributor.id': 1 }) db.recipes.ensureIndex({ 'contributor': 1 })
  • 19. How do I manage indexes? // List a collection's indexes db.recipes.getIndexes() db.recipes.getIndexKeys() // Drop a specific index db.recipes.dropIndex({ ingredients: 1 }) // Drop all indexes and recreate them db.recipes.reIndex() // Default (unique) index on _id
  • 20. Background Index Builds // Index creation is a blocking operation that can take a long time // Background creation yields to other operations db.recipes.ensureIndex( { ingredients: 1 }, { background: true } )
  • 21. Options • Uniqueness constraints (unique, dropDups) • Sparse Indexes • Geospatial (2d) Indexes • TTL Collections (expireAfterSeconds)
  • 22. Uniqueness Constraints // Only one recipe can have a given value for name db.recipes.ensureIndex( { name: 1 }, { unique: true } ) // Force index on collection with duplicate recipe names – drop the duplicates db.recipes.ensureIndex( { name: 1 }, { unique: true, dropDups: true } ) * dropDups is probably never what you want
  • 23. Sparse Indexes // Only documents with field calories will be indexed db.recipes.ensureIndex( { calories: -1 }, { sparse: true } ) // Allow multiple documents to not have calories field db.recipes.ensureIndex( { name: 1 , calories: -1 }, { unique: true, sparse: true } ) * Missing fields are stored as null(s) in the index
  • 24. Geospatial Indexes // Add latitude, longitude coordinates { name: '10gen Sydney’, loc: [ 151.21037, -33.88456 ] } // Index the coordinates db.locations.ensureIndex( { loc : '2d' } ) // Query for locations 'near' a particular coordinate db.locations.find({ loc: { $near: [ 151.21, -33.88 ] } })
  • 25. TTL Collections // Documents must have a BSON UTC Date field { 'status' : ISODate('2012-11-09T11:44:07.211Z'), … } // Documents are removed after 'expireAfterSeconds' seconds db.recipes.ensureIndex( { submitted_date: 1 }, { expireAfterSeconds: 3600 } )
  • 26. Limitations • Collections can not have > 64 indexes. • Index keys can not be > 1024 bytes (1K). • The name of an index, including the namespace, must be < 128 characters. • Queries can only use 1 index* • Indexes have storage requirements, and impact the performance of writes. • In memory sort (no-index) limited to 32mb of return data.
  • 28. Profiling Slow Ops db.setProfilingLevel( n , slowms=100ms ) n=0 profiler off n=1 record operations longer than slowms n=2 record all queries db.system.profile.find() * The profile collection is a capped collection, and fixed in size
  • 29. The Explain Plan (Pre Index) db.recipes.find( { calories: { $lt : 40 } } ).explain( ) { "cursor" : "BasicCursor" , "n" : 42, "nscannedObjects” : 12345 "nscanned" : 12345, ... "millis" : 356, ... } * Doesn’t use cached plans, re-evals and resets cache
  • 30. The Explain Plan (Post Index) db.recipes.find( { calories: { $lt : 40 } } ).explain( ) { "cursor" : "BtreeCursor calories_-1" , "n" : 42, "nscannedObjects": 42 "nscanned" : 42, ... "millis" : 0, ... } * Doesn’t use cached plans, re-evals and resets cache
  • 32. The Query Optimiser • For each "type" of query, MongoDB periodically tries all useful indexes • Aborts the rest as soon as one plan wins • The winning plan is temporarily cached for each “type” of query
  • 33. Manually Select Index to Use // Tell the database what index to use db.recipes.find({ calories: { $lt: 1000 } } ).hint({ _id: 1 }) // Tell the database to NOT use an index db.recipes.find( { calories: { $lt: 1000 } } ).hint({ $natural: 1 })
  • 34. Use Indexes to Sort Query Results // Given the following index db.collection.ensureIndex({ a:1, b:1 , c:1, d:1 }) // The following query and sort operations can use the index db.collection.find( ).sort({ a:1 }) db.collection.find( ).sort({ a:1, b:1 }) db.collection.find({ a:4 }).sort({ a:1, b:1 }) db.collection.find({ b:5 }).sort({ a:1, b:1 })
  • 35. Indexes that won’t work for sorting query results // Given the following index db.collection.ensureIndex({ a:1, b:1, c:1, d:1 }) // These can not sort using the index db.collection.find( ).sort({ b: 1 }) db.collection.find({ b: 5 }).sort({ b: 1 })
  • 36. Index Covered Queries // MongoDB can return data from just the index db.recipes.ensureIndex({ main_ingredient: 1, name: 1 }) // Return only the ingredients field db.recipes.find( { main_ingredient: 'chicken’ }, { _id: 0, name: 1 } ) // indexOnly will be true in the explain plan db.recipes.find( { main_ingredient: 'chicken' }, { _id: 0, name: 1 } ).explain() { "indexOnly": true, }
  • 37. Absent or suboptimal indexes are the most common avoidable MongoDB performance problem.
  • 39. Trying to Use Multiple Indexes // MongoDB can only use one index for a query db.collection.ensureIndex({ a: 1 }) db.collection.ensureIndex({ b: 1 }) // Only one of the above indexes is used db.collection.find({ a: 3, b: 4 })
  • 40. Compound Key Mistakes // Compound key indexes are very effective db.collection.ensureIndex({ a: 1, b: 1, c: 1 }) // But only if the query is a prefix of the index // This query can't effectively use the index db.collection.find({ c: 2 }) // …but this query can db.collection.find({ a: 3, b: 5 })
  • 41. Low Selectivity Indexes db.collection.distinct('status’) [ 'new', 'processed' ] db.collection.ensureIndex({ status: 1 }) // Low selectivity indexes provide little benefit db.collection.find({ status: 'new' }) // Better db.collection.ensureIndex({ status: 1, created_at: -1 }) db.collection.find( { status: 'new' } ).sort({ created_at: -1 })
  • 42. Regular Expressions db.users.ensureIndex({ username: 1 }) // Left anchored regex queries can use the index db.users.find({ username: /^joe smith/ }) // But not generic regexes db.users.find({username: /smith/ }) // Or case insensitive queries db.users.find({ username: /Joe/i })
  • 43. Negation // Indexes aren't helpful with negations db.things.ensureIndex({ x: 1 }) // e.g. "not equal" queries db.things.find({ x: { $ne: 3 } }) // …or "not in" queries db.things.find({ x: { $nin: [2, 3, 4 ] } }) // …or the $not operator db.people.find({ name: { $not: 'John Doe' } })
  • 44. Choosing the right indexes is one of the most important things you can do as a MongoDB developer so take the time to get your indexes right!