Query performance can either be a constant headache or the unsung hero of an application. MongoDB provides extremely powerful querying capabilities when used properly. As a member of the solutions architecture team I will share more common mistakes observed and some tips and tricks to avoiding them.
HTML Injection Attacks: Impact and Mitigation Strategies
MongoDB .local Munich 2019: Tips and Tricks++ for Querying and Indexing MongoDB
1. WHEN
WHERE
Tips and Tricks++ for Querying and
Indexing MongoDB
Dr. Christian Kurze, Principal Solutions Architect
22 October 2019, 03:00 – 3:45PM
Room 1
11. #MDBLocal
E-S-R
The ordering of index keys in a compound index is critically important.
E-S-R provides guidance that is useful in most cases:
• Equality first
• Sort next
• Range last
12. #MDBLocal
E-S-R
The ordering of index keys in a compound index is critically important.
E-S-R provides guidance that is useful in most cases:
• Equality first
• Sort next
• Range last
What is the difference between
Equality and Range?
14. #MDBLocal
Definitions
Equality Fields Sort
The (entire) requested sort.
.sort({a:1, b:-1})
Bounds:
"[MinKey, MaxKey]",
"[MinKey, MaxKey]"
An exact match on a single
value. For example:
• {x:123}
• {x:{$eq:123}}
• {"x.y":123}
Bounds:
"[123.0, 123.0]"
15. #MDBLocal
Definitions
Equality Fields Sort Range Predicates
Any predicates that are not
exact matches. Some
operators include:
• {z:{$gt:9000}}
• {z:{$lte:1000}}
Bounds:
"[9000.0, inf.0]",
The (entire) requested sort.
.sort({a:1, b:-1})
Bounds:
"[MinKey, MaxKey]",
"[MinKey, MaxKey]"
An exact match on a single
value. For example:
• {x:123}
• {x:{$eq:123}}
• {"x.y":123}
Bounds:
"[123.0, 123.0]"
17. #MDBLocal
Equality
Equality keys are placed first in any order
If present in the query shape,
equality fields should always form the prefix for the index.
18. #MDBLocal
Equality
Equality keys are placed first in any order
If present in the query shape,
equality fields should always form the prefix for the index.
Why?
19. #MDBLocal
Equality
Equality keys are placed first in any order
If present in the query shape,
equality fields should always form the prefix for the index.
db.games.find( {gamertag: "Ace", score: {$gt: 9000}} )
20. #MDBLocal
Equality
db.games.find( {gamertag: "Ace", score: {$gt: 9000}} )
If present in the query shape,
equality fields should always form the prefix for the index.
Equality keys are placed first in any order
43. #MDBLocal
Sort fields are placed next
Placing sort predicates after sequential equality keys allow for the index to:
Provide a non-blocking sort.
Minimize the amount of scanning required.
db.games.find( {gamertag: "Ace"} ).sort( {score: 1} )
Sort
71. #MDBLocal
Range fields are usually last
This allows them to:
Still participate in filtering the data
But does not force a blocking sort.
db.games.find( {gamertag: "Ace", score: {$gt: 9000}} )
Range
124. #MDBLocal
Operator Type Check - Regex
Regex Operators: E, S, or R?
• {str:/car/}
• {str:/^car/i}
Range!
raincarry Carpool
125. #MDBLocal
Operator Type Check - Regex
Regex Operators: E, S, or R?
• {str:/car/}
• {str:/^car/i}
Range!
raincarry Carpool
126. #MDBLocal
Operator Type Check - Regex
Regex Operators: E, S, or R?
• {str:/car/}
• {str:/^car/i}
Range!
raincarry Carpool
127. #MDBLocal
Operator Type Check - Regex
Regex Operators: E, S, or R?
• {str:/car/}
• {str:/^car/i}
Range!
raincarry Carpool
Since the regex is left
anchored, can’t we
skip the middle tree?
128. #MDBLocal
Operator Type Check - Regex
Regex Operators: E, S, or R?
• {str:/car/}
• {str:/^car/i}
Range!
raincarry Carpool
Since the regex is left
anchored, can’t we
skip the middle tree?
129. #MDBLocal
Operator Type Check - Regex
Regex Operators: E, S, or R?
• {str:/car/}
• {str:/^car/i}
Range!
cArrycarry Carpool
Since the regex is left
anchored, can’t we
skip the middle tree?
130. #MDBLocal
Operator Type Check - Regex
Regex Operators: E, S, or R?
• {str:/car/}
• {str:/^car/i}
Range!
cArrycarry Carpool
Since the regex is left
anchored, can’t we
skip the middle tree?
131. #MDBLocal
Operator Type Check - Regex
Regex Operators: E, S, or R?
• {str:/car/}
• {str:/^car/i}
Range!
cArrycarry Carpool
132. #MDBLocal
Operator Type Check - $in
$in filters: E, S, or R?
• {field:{$in:[1,3]}}
… it depends with respect to key
ordering
Alone: a series of Equality matches
Combined: possibly a Range
134. #MDBLocal
Operator Type Check - $in
.find({field:{$in:[1,3]}})
"indexBounds" : {
"field" : [
"[1.0, 1.0]",
"[3.0, 3.0]"
]
}
?
1 2 3
C G B F A D
db.coll.createIndex({field:1, sortField:1})
135. #MDBLocal
Operator Type Check - $in
.find({field:{$in:[1,3]}})
"indexBounds" : {
"field" : [
"[1.0, 1.0]",
"[3.0, 3.0]"
]
}
?
1 2 3
C G B F A D
db.coll.createIndex({field:1, sortField:1})
136. #MDBLocal
Operator Type Check - $in
.find({field:{$in:[1,3]}})
"indexBounds" : {
"field" : [
"[1.0, 1.0]",
"[3.0, 3.0]"
]
}
?
1 2 3
C G B F A D
db.coll.createIndex({field:1, sortField:1})
137. #MDBLocal
Operator Type Check - $in
.find({field:{$in:[1,3]}})
"indexBounds" : {
"field" : [
"[1.0, 1.0]",
"[3.0, 3.0]"
]
}
?
1 2 3
C G B F A D
db.coll.createIndex({field:1, sortField:1})
138. #MDBLocal
Operator Type Check - $in
.find({field:{$in:[1,3]}})
"indexBounds" : {
"field" : [
"[1.0, 1.0]",
"[3.0, 3.0]"
]
}
?
1 2 3
C G B F A D
db.coll.createIndex({field:1, sortField:1})
139. #MDBLocal
Operator Type Check - $in
.find({field:{$in:[1,3]}})
"indexBounds" : {
"field" : [
"[1.0, 1.0]",
"[3.0, 3.0]"
]
}
?
1 2 3
C G B F A D
db.coll.createIndex({field:1, sortField:1})
140. #MDBLocal
Operator Type Check - $in
.find({field:{$in:[1,3]}})
.sort({sortField:1})
"indexBounds" : {
"field" : [
"[1.0, 1.0]",
"[3.0, 3.0]"
]
}
?
1 2 3
C G B F A D
db.coll.createIndex({field:1, sortField:1})
141. #MDBLocal
Operator Type Check - $in
.find({field:{$in:[1,3]}})
.sort({sortField:1})
"indexBounds" : {
"field" : [
"[1.0, 1.0]",
"[3.0, 3.0]"
],
"sortField" : [
"[MinKey, MaxKey]"
]
}
?
1 2 3
C G B F A D
db.coll.createIndex({field:1, sortField:1})
142. #MDBLocal
Operator Type Check - $in
.find({field:{$in:[1,3]}})
.sort({sortField:1})
"indexBounds" : {
"field" : [
"[1.0, 1.0]",
"[3.0, 3.0]"
],
"sortField" : [
"[MinKey, MaxKey]"
]
}
?
1 2 3
C G B F A D
db.coll.createIndex({field:1, sortField:1})
143. #MDBLocal
Operator Type Check - $in
.find({field:{$in:[1,3]}})
.sort({sortField:1})
"indexBounds" : {
"field" : [
"[1.0, 1.0]",
"[3.0, 3.0]"
],
"sortField" : [
"[MinKey, MaxKey]"
]
}
?
1 2 3
C G B F A D
db.coll.createIndex({field:1, sortField:1})
144. #MDBLocal
Operator Type Check - $in
.find({field:{$in:[1,3]}})
.sort({sortField:1})
"indexBounds" : {
"field" : [
"[1.0, 1.0]",
"[3.0, 3.0]"
],
"sortField" : [
"[MinKey, MaxKey]"
]
}
?
1 2 3
C G B F A D
db.coll.createIndex({field:1, sortField:1})
145. #MDBLocal
Operator Type Check - $in
.find({field:{$in:[1,3]}})
.sort({sortField:1})
"indexBounds" : {
"field" : [
"[1.0, 1.0]",
"[3.0, 3.0]"
],
"sortField" : [
"[MinKey, MaxKey]"
]
}
?
1 2 3
C G B F A D
BS
db.coll.createIndex({field:1, sortField:1})
146. #MDBLocal
Operator Type Check - $in
.find({field:{$in:[1,3]}})
.sort({sortField:1})
"indexBounds" : {
"field" : [
"[1.0, 1.0]",
"[3.0, 3.0]"
],
"sortField" : [
"[MinKey, MaxKey]"
]
}
?
1 2 3
C G B F A D
BS
db.coll.createIndex({field:1, sortField:1})
147. #MDBLocal
Operator Type Check - $in
.find({field:{$in:[1,3]}})
.sort({sortField:1})
?
1 2 3
C G B F A D
db.coll.createIndex({field:1, sortField:1})
148. #MDBLocal
Operator Type Check - $in
.find({field:{$in:[1,3]}}).sort({sortField:1}) ?
1 2 3
C G B F A D
db.coll.createIndex({field:1, sortField:1})
149. #MDBLocal
Operator Type Check - $in
.find({field:{$in:[1,3]}}).sort({sortField:1})
"indexBounds" : {
"field" : [
"[1.0, 1.0]",
],
"sortField" : [
"[MinKey, MaxKey]"
]
}
?
1 2 3
C G B F A D
db.coll.createIndex({field:1, sortField:1})
150. #MDBLocal
Operator Type Check - $in
.find({field:{$in:[1,3]}}).sort({sortField:1})
"indexBounds" : {
"field" : [
"[1.0, 1.0]",
],
"sortField" : [
"[MinKey, MaxKey]"
]
}
?
1 2 3
C G B F A D
db.coll.createIndex({field:1, sortField:1})
151. #MDBLocal
Operator Type Check - $in
.find({field:{$in:[1,3]}}).sort({sortField:1})
"indexBounds" : {
"field" : [
"[1.0, 1.0]",
],
"sortField" : [
"[MinKey, MaxKey]"
]
}
?
1 2 3
C G B F A D
db.coll.createIndex({field:1, sortField:1})
152. #MDBLocal
Operator Type Check - $in
.find({field:{$in:[1,3]}}).sort({sortField:1})
"indexBounds" : {
"field" : [
"[1.0, 1.0]",
],
"sortField" : [
"[MinKey, MaxKey]"
]
}
?
1 2 3
C G B F A D
db.coll.createIndex({field:1, sortField:1})
153. #MDBLocal
Operator Type Check - $in
.find({field:{$in:[1,3]}}).sort({sortField:1})
"indexBounds" : {
"field" : [
"[1.0, 1.0]",
],
"sortField" : [
"[MinKey, MaxKey]"
]
}
?
1 2 3
C G B F A D
db.coll.createIndex({field:1, sortField:1})
154. #MDBLocal
Operator Type Check - $in
.find({field:{$in:[1,3]}}).sort({sortField:1})
"indexBounds" : {
"field" : [
"[1.0, 1.0]",
],
"sortField" : [
"[MinKey, MaxKey]"
]
}
"indexBounds" : {
"field" : [
"[3.0, 3.0]",
],
"sortField" : [
"[MinKey, MaxKey]"
]
}
?
1 2 3
C G B F A D
db.coll.createIndex({field:1, sortField:1})
155. #MDBLocal
Operator Type Check - $in
.find({field:{$in:[1,3]}}).sort({sortField:1})
"indexBounds" : {
"field" : [
"[1.0, 1.0]",
],
"sortField" : [
"[MinKey, MaxKey]"
]
}
"indexBounds" : {
"field" : [
"[3.0, 3.0]",
],
"sortField" : [
"[MinKey, MaxKey]"
]
}
?
1 2 3
C G B F A D
?
1 2 3
C G B F A D
db.coll.createIndex({field:1, sortField:1})
156. #MDBLocal
Operator Type Check - $in
.find({field:{$in:[1,3]}}).sort({sortField:1})
"indexBounds" : {
"field" : [
"[1.0, 1.0]",
],
"sortField" : [
"[MinKey, MaxKey]"
]
}
"indexBounds" : {
"field" : [
"[3.0, 3.0]",
],
"sortField" : [
"[MinKey, MaxKey]"
]
}
?
1 2 3
C G B F A D
?
1 2 3
C G B F A D
db.coll.createIndex({field:1, sortField:1})
157. #MDBLocal
Operator Type Check - $in
.find({field:{$in:[1,3]}}).sort({sortField:1})
"indexBounds" : {
"field" : [
"[1.0, 1.0]",
],
"sortField" : [
"[MinKey, MaxKey]"
]
}
"indexBounds" : {
"field" : [
"[3.0, 3.0]",
],
"sortField" : [
"[MinKey, MaxKey]"
]
}
?
1 2 3
C G B F A D
?
1 2 3
C G B F A D
db.coll.createIndex({field:1, sortField:1})
158. #MDBLocal
Operator Type Check - $in
.find({field:{$in:[1,3]}}).sort({sortField:1})
"indexBounds" : {
"field" : [
"[1.0, 1.0]",
],
"sortField" : [
"[MinKey, MaxKey]"
]
}
"indexBounds" : {
"field" : [
"[3.0, 3.0]",
],
"sortField" : [
"[MinKey, MaxKey]"
]
}
?
1 2 3
C G B F A D
?
1 2 3
C G B F A D
{2} {4} {1} {3}
Sort Merge
db.coll.createIndex({field:1, sortField:1})
161. #MDBLocal
Consecutive Index Keys
Given Indexes:
{gamertag:1, date:1, game:1}
{gamertag:1, game:1, date:1}
Which one is “better”?
It depends on the query shape(s)!
Consider the query:
.find({
gamertag:"Ace",
game: "Halo”
})
180. #MDBLocal
Exceptions
Is the E-S-R “rule” always optimal?
Nope.
Consider the following query:
db.games
.find({gamertag:"Ace", date:{$gte:2019}})
.sort({score:1})
207. #MDBLocal
E-S-R Guidance
A good starting place applicable to
most use cases
Place keys in the following order:
Equality first
Sort next
Range last
Remember:
• Indexing is important from day one,
things get slow as the data grows
• Some operators may be range
instead of equality
• Having consecutive keys used in
the index is important
• Specifics about your data set may
need a different approach
M201: Performance in MongoDB
university.mongodb.com