Jonas: Welcome back.
So let's now talk a little bit
about read performance in MongoDB,
why something called indexes are so important,
and how we can actually create them ourselves.
And I want to start this demonstration about indexes
by firing off a simple query on all our tours.
So let's come here to get all tours
and I will also filter
for a price less than 1,000.
Okay and so let's see.
Yeah, so we get three results back, all right.
And that's important to keep in mind
for what I'm gonna show you next which is
that we can actually also get a couple of statistics
about the query itself.
So let's go here basically to the handler function.
And so this remember right now in the handler factory.
Right, so it's this,
I think it's at the bottom.
Yeah so it's this, get all factory function
that will create this handler that is called
for the query that we just performed.
And so here on the query,
I'll actually now add an explain method.
Okay so after the query we will then call explain all right.
And so let's take a look at that.
And so we now get a completely different result,
which is basically these statistics.
Now there's a lot of stuff in here.
But what I'm really interested
in is these execution statistics, okay.
So you can see here that the number of documents
that were returned were three.
And so that's exactly the result that we got before.
So before doing the explain right.
But what's really important to note here is
that the number of documents
that were examined is nine, okay.
And so this means that MongoDB had to examine,
so basically to scan all of the nine documents
in order to find the correct three ones.
So the three ones that match the query okay.
And so that's not efficient at all right?
Now of course at this scale
with only nine documents it makes absolutely no difference.
But if we had hundreds of thousands,
or even millions of documents here,
then this would significantly affect
the read performance of this query.
So again, it's not going to be the case in this application,
but it might be in an app that you will build someday.
And so you really need to learn about indexes.
Because with indexes, we will be able
to kind of solve this problem.
So we can create indexes on specific fields in a collection.
For example Mongo automatically creates an index
on the ID field by default.
And let's actually see that in Compass okay.
For example in all tours, we have here the indexes tab.
So up until this point we only ever were here
in the documents tab,
but as you see we have a lot of different stuff here
and one of them is the indexes.
And so again you see that by default we have an ID index.
And this ID index is then basically an ordered list
of all the IDs that get stored somewhere
outside of the collection okay.
And you can actually see a size here,
that it has 36 kilobytes, this index all right.
And this index is extremely useful.
Because whenever documents are queried
by the ID MongoDB will search that ordered index
instead of searching through the whole collection
and look at all the documents one by one,
which is of course much slower.
So again without an index Mongo has to look
at each document one by one.
But with an index on the field that we are querying for,
this process becomes much more efficient.
So that is pretty smart, right?
And of course, we can set our own indexes
on fields that we query very often.
So let's actually do that with the price field
that we just queried for before
because I believe that it is one of the most important
that people will query for, okay.
And so this is how it works.
So we need to go to the tour model right.
And then let's do it right here after this inner declaration
and we say tourschema.index okay.
And then an object with the name of the field
and remember how I said we were gonna set the index
on the price and then either a one or a minus one.
And a one means that we're sorting the price index
in an ascending order,
while the minus one stands for descending order okay.
And there are actually other types of indexes as well,
like for text or for geospatial data,
but we will see that a bit later.
Okay so let's give it a save now
and try our query again.
And actually I will do it a couple of times here
to make sure that the index is really set.
Because sometimes it is not set right away.
But let's now take a look here.
And so indeed we get still our number of returned at three
but this time the number of documents that were examined,
so that were scanned, were also only three.
And so that proves that with this index,
we basically achieved exactly what we wanted.
So before we had to scan through all of the nine documents
and now the engine only needs to scan the three documents
that are actually also returned.
And again because their prices
are now ordered in that index.
And so that makes it much easier and much faster
for the MongoDB engine to find them.
And so this is of course a huge performance gain.
Let's now also check out Compass here,
and actually let's reload the entire database,
and now it should actually be here
but for some reason it's not.
Let's maybe try to reload the documents.
Maybe it appears then.
Not really, let's also analyze the schema,
and that's something that we're gonna talk
about a bit later.
But still as you see we still only have these two indexes.
But that doesn't matter at all because we already know
that the index is actually working right?
So it's completely normal
that sometimes this takes some time to update.
Now another thing that you might notice here is
how this ID index that we talked
about earlier says unique here okay.
And so unique is also a property
that we can give to indexes.
And this is actually the reason
why the IDs have always to be unique.
So simply because the index
of the ID has this unique property.
You probably also noticed that there is an index
for the name here right?
But we didn't actually create that manually ourselves right?
So can you guess why it is here?
Well it is because in our schema definition,
we set the name field to be unique.
And so what Mongos then does behind the scenes
in order to ensure the uniqueness of this field is
to create a unique index for it, all right.
And so because of that, not only the ID
but also the name always have to be unique.
Okay and this is great already.
So when all we ever do is to just query
for one single field alone,
then a single field index is perfect
because remember the index that we just set
before is called a single field index.
Not sure if I mentioned it back then but I think I did.
But anyway, if we sometimes query for that field
but combined with another one,
then it's actually more efficient
to create a compound index.
So one with two fields and not just one.
So let's create a query for that just to illustrate it.
And so another field that I think is going
to be queried for all the time is the ratings average.
So ratings average, I think that's how it's called,
and let's say that we want greater or equal than 4.7 okay.
Let's send that query.
And let's see how many results we get.
Where is that?
Yeah here.
So the number of results,
so the number of documents that are returned,
so that match this query is two.
But we still had to examine three documents.
And again, at this scale of course,
that doesn't make any difference.
But as you understand, this is just an example.
And so now we want to fix the situation as well
and for that we're gonna use a compound index.
So let's go back here.
Comment this one out.
Or actually first duplicate it and then comment.
And so it's actually very simple.
All we need to do is to add here the other field.
So ratings average and let's put this one
in the ascending order.
Or actually, that's the descending order all right.
So let's give this a save.
Let's go back here.
And again, I'm going to do it a couple of times here okay.
And let's see our results.
And so now we get the result that we wanted.
So only two documents were scanned in order
to find the two documents that we were actually looking for.
Perfect and actually this compound index
that we just created is also going to work when the query
for just one of these two fields here individually,
so price or ratings average.
So when we create a compound index like this,
we do not have to then create one individual
for each of the fields as well okay.
I just want to check out how it looks in Compass now.
But well it still looks the same.
But again, not really important.
One thing that we can still see here
and which is pretty interesting is
that actually the size of these indexes.
So 72 kilobytes is actually way bigger
than the total size of all the documents combined,
which is only 14 kilobyte right?
And so basically these indexes that we create
to search the documents take up a lot more space
than the documents themselves.
But again that's just because we're operating
on such a small scale in this example.
And so that's not really relevant okay.
But it's still important to talk about this
because actually this leads me to our next question.
And that question is,
how do we decide which field we actually need to index?
And why don't we set indexes on all the fields?
Well we kind of used the strategy that I used
to set the indexes on the price and on the average rating.
So basically we need to carefully study the access patterns
of our application in order to figure out
which fields are queried the most
and then set the indexes for these fields.
For example, I'm not setting an index here on the group size
because I don't really believe
that many people will query for that parameter,
and so I don't need to create an index there.
Because we really do not want to overdo it with indexes.
So we don't want to blindly set indexes on all the fields
and then hope for the best basically.
And the reason for that is
that each index actually uses resources,
so as you can actually see here right.
And also, each index needs to be updated each time
that the underlying collection is updated.
So if you have a collection with a high write-read ratio,
so a collection that is mostly written to,
then it would make absolutely no sense
to create an index on any field in this collection
because the cost of always updating the index
and keeping it in memory clearly outweighs the benefit
of having the index in the first place
if we rarely have searches,
so have queries, for that collection.
So in summary, when deciding whether
to index a certain field or not,
we must kind of balance the frequency
of queries using that exact field
with the cost of maintaining this index,
and also with the read-write pattern of the resource.
However, just like it is with data modeling,
there are not really hard rules here.
So it's all a bit fuzzy
and you really need some experimentation
and also experience to get it right, all right.
But whatever you do, please don't just ignore indexing.
Because even if it's not perfect,
it will always have a huge benefit for your application.
All right, and that's actually all I had
to tell you about indexes.
There's just one more index that I actually want to set here
which is for the tour slug okay.
Because later on we will actually want
to use the unique slug to query for tours.
So meaning that the slug will then probably
become the most queried field.
And so it makes all the sense
to also have an index for that one.
So tourschema.index
and slug one.
And most times the one or minus one is not that important.
Okay now I'm really curious
to try to see this here in Compass.
So what I'm gonna do is to try to connect
to the database again here.
And so I can just get that here from the most recent ones.
Click, connect, and then we should get connected
to the same database.
So natours, tours, let's come to our indexes.
And now here we go.
So now we actually have our indexes right here.
So let's make this window bigger
and take a look at what we got.
So we have our slug here all right.
We have the price, which is the first one that we set.
And then we also have the price and ratings average,
which is compound and you also see here
that this one here is ascending and this one is descending
because it has the minus one.
And another thing you might notice is
that actually we no longer
have this price index in our code.
But it's still here.
And so it's not enough to remove the index from our code,
we really need to delete it from the database itself right.
So remember we had it here in the beginning
and then we commented it out
and created this new compound index,
but again the index is stand still here.
But since we actually no longer need it,
we can just go ahead and delete it okay.
Now we need to type the name just to make sure
that we do not do any mistakes.
And so here we go, great.
So that is the power of indexes.
They really can make our read performance
on databases much, much better.
And so in your own applications
you should really never ignore them.
And before we finish let's actually take back
that explain method that we added here
in our handler function.
And actually just as a reference,
I will leave it here like this okay.
Give it a save.
Back in post menu.
Let's try that now.
And indeed, it's back to working.
Okay and now that's really it.