Narrator: Welcome back.
In this video and the next one
I want to introduce you to the MongoDB
aggregation pipeline which is an extremely powerful
and extremely useful MongoDB framework
for data aggregation.
And the idea is that we basically
define a pipeline that all documents
from a certain collection go through
where they are processed step by step
in order to transform them into aggregated results.
For example, we can use the aggregation pipeline
in order to calculate averages
or calculating minimum and maximum values
or we can calculate distances even,
and we can really do all kinds of stuff.
It's really amazing how powerful
this aggregation pipeline is.
All right, but enough talk.
Let's now actually start using the aggregation pipeline
and I'm gonna create a new header function here
because later on we will then define a new route
and then use that function for that.
But that new route, I'm gonna create it later.
So I want to create a function here that's gonna
calculate a couple of statistics about our tours.
So I'm gonna call this one getTourStats.
Okay, so Stats for statistics,
and it's gonna be an async function
and so a vs code now automatically prefilled off this.
All right, let's create or try catch block here.
Actually, I'm gonna copy this from right up here,
and now in here I will create
a variable called stats.
Now the aggregation pipeline really is a MongoDB feature.
But Mongoose, of course, gives us access to it,
so that we can use it in the Mongoose driver, right?
So using our tour model in order to access
the tour collection, we say
tour.aggregate.
And so the aggregation pipeline is a bit
like a regular query and so using the aggregation pipeline
it's a just a bit like doing a regular query.
The difference here is that in aggregations,
as I already mentioned, we can manipulate the data
in a couple of different steps
and so let's now actually define these steps.
And for that, we pass in an array of so-called stages.
So we pass in an array, and then here we will
then have a lot of stages.
And again the documents then pass through these stages
one by one, step by step in the define sequence
as we define it here.
So each of the elements in this array
will be one of the stages.
And there are a ton of different stages
that we can choose from, but I will just tell you
the most common ones in this lecture
and also in the next one.
But anyway, I now want to take a minute
and take a quick look at the MongoDB documentation
because I think we didn't do that yet.
So MongoDB and we already know this website.
So on our MongoDB website we go here to learn,
then documentation and then in here we select
the MongoDB Manual and this documentation is huge.
It really, it has a lot of stuff in here
and if you were gonna go through it all,
well, then you'd become really a MongoDB master.
So it gives you a nice introduction to MongoDB,
you have the installation here.
You have the CRUD Operations, so a nice overview
about a lot of the stuff
that we already learned before, right?
Then you have stuff about the aggregation
and in particular, the aggregation pipeline
which we're gonna talk about now,
but what I want to really show you here
is actually this reference that we have at the bottom
because in the reference we have here for example
the operators and that's where we have
a lot of important stuff that we've been using all the time.
So let's take a look here at the query operators,
for example and here you see, we actually have
these operators that we already used.
So for example, greater than or equal,
or less than and equal, but if someday
in your own applications you need some other operator,
well, now you know where you can find them.
Now of course, it's also always very helpful
to just do a quick Google search, probably find
some stack overflow post about it,
but believe me, many times it's easier
and faster to just quickly jump into the documentation
and quickly search for whatever you're needing.
You just need to know how the documentation works
and once you know that, it's then gonna be quite easy
for you to find what you're looking for, all right?
Here you also have some logical operators
for example, the OR operator we already used.
Again, really a lot of stuff, geospatial operators
and we're gonna use some actually, later,
and yeah, really a lot of stuff.
Then what I wanted to show you after this lecture actually,
are the aggregation pipeline stages.
So here you have a lot of them
and what we're gonna use in this lecture here is match,
for example and group,
where is that?
Yeah group, that is a very important one,
and so as I mentioned, I'm gonna show you
the most important ones in this lecture and the next one.
But again, if you in some case need something else,
then just go ahead, come here to this documentation
and you will find what you're looking for.
All right?
Anyway, let's now learn how we can actually define
one of these stages, all right?
And I'm gonna start with match.
And match is basically to select
or to filter certain documents.
And so it's very simple.
It's really just like a filter object in MongoDB,
such like we've been using so many times.
So each of the stages is an object
and then here comes the name of the stage.
So this one is the match stage, all right?
And as I mentioned, it's really just a query.
And so let's say that for starters,
we only want to select documents
which have a ratings average greater or equal than 4.5.
So can you do that?
I'm just gonna give you a second to do that here.
So I hope you did it.
So ratings average, should be
greater or equal than 4.5.
And so that's it, that's the first stage.
And usually this match stage is just a preliminary stage
to then prepare for the next stages which come ahead.
So the next one is now the group stage.
So group and then in there
we need to always pass just another object.
So it looks kinda weird with all these objects,
but, yeah, you've seen that before
and MongoDB just works this way.
It's always objects, inside of objects, inside of objects.
And this group here is where the real magic happens
because as the name says,
it allows us to group documents together,
basically using accumulators.
And an accumulator is for example,
even calculating an average.
So if we have five tours, each of them has a rating,
we can then calculate the average rating using group.
And so let's do exactly that right here.
Now the first thing, is we always need to specify
is the id because this is where we're gonna specify
what we want to group by.
For now, we say null here because we want
to have everything in one group so that we can calculate
the statistics for all of the tours together
and not separate it by groups.
We will, a bit later then also group by different stuff,
for example we can group by the difficulty
and we can then, for example calculate the average
for the easy tours, the average for the medium tours
and the average for the difficult tours.
So again, we can group by one of our fields
and that field, we are gonna specify in here,
but for now, as I said, we want to calculate these averages
for all the tours together in one big group.
So in that case we say _id and set it to null.
Now let's actually calculate the average rating.
In order to do that, we simply specify a new field,
so let's simply call it the average rating,
so like this, and this will be
well, the average,
which is yet another MongoDB operator, so this one here.
You will find it in the reference if you look it up.
So this is a mathematical operator calculating
the average and now the name of the field.
And again, I know that this is gonna look very weird,
but in order to specify the field which we want
to calculate the average from, we need to use
the dollar sign, but in quotes here
and then the name of the field.
So ratings average in this case.
And let's also calculate the average price.
So average price, and again, did it wrong here,
and so
the average and then again,
the name of the field and in quotes
and with this dollar sign.
Right, so it looks kind of weird again,
but this is just the way
that the aggregation pipeline works.
So we have the average price.
Let's actually also calculate the minimum price,
so the smallest price, and the largest price.
So minPrice
and from now on, it's gonna all look kind of similar,
so now we use the min operator
and then again the field name like this.
And a comma here and let's now just duplicate this one
in order to calculate the maximum price.
And now, let's actually take a look
at what we already got at this point.
So we now need to actually send out this response,
and so again, I'm just gonna go head
and grab some of these already done responses here.
So the data that we want to send out is called stats.
And now all we need to do is actually add a new route
here in our tour routes and again,
I'm gonna edit here, right at the top.
So router.route
/gettour, or actually just tour-stats, just like this
because the get is already defined in the HTTP method.
So no need to repeat it in the name.
So tourController.getTourStats.
Give it a safe and let's actually test it out now.
So I'm really exciting to test it out now, actually
to see if it works because again it's really an amazing
and really powerful tool
which allows you to do so much stuff.
So tour-stats, send it
and that's not really the result
that we were waiting for, so that's just the pipeline
that we defined, but I already know why that is.
So we did not await the result.
So this, basically, just like a normal query
is gonna return an aggregate object.
So .find is gonna return a query, and .aggregate
is gonna return an aggregate object.
And then only when we await it,
it actually comes back with the result.
So that's also why we defined this function here
as an async function, so that we can then use await
in there and so this is the right place.
Let's try it again.
And here we go!
So great!
We have the average rating of all our tours,
we have the average price which you see is
1563 and then the minimum and the maximum, right?
And from our data, we can actually confirm
that this is true.
So great!
Really, really great.
Now, one more thing that I want to do here
is to actually calculate the total number
of ratings that we have and also the total number of tours.
So we have the average rating here
and let's actually do it before,
so numRatings,
like this
and you can probably guess
that this one is called sum and then the ratingsAverage,
or actually not ratingsAverage, but ratingsQuantity, right?
So that's where the number of ratings is stored
and so the number of ratings, the total,
will be the sum of all of these together.
And now, the last one, is the number of tours
and that one is a bit trickier,
and so that's a nice one to show you here.
So we still want to sum, so to add everything together,
basically, so we still use sum,
but what are we gonna add together?
Well, we basically add one for each document,
and so we say 1, and that's it.
So basically for each of the document
that's gonna go through this pipeline,
one will be added to this num counter.
Let's call it numTours.
And so, yeah, let's test it out now, again.
And indeed we get our nine tours
and we already know that we have nine.
So these are all the statistics for all the tours together,
but let's now take it to the next level.
As I said before, we can now group our results
for different fields.
And let's actually start out with the difficulty.
And so that's quite similar to specifying the fields
down there which is simply the dollar sign,
and then the name of the field.
So $difficulty, give it a safe and that's it.
Send it here, and now we have the statistics
all defined for each of these three difficulties.
So that's really, really amazing.
So this is the part that blows my mind.
It's absolutely fantastic.
I mean, you can start to see all the kinds of stuff,
all the kinds of data manipulation
that you can do using this pipeline, right?
So we really can now start to analyze stuff,
for example, we see that the easiest tours
are the ones who get the poorest ratings
with a 4.6, while the medium tours
are the ones who get the highest ratings.
We can also see that the difficult tours
are the most expensive ones with an average price
of 1997, while the easiest ones are also the cheapest ones
with a price of 1272, right?
We also see that the easiest tours
are the ones which we have the most,
so four easy ones and only two difficult ones, right?
So we can use this, of course,
to get all kinds of insights into our data.
Let's comment this one out here
and let's just try another one,
for example, by ratings, why not,
let's just try out, see what we get.
So for example, we have two tours,
so basically with a rating of 4.5,
we have two tours with 4.7, one tour with 4.8,
and three tours with 4.9.
Oh, and also down here, one tour with 4.6,
and so for example, one insight that we can get here
is that the very expensive tours with this average price
of 1997 get extremely good ratings.
So all kinds of stuff, really that we can do here.
Let's get rid of this one, let's add this one back,
and we can now actually even,
do some operations with this one.
So just for fun, let's put this difficulty to uppercase.
So this of course, is gonna be another object here
and then here in front is the operator.
And so the operator in this case, is called $toUpper.
So just, yet another MongoDB operator.
So check it out, and now it's spelled uppercase.
Great!
So we have the group stage here which is quite complete.
So let's try another one which is a sort stage.
So another object in our array for yet another stage
and this one is called sort.
So sort and then in there, we need yet another object,
and it kind of auto-completed this one, I don't know why,
and so here we can now specify which field we want
to sort this by and let's actually use the average price.
And now here in the sorting we actually need to use
the field names that we specified up here in the group.
We can no longer use the old names because at this point
they are already gone.
They no longer exist.
So at this point, in the aggregation pipeline,
we really already have these results.
So these are now our documents basically.
So if you want to sort by the average price,
then this is the field name we gotta use.
So we can say, the average price
and then we can say 1 for ascending.
So let's try that out.
And yeah, indeed.
That's the lowest one and that's the highest one.
Great!
So we did a lot of stuff.
Let me just show you that we can also repeat stages.
So let's actually do another match here
and this one is really just to show you
that we actually can repeat stages
and I also want to show you another operator.
So let's now select by id and remember that id
is now the difficulty, right?
So we just specified that up here.
So we want the id and now a new operator
that we didn't use yet.
We want it to be not equal to, and let's say easy.
And so, just like this we're gonna select
all the documents that are not easy.
So in this case, the one that says
medium and difficult, right?
So basically, excluding the one that says easy, right?
So again, it's not really useful here
because that will take away a lot of our meaningful data,
but just to show you that we can of course,
match multiple times.
So in this case, we matched once before we actually
did the group and then we matched once we were ready,
doing the grouping.
Now, let's actually keep this here,
out of our aggregation pipeline,
but I'm gonna leave it here,
just so that you keep it as a reference.
Give it another save, give it another try,
and so these are our results
for getting our tour statistics.
So, really nice, really great.
I hope you enjoyed this video
and in the next one, we're actually gonna keep using
the aggregation pipeline to do some more calculations.
And to show you some more different stages.
So keep tuned for that one.