Narrator: One of the most important steps
in building data intensive apps is to actually model
all this data in MongoDB.
And so that's what we're gonna talk about in this lecture.
So it's really crucial that you follow it
through even at first its a lot to take in. All right.
Anyway, lets now get started.
Now, data modeling is probably a very new concept to you.
So before we start;
lets make clear what we're actually gonna talk about.
So, data modeling is the process of taking unstructured data
generated by a real world scenario
and then structure it into a logical data model
in a database.
And we do that according to a set of criteria
which we're gonna learn about in this video.
For example; lets say that we want to design
an online shop data model.
There will be initially a ton of unstructured data
that we know we need.
Right.
Stuff like products, categories,
customer's orders, shopping carts, suppliers.
And so on and so forth.
Our goal with data modeling is to then structure
this data into a logical way.
Reflecting the real-world relationships
that exists between some of these data sets.
A bit like you can see in this example.
And this is of course just a kind of imaginary situation
but you get the idea.
Right.
Now, many backend developers say that data modeling
is where we have to think the most.
That its the most demanding part of building
an entire application.
Because it really is not always straight-forward.
And sometimes there are simply no right answers.
So not just one unique correct way of structuring the data.
But anyway I will do my best to lay down the process
in this video.
And for that we're gonna go through four steps.
So in the first step; we learned about how to identify
different types of relationships between data.
Then we're gonna understand the difference
between referencing or normalization
and embedding or denormalization.
In the next and most important step;
I will show you my framework for deciding
whether we should embed documents
or reference to other documents
based on a couple of different factors.
Also, we have to quickly talk about
different types of referencing.
Because that's important if that is the type of design
that we choose for our data.
So this is gonna be in fact a quite theoretical lecture.
But also an absolutely essential one for your progress
as a back-end developer.
Because the way we design data so the way we model our data
can make or break our entire application.
And there will be a lot of examples along the way
to make this process easier.
All right.
And the first thing that we are gonna talk about
is the different types of relationships
that can exist between data.
So there are three big types of relationships.
One to one, one to many, and many to many.
And I'm gonna use a movie application
as an example in this slide.
Okay?
So first a one to one relationship
between data is basically
when one field can only have one value.
So in our movie application example; one movie only ever
have one name.
And so this is a simple example
of a one to one relationship.
But these relationships are not really that important
in terms of data modeling.
Now the most important relationships
are the one to many relationships.
And they are so important that in MongoDB
we actually distinguish between three types
of one to many relationships.
One to a few, one to many, and one to a ton or to a million
or something like that. So the difference here is based
on the relative amount of the many. All right.
So an example to a one to a few relationship is that
one movie can win many awards but actually just a few.
So movie is not gonna win a thousand awards
but it can win some.
And so this is a typical one to few relationship.
So you see that in general a one to many relationship
means that one document can relate to many other documents.
Now this might look a bit abstract without the JSON data
but that's actually the purpose here.
I just wanna show you a conceptual overview
of these different types of relationships.
Anyway, any one to many relationship
one document can relate to hundreds or thousands
of other documents.
For example; one movie can have thousands of reviews
in our application.
And so this not really a one to few
but one to many relationship. Okay?
And finally we have the one to ton relationship.
Imagine we wanted to implement some logging functionality
in our app. So basically to know exactly what's going on
on our server.
This logs can then easily grow to millions of documents.
And so this is a very typical example
of a one to tons a relationship.
And the difference between many and a ton is of course
a bit fuzzy but just think that if something can grow
almost to infinity then its definitely
a one to a ton relationship.
So again the one to many relationships
are the most important ones to know.
By the way; in relational databases
there is just one to many without quantifying
how much that many actually is.
In MongoDB databases though
it is an extremely important difference.
Because its one of the factors that we're gonna use
to decide if we should denormalize or normalize data
as you will learn a bit later.
Anyway, the less type of relationship is the many to many
where one movie can have many actors.
But at the same time one actor can play in many movies.
And so here the relationship basically
goes in both directions.
Where before in the other types
it was only in one direction.
For example one movie can have many reviews
but one specific is only for that one movie. Right.
And the same goes for the awards.
So one specific award like for the best actor
goes to only one movie not multiple ones.
But with movies and actors it is indeed different.
So again one movie stars many actors
but one actor plays many movies
and so its a many to many relationship.
Okay.
So keep all this in mind as we now move forward
in this lecture.
And probably the most important aspect that we need to learn
about MongoDB databases is referencing
and embedding two datasets.
And we actually already talked a little bit
about this before but lets review it here
and go a bit deeper also.
So each time we have two related datasets
we can either represent that related data in a reference
or normalized form or in an embedded or denormalized form.
And I keep using the two related terms together
like referencing and normalizing
because you will see them both being used
and so its important that you know all of them.
Anyway, in the referenced form we keep two related
datasets and all the documents separated.
So again all the data is nicely separated
which is exactly what normalized means.
So continuing, the movie database example from before
we would have one movie document and one actor document
for each actor. Now how would we then make the connection
between movie and the actors so that later in our app
we can show which actors played in a particular movie.
Because if they are all completely different document
the movie has no way of knowing about the actors. Right.
Well that's where the IDs come in.
So we use the actor IDs in order to create references
on the movie document.
Effectively connecting movies with actors.
So you see that in a movie document we have an array
where we stored the IDs of all the actors
so that when we request data about a certain a movie
we can easily identify its actors. Does that make sense?
Now this type of referencing is called child referencing
because its the parent in this case the movie
who references its children. In this case the actors.
So we're really creating some sort of hierarchy here. Right.
Now there is also parent referencing
and we are gonna talk about that a bit later.
And by the way in relational databases; all data is always
represented in normalized form like this.
But in a no sequel database like MongoDB
we can denormalize data into a denormalized form
simply by embedding the related document
right into the main document.
So now we have all the relevant data about actors
right inside in one main movie document without the need
for separate documents, collections, and IDs.
So again, if we choose to denormalize or to embed our data
we will have one main document containing all the main data
as well as the related data. All right.
And the result of this is that our application
will need to fewer queries to the database.
Because we can get all the data
about movies and actors all at the same time
which will of course increase our performance.
Now the downside here is of course
that we can't really query the embedded data on its own.
And so if that's a requirement for the application
you would have to choose a normalized design
and since we're talking about pros and cons
of the denormalized form; lets do the same
about the normalized design.
And basically its kind of the opposite
of what we just talked about.
So there is an improvement in performance
when we often need to query the related data on it's own
because we then can just query the data that we need
and not always movies and actors together.
But on the other hand; when we need to actually query
movies and actors together we then are gonna need
many queries to the database.
So first the query for the movie and then from there
we will also need a query for the actor
and that is of course works for performance.
So when designing your database; this is the kind of stuff
that you need to keep in mind. All right.
And now just as a side note; we could of course begin
our thought process with denormlized data
and then come to the conclusion
that its best to actually normalize the data.
So when thinking about our data model
this way of organizing data works of course in both ways.
Now, how do we actually decide if we should
normalize or denormalize the data?
Well that's exactly what we're gonna learn next.
So when we have two related datasets; we have to decide
if we're gonna embed the datasets or if we're gonna
keep them separated and reference them
from one dataset to the other.
And I kind of developed this decision framework
which I'm gonna show you where we use three criteria
to take that decision.
First we look at the type of relationships
that exists between datasets.
Second we try to determine the data access pattern
of the dataset that we want to either embed or reference.
And this just means to analyze how often data is read
and written in that dataset.
Then we also look at something that I call data closeness.
And data closeness is term that I actually just made up
but what it means is how much the data is really related
and how we want to query the data from the database.
And this will make more sense
when we talk about it in a moment.
Now to actually take the decision; we need to combine
all of these three criteria
and not just use one of them in isolation.
So for example; just because criteria number one
says to embed it doesn't mean that we don't need to look
at the other two criteria.
All right and lets start with the relationship type.
So usually when we have one to few relationship
we will always embed the related dataset
into the main dataset just like we learned
in the last slide. Right.
Now in a one to many relationship; things are a bit
more fuzzy so its okay to either embed or reference.
In that case we will have to decide
according to the other two criteria.
Now on the other hand, on a one to a ton
or a many to many relationship we usually always reference
the data. That's because if we actually did embed
in this case we could quickly create way too large document.
Even potentially surpassing the maximum of 16 megabytes.
And so the solution for that is of course referencing
or normalizing the data. And as a quick example;
lets say that in our movie database example
we have around 100 images associated to each movie.
So we could say its a one to many relationship
but are we gonna embed the dataset or should we rather
reference them here. Well we don't really know.
So lets take a look at the other two criteria.
So the second one is about data access patterns
where its just a fancy description
for evaluating whether a certain dataset
is mostly written to or mostly read from.
So if the dataset that we're deciding about is mostly read
and the data is not updated a lot
then we should probably embed that dataset.
So a high read/write ratio just means
that there is a lot more reading than writing.
And a again, a dataset like that is a good candidate
for embedding.
The reason for this is that by embedding we only need
one trip to the database per query.
While for referencing we need two trips. Right.
So if we embed data that is read a lot;
in each query we save one trip to the database
making the entire process way more performant.
So I think that our movie image example
would actually be a good candidate for embedding.
Because once the 100 image are saved to the database
they are not really updated anymore
because there is not really anything to update
about an image. Right, so its all about reading
and therefore based on this criteria
we would embed the imaged documents.
Now on the other hand, if our data is updated a lot
then we should consider referencing or normalizing the data.
That's because its more work for the database engine
to update and embed a document
than a more simple standalone document.
And since our main goal is performance;
we just normalize the dataset.
In our example lets say each movie has many reviews
and each review can be marked as helpful by the user.
So each time someone clicks on this review was helpful
in our application. We need to update
the corresponding document.
And this means that the data can change all the time
and so this is a great candidate for normalizing.
Again because we don't want to be querying the movies
all the time if all we really wanna update is the reviews
by marking them as helpful.
Okay, does that make sense?
And finally the last criteria I call data closeness;
which is just like a measure for how much the data
is related. So if the two datasets really
intrinsically belong together then they should
probably be embedded into one another.
In our example; all users can have many email addresses
on their account and since they are so intrinsically
connected to the user, there is no doubt emails
should be embedded into the document.
Now if we frequently need to query both of datasets
on their own then that's a very good reason
to normalize the data into two separate datasets.
Even if they are closely related.
So imagine that in our app we have a quiz
where users have to identify a movie based on images.
This means that we're gonna query a lot of images
on their own.
So without necessarily querying for the movies themselves.
And so if we apply this third criteria;
we come to the conclusion that we should actually normalize
the image dataset. All right.
Because again if we implement this quiz functionality;
images are gonna be queried on their own all the time.
So, all of this shows that we should really look
all the three criteria together
rather than just one of them in isolation.
Because that might lead to less optimal decisions.
And I say less optimal instead of wrong
because they are not really completely right
or completely wrong ways of modeling our data.
There are no hard rules; these are just like guidelines
that you can follow to find the probably
most correct way of structuring your data.
But again, it's hard to be really really wrong.
Okay?
Now, lets say that we have chosen to normalize
our datasets.
So in other words to reference data.
Then after that we still have to choose
between three different types of referencing.
Child referencing, parent referencing
and two-way referencing.
So the first type is child referencing.
Which is the referencing type I actually showed you before.
Okay?
And lets not take the error logging example
that I mentioned earlier. Where we could potentially
have millions of locked documents.
So in child referencing; we basically keep references
to the related child documents in a parent document.
And they are usually stored in an array.
So you see that each log has an ID
and then in the app document there is that array
with all of these IDs. Right?
However, the problem here is that this array
of IDs can become very large if there are lots of children.
And this is an anti-pattern in MongoDB.
So something that we should avoid at all costs.
Also, child referencing makes it
so that parents and children are very tightly coupled.
Which is not always ideal. But that's exactly
why we have parent referencing.
So in parent referencing; it actually works
the other way around.
So here in each child document we keep a reference
to the parent element.
Therefore the name parent referencing.
In this example the app ID is 23
and so in each log there is the app field
with the 23 ID in it.
So that the child always knows its parent.
And so in this case the parent actually knows nothing
about the children.
Not who they are and not how many they are.
So, they are way more isolated and more standalone.
In that, it can sometimes be beneficial.
So which of these two types is actually better
for this data relationship.
And remember how I said that there
could be millions of logs and so lets suppose
there is two million logged documents.
In a case of child referencing, that would mean
that there are two million ID references
in the app document.
Right? Now also remember how I said that
there is 16 megabyte limit on documents.
So if we kept adding and adding these child IDs
into the array on the parent; then we would
pretty quickly hit that 16 megabytes limit
that each Bson document can hold.
Simply because that array will grow so much.
So that's not really gonna work.
Is it?
On the other hand with parent referencing
that problem is not gonna happen.
We will simply have two million locked documents
just like before but each of them holds ID of its parent.
But there is no array that will grow indefinitely
and therefore parent referencing
would be best solution here.
So the conclusion of all this is that in general
child referencing is best used
for one to a few relationships. Where we know before hand
that the array of child documents won't grow that much.
On the other hand, parent referencing is best used
for one to many and one to a ton relationships
like this one. Okay?
So again always keep in mind that one of the most
important principals of MongoDB data modeling
is that array should never be allowed to grow indefinitely.
In order to never break that 16 megabyte limit.
We also don't want to send our users an array
with thousands of IDs each time
they request a parent dataset. Okay?
So did this logic make sense to you?
Then lets move on to third type of referencing
which is two-way referencing.
And this time with the movie and actor example
I showed you when we talked about
many to many relationships. Remember that?
So again, each movie has many actors
and each actor plays in many movies.
And so that's a typical many to many relationship.
And we usually use this two-way referencing to design
many to many relationships. And it works like this;
in each movie we will keep references to all the actors
that star in that movie. So a bit like in child referencing.
However and at the same time in each actor
we also keep references to all the movies
that the actor played in.
So movies and actors are connected in both directions.
In therefore the name two-way referencing.
And this makes it really easy to search
for both movies and actors completely independently.
While also making it easy to find the actors
associated to each movie and the movies associated
to each actor.
(deep breath)
This was quite a long lecture indeed.
With a lot of new concepts and principals
and guidelines to remember.
So in order to help you with that; here goes a quick
summary and some more general guidelines that you can
take a look at when you need it.
So the most important principal is: structure your data
to match the ways that your application queries
and updates data.
Or in other words: identify the questions that arise
from your application's use cases first, and then model
your data so that the questions can get answered
in the most efficient way.
For example; when I need to query movies and actors
always together or are there scenarios where I only
query movies or only actors.
That kind of questions is what your data model
will be based on.
In general, always favor embedding unless there is a good
reason not to embed.
Especially on one to a few and one to many relationships.
Next up, a one to a ton or a many to many relationship
is usually a good reason to reference instead of embedding.
Also, favor referencing when data is updated a lot
and if you need to frequently access a dataset on its own.
Use embedding when data is mostly read but rarely updated
and when two dataset belong intrinsically together.
Don't allow arrays to grow indefinitely.
Therefore, if you want to normalize; use child referencing
for one to many relationships and parent referencing
for one to a ton relationships.
And finally use two-way referencing
for many to many relationships.
All right?
And that pretty much sums it up.
I would actually recommend you watching this video
twice if you can, just because of how important
this material really is. All right?
Anyway, see you in the next video.