Now we talked about replica sets,
now let's talk about sharding. Sometimes the two things are well kind of confused with each other,
they are actually different things.
Sharding is all about horizontal scaling,
now a quick word about scaling.
If you have a mongodb server,
here it is
and with that, I really mean a computer which runs your mongodb server and where your database
files are then stored on, if you have that server and you need more power because your application started
and everything is going good and you've got more requests coming in,
you've got more users, you've got more read and write operations, what can you do?
Well you can upgrade your server,
you can buy more CPU, more memory and put that into the server
and if you use a cloud provider, you can typically upgrade this with a single click.
Now that is of course a solution but it only gets you so far because at some point, you can't squeeze more
CPU and more memory into a single machine
and at that point, you need horizontal scaling which means you need more servers.
The issue here of course is this might sound logical but the servers
now don't duplicate the data,
they are not backups,
they split the data. So server A, the one on the left let's say stores data for the same application
as the other servers but a different chunk of it
and that is really important to understand. With sharding, you have multiple computers who all run mongodb
servers but these serverss don't work standalone but work together and split up the available
data, so the data is distributed across your shards, not replicated. And queries,
so queries where you find data but also insert, update and delete operations
therefore have to be run against all the servers or the right server because each chunk manages its data
and its range of the data
you can kind of say if you would store alphabetical data, then the first server would store A to F, the next
server would store F to L and so on
so you could split it like this
and therefore, the operations have to be forwarded to the right servers.
Now how does this work?
Well if you've got your different mongod instances, so your different servers, your different shards
and each shard here by the way can and will typically be a replica set, that's also important,
so you have multiple replica sets because each shard is also a replica set,
it's this set of nodes.
So if you have these servers and you have your client, you then have a new middleman which we haven't
used in this course,
mongos is then the executable and that's the router mongodb offers. This router is responsible for
forwarding your operations, inserts, reads and so on
to the right shards.
So the router has to find out which shard is responsible for the data you're inserting, so where this should
be stored and which shard will hold the data you want to retrieve.
For this splitting, you'll use a so-called shard key.
Now a shard key is essentially just a field that's added to every document which kind of is important
for the server to understand where this document belongs to.
Now this shard key configuration is actually something which is not trivial because you want to ensure
that you have a shard key that is evenly distributed,
you can assign your own values here, you could say the shard key is the name of my user but then you should
ensure that your usernames are roughly evenly distributed and not are old clunked at let's say names
starting with A and then you've got no names starting with M, N, O, P, Q, R, S and so on
because then you would put them all on one server which is not great.
So choosing the shard key wisely is important and also something where you can read a lot in the official
docs which is a job of your admin
but this is then an important part of mongos, of the router, finding out where an operation should go,
so which server is responsible for storing the incoming data and so on.
Now how does querying work with sharding then? You've got all your shards and then let's say you're
issuing a find query on your client. Now that reaches mongos, that server
and now there are two options.
The first option is that your find did not specify a value for the shard key, maybe you are looking for
a user named Max but your shard key is some other value,
so not the name.
Well then in your find filter, there is no information about the shard key and mongos can't know which
shard is responsible for handling this find request,
mongos does not know which shard contains your data. In such a case,
mongos has to broadcast your request to all shards and then each shard has to check am I responsible,
do I have that data
and then each shard returns its response which is either the data or no data and then mongos has to merge
all that data together and return it.
Option 2 is that
your find does contain a shard key,
let's say now your shard key is the username and you are searching for the username in your find filter
and then mongos can directly forward this to the right shard and fetch the data from there which is
of course more efficient. And that is the part that then matters for you as a developer
again, if you know you are using sharding, you should sit together with your administrators and choose
a wise shard key keep based on your application needs that is evenly distributed so that you can write
queries that use that shard key as often as possible.
This is sharding,
it's all about distributing data across servers and then setting up everything such that it can be queried
and used efficiently.
Now setting up sharding is also an advanced topic and that is then a task you'll not have to worry about.
Still just as with replica sets, when we dive into deploying a mongodb server, I will show you a solution
that allows you to easily add sharding.