1
00:00:02,200 --> 00:00:03,970
Now we talked about replica sets,

2
00:00:03,970 --> 00:00:11,020
now let's talk about sharding. Sometimes the two things are well kind of confused with each other,

3
00:00:11,020 --> 00:00:13,040
they are actually different things.

4
00:00:13,060 --> 00:00:16,310
Sharding is all about horizontal scaling,

5
00:00:16,300 --> 00:00:18,310
now a quick word about scaling.

6
00:00:18,340 --> 00:00:19,970
If you have a mongodb server,

7
00:00:19,990 --> 00:00:20,490
here it is

8
00:00:20,500 --> 00:00:25,600
and with that, I really mean a computer which runs your mongodb server and where your database

9
00:00:25,600 --> 00:00:31,740
files are then stored on, if you have that server and you need more power because your application started

10
00:00:31,750 --> 00:00:34,540
and everything is going good and you've got more requests coming in,

11
00:00:34,540 --> 00:00:39,200
you've got more users, you've got more read and write operations, what can you do?

12
00:00:39,460 --> 00:00:41,340
Well you can upgrade your server,

13
00:00:41,380 --> 00:00:46,740
you can buy more CPU, more memory and put that into the server

14
00:00:46,840 --> 00:00:51,460
and if you use a cloud provider, you can typically upgrade this with a single click.

15
00:00:51,550 --> 00:00:57,730
Now that is of course a solution but it only gets you so far because at some point, you can't squeeze more

16
00:00:57,730 --> 00:01:00,780
CPU and more memory into a single machine

17
00:01:01,270 --> 00:01:06,770
and at that point, you need horizontal scaling which means you need more servers.

18
00:01:06,970 --> 00:01:12,020
The issue here of course is this might sound logical but the servers

19
00:01:12,190 --> 00:01:14,350
now don't duplicate the data,

20
00:01:14,470 --> 00:01:15,970
they are not backups,

21
00:01:16,000 --> 00:01:23,170
they split the data. So server A, the one on the left let's say stores data for the same application

22
00:01:23,170 --> 00:01:26,510
as the other servers but a different chunk of it

23
00:01:26,740 --> 00:01:32,350
and that is really important to understand. With sharding, you have multiple computers who all run mongodb

24
00:01:32,350 --> 00:01:38,140
servers but these serverss don't work standalone but work together and split up the available

25
00:01:38,140 --> 00:01:44,350
data, so the data is distributed across your shards, not replicated. And queries,

26
00:01:44,380 --> 00:01:49,690
so queries where you find data but also insert, update and delete operations

27
00:01:49,700 --> 00:01:57,730
therefore have to be run against all the servers or the right server because each chunk manages its data

28
00:01:58,090 --> 00:01:59,790
and its range of the data

29
00:01:59,910 --> 00:02:06,220
you can kind of say if you would store alphabetical data, then the first server would store A to F, the next

30
00:02:06,220 --> 00:02:08,930
server would store F to L and so on

31
00:02:09,790 --> 00:02:11,390
so you could split it like this

32
00:02:11,390 --> 00:02:16,130
and therefore, the operations have to be forwarded to the right servers.

33
00:02:16,130 --> 00:02:17,650
Now how does this work?

34
00:02:17,760 --> 00:02:22,940
Well if you've got your different mongod instances, so your different servers, your different shards

35
00:02:23,270 --> 00:02:28,760
and each shard here by the way can and will typically be a replica set, that's also important,

36
00:02:28,850 --> 00:02:33,580
so you have multiple replica sets because each shard is also a replica set,

37
00:02:33,590 --> 00:02:35,890
it's this set of nodes.

38
00:02:35,990 --> 00:02:41,570
So if you have these servers and you have your client, you then have a new middleman which we haven't

39
00:02:41,570 --> 00:02:42,690
used in this course,

40
00:02:42,740 --> 00:02:49,520
mongos is then the executable and that's the router mongodb offers. This router is responsible for

41
00:02:49,520 --> 00:02:53,310
forwarding your operations, inserts, reads and so on

42
00:02:53,390 --> 00:02:55,490
to the right shards.

43
00:02:55,570 --> 00:03:01,250
So the router has to find out which shard is responsible for the data you're inserting, so where this should

44
00:03:01,250 --> 00:03:04,860
be stored and which shard will hold the data you want to retrieve.

45
00:03:05,000 --> 00:03:09,160
For this splitting, you'll use a so-called shard key.

46
00:03:09,170 --> 00:03:14,630
Now a shard key is essentially just a field that's added to every document which kind of is important

47
00:03:14,630 --> 00:03:17,980
for the server to understand where this document belongs to.

48
00:03:18,110 --> 00:03:25,700
Now this shard key configuration is actually something which is not trivial because you want to ensure

49
00:03:25,700 --> 00:03:28,520
that you have a shard key that is evenly distributed,

50
00:03:28,520 --> 00:03:33,770
you can assign your own values here, you could say the shard key is the name of my user but then you should

51
00:03:33,770 --> 00:03:41,570
ensure that your usernames are roughly evenly distributed and not are old clunked at let's say names

52
00:03:41,630 --> 00:03:46,760
starting with A and then you've got no names starting with M, N, O, P, Q, R, S and so on

53
00:03:46,880 --> 00:03:50,560
because then you would put them all on one server which is not great.

54
00:03:50,720 --> 00:03:57,200
So choosing the shard key wisely is important and also something where you can read a lot in the official

55
00:03:57,200 --> 00:03:59,750
docs which is a job of your admin

56
00:04:00,020 --> 00:04:06,760
but this is then an important part of mongos, of the router, finding out where an operation should go,

57
00:04:06,770 --> 00:04:11,470
so which server is responsible for storing the incoming data and so on.

58
00:04:11,510 --> 00:04:16,880
Now how does querying work with sharding then? You've got all your shards and then let's say you're

59
00:04:16,880 --> 00:04:22,130
issuing a find query on your client. Now that reaches mongos, that server

60
00:04:22,130 --> 00:04:23,630
and now there are two options.

61
00:04:23,630 --> 00:04:28,910
The first option is that your find did not specify a value for the shard key, maybe you are looking for

62
00:04:28,910 --> 00:04:32,550
a user named Max but your shard key is some other value,

63
00:04:32,570 --> 00:04:33,610
so not the name.

64
00:04:33,860 --> 00:04:40,160
Well then in your find filter, there is no information about the shard key and mongos can't know which

65
00:04:40,220 --> 00:04:43,350
shard is responsible for handling this find request,

66
00:04:43,370 --> 00:04:48,680
mongos does not know which shard contains your data. In such a case,

67
00:04:48,740 --> 00:04:57,470
mongos has to broadcast your request to all shards and then each shard has to check am I responsible,

68
00:04:57,470 --> 00:04:58,700
do I have that data

69
00:04:58,820 --> 00:05:05,300
and then each shard returns its response which is either the data or no data and then mongos has to merge

70
00:05:05,330 --> 00:05:08,320
all that data together and return it.

71
00:05:08,330 --> 00:05:09,380
Option 2 is that

72
00:05:09,380 --> 00:05:11,820
your find does contain a shard key,

73
00:05:11,990 --> 00:05:17,800
let's say now your shard key is the username and you are searching for the username in your find filter

74
00:05:18,170 --> 00:05:23,990
and then mongos can directly forward this to the right shard and fetch the data from there which is

75
00:05:23,990 --> 00:05:28,470
of course more efficient. And that is the part that then matters for you as a developer

76
00:05:28,470 --> 00:05:34,130
again, if you know you are using sharding, you should sit together with your administrators and choose

77
00:05:34,190 --> 00:05:41,000
a wise shard key keep based on your application needs that is evenly distributed so that you can write

78
00:05:41,000 --> 00:05:46,950
queries that use that shard key as often as possible.

79
00:05:46,990 --> 00:05:47,990
This is sharding, 

80
00:05:48,020 --> 00:05:53,570
it's all about distributing data across servers and then setting up everything such that it can be queried

81
00:05:53,600 --> 00:05:55,720
and used efficiently.

82
00:05:55,710 --> 00:06:02,000
Now setting up sharding is also an advanced topic and that is then a task you'll not have to worry about.

83
00:06:02,090 --> 00:06:09,350
Still just as with replica sets, when we dive into deploying a mongodb server, I will show you a solution

84
00:06:09,350 --> 00:06:11,550
that allows you to easily add sharding.