1
00:00:00,970 --> 00:00:05,310
I am really excited about this video because we finally get to see some really weird concurrency stuff

2
00:00:05,340 --> 00:00:06,380
in action.

3
00:00:06,450 --> 00:00:11,100
So let me tell you what we're going to do between the last video in this one I put together a little

4
00:00:11,100 --> 00:00:12,390
test script.

5
00:00:12,480 --> 00:00:17,820
So this is a little script I put together that didn't go through a very particular series of actions.

6
00:00:18,000 --> 00:00:23,510
And as we start to run the script we're going to be firing off a ton of requests to our server once

7
00:00:23,510 --> 00:00:24,310
the script runs.

8
00:00:24,320 --> 00:00:29,000
We're going to then take a look at the consistency of data between our two databases and we're going

9
00:00:29,000 --> 00:00:35,520
to see that in some very particular cases we can definitely get events being processed out of order.

10
00:00:35,540 --> 00:00:38,660
So let me tell you a little bit more about this little test script I put together.

11
00:00:38,660 --> 00:00:40,460
Now you do not have to run this test script.

12
00:00:40,460 --> 00:00:44,660
I just put together very quickly and I did change a couple of other things by the scenes to get it to

13
00:00:44,660 --> 00:00:45,620
work correctly.

14
00:00:45,620 --> 00:00:49,700
So we're going to take a look at my setup rather than trying to replicate the script on your side.

15
00:00:50,240 --> 00:00:51,650
So here's what the script does.

16
00:00:52,220 --> 00:00:57,320
This script is going to try to create a ticket with a price of five.

17
00:00:57,380 --> 00:01:00,130
It's then going to update a ticket to a price of 10.

18
00:01:00,170 --> 00:01:04,170
And then finally update a ticket to a price of fifteen dollars.

19
00:01:04,170 --> 00:01:07,460
And it's going to repeat that entire process to 100 times in a row.

20
00:01:08,440 --> 00:01:13,240
What's more is that it's going to try to repeat this process all in parallel.

21
00:01:13,240 --> 00:01:16,810
So in other words simultaneously as much as we can.

22
00:01:16,840 --> 00:01:23,290
Anyways we're going to try to run all these requests to create and update tickets at the same time about

23
00:01:23,290 --> 00:01:24,850
200 times in parallel.

24
00:01:24,850 --> 00:01:30,070
Now technically because the way no J.S. works it's not truly in parallel but we can imagine that more

25
00:01:30,070 --> 00:01:35,920
or less kind of somewhat happening at the same time all these requests being issued in parallel.

26
00:01:35,920 --> 00:01:40,290
However the actual update requests will occur in series.

27
00:01:40,360 --> 00:01:45,010
So we're going to first create the ticket then only after that request is completed will we attempt

28
00:01:45,010 --> 00:01:45,720
to update it.

29
00:01:45,760 --> 00:01:50,020
And then only after that request is completed will we attempt to update it a second time to a final

30
00:01:50,020 --> 00:01:57,740
price of 15 let's think about how this is actually gonna play out on our back and servers so over here

31
00:01:57,740 --> 00:02:00,790
on the top left hand side we've got these requests being issued.

32
00:02:00,890 --> 00:02:03,500
Now I'm only going to show one set of requests right here.

33
00:02:03,500 --> 00:02:05,690
So one series of create update update.

34
00:02:05,690 --> 00:02:11,750
But again this is going to happen at two hundred times more or less in parallel so we're going to first

35
00:02:11,750 --> 00:02:17,490
make the request to create a ticket with the price of five that's going to go directly to our ticket

36
00:02:17,490 --> 00:02:22,050
service the ticket services are going to get then going to save a new record inside of its database

37
00:02:22,260 --> 00:02:23,230
with the price of five.

38
00:02:23,520 --> 00:02:28,250
So I would say maybe it gets an idea of C.C. q and a price of five and that's it.

39
00:02:28,260 --> 00:02:34,580
That request has been promised to me processed then the same instant that we create that actual record

40
00:02:34,580 --> 00:02:35,600
inside the database.

41
00:02:35,600 --> 00:02:41,910
We're also going to emit an event from our ticket service of ticket created that event will then go

42
00:02:41,910 --> 00:02:47,250
off to our ticket created channel and then eventually be assigned to a instance of the order service

43
00:02:47,820 --> 00:02:49,470
for the purposes of this test.

44
00:02:49,500 --> 00:02:52,590
I'm running four different copies of the order service.

45
00:02:52,590 --> 00:02:56,480
Right now you are running one copy one instance of it inside your communities cluster.

46
00:02:56,550 --> 00:03:02,420
I'm running four separate copies just to kind of instigate some little concurrency issues here so this

47
00:03:02,420 --> 00:03:08,210
event will be processed by one of our order services maybe it gets processed by instance say and so

48
00:03:08,210 --> 00:03:13,640
inside of our orders database we would replicate some data from that event into our orders database.

49
00:03:13,820 --> 00:03:20,980
So we'd end up with a ticket idea here of Susie Q and A price of five then.

50
00:03:20,980 --> 00:03:25,900
Next up we're gonna make the request to update the ticket to a price of 10 they're going to process

51
00:03:25,900 --> 00:03:31,810
that update to 10 and we're going to update or make the request to try to update B ticket to a price

52
00:03:31,810 --> 00:03:39,440
of 15 as well now because my script that I put together is running all these three requests in series

53
00:03:39,800 --> 00:03:45,030
I'm going to expect my ticket database to always end up with every ticket having a price of fifteen

54
00:03:45,170 --> 00:03:51,060
no two ways about it I really expect the ticket database to have some very consistent data should actually

55
00:03:51,060 --> 00:03:58,350
have every single ticket with a price of 15 however we're going to see very quickly that because we

56
00:03:58,350 --> 00:04:01,330
are creating so many different events here.

57
00:04:01,410 --> 00:04:05,970
So for every single ticket we're creating a total of three events and all these events are going to

58
00:04:05,970 --> 00:04:08,850
be thrown out to these four separate copies of the order service.

59
00:04:08,910 --> 00:04:14,910
There might be some scenarios or some cases where these update events get processed slightly out of

60
00:04:14,940 --> 00:04:15,530
order.

61
00:04:17,140 --> 00:04:23,890
So this ticket update to a price of 10 might get as scientists say and this update for a price of 15

62
00:04:23,980 --> 00:04:26,220
might get assigned to service D.

63
00:04:26,230 --> 00:04:33,140
Now in theory we really really hope for the price 10 event to get processed first.

64
00:04:33,160 --> 00:04:37,760
So what we really want to have happen is see this thing get updated to a price of 10.

65
00:04:37,960 --> 00:04:43,420
So that thing is now processed and then this other event down here the second update event get processed

66
00:04:43,480 --> 00:04:46,150
and have this record end up with a price of 15.

67
00:04:46,300 --> 00:04:51,850
That is what we really really want to have happen is that would lead to our data being consistent between

68
00:04:51,850 --> 00:04:57,830
the tickets database and the orders database however you're going to see very quickly that in some number

69
00:04:57,830 --> 00:05:02,720
of cases because we are emitting so many events and they're going to so many instances of the order

70
00:05:02,720 --> 00:05:09,620
service there might be some scenarios where we accidentally process the update to a price of 15 first

71
00:05:11,210 --> 00:05:16,770
and then only after that process the this other update which it technically should have been process

72
00:05:16,770 --> 00:05:17,270
second.

73
00:05:17,930 --> 00:05:24,770
So process this one right here and end up with a price of 10 and so that would be a very bad scenario.

74
00:05:24,790 --> 00:05:29,440
We would not want to see any tickets with a price or a final price of ten because then we would have

75
00:05:29,440 --> 00:05:34,040
some inconsistent data between our orders database Andy tickets database.

76
00:05:34,120 --> 00:05:38,830
So again I just want to make sure it's super clear after I run the script I'm going to expect that all

77
00:05:38,830 --> 00:05:41,600
the tickets inside the tickets database have a price at 15.

78
00:05:41,770 --> 00:05:44,170
And I feel pretty confident that was going to happen.

79
00:05:44,170 --> 00:05:49,570
However I would really want to also double check and make sure that all the tickets inside the orders

80
00:05:49,570 --> 00:05:51,670
database end up with a price of 15.

81
00:05:51,790 --> 00:05:56,710
And if we see any with the price of 10 that is going to be a pretty bad sign that is going to be a sign

82
00:05:56,710 --> 00:06:01,120
that there are some major concurrency issues inside of our app and we're going to have to somehow fix

83
00:06:01,120 --> 00:06:05,560
those issues cats now that we understand this entire test setup.

84
00:06:05,560 --> 00:06:08,400
We understand what we are really looking for at the end of the day.

85
00:06:08,410 --> 00:06:15,300
Let me show you one last piece of this testing setup to actually figure out how many tickets have a

86
00:06:15,300 --> 00:06:18,830
price of 10 and a price of 15 across these two services.

87
00:06:18,830 --> 00:06:23,780
I set up to terminal windows right here the one on the right is running a mongo shell.

88
00:06:23,790 --> 00:06:27,310
So this is a mongo shell to the orders DB.

89
00:06:27,840 --> 00:06:32,420
So I can use this terminal window over here to essentially make queries directly to my orders database.

90
00:06:32,430 --> 00:06:38,940
For example I can write out DV tickets dot fines now Natalie return all the tickets I've created over

91
00:06:38,940 --> 00:06:48,620
time on the right hand side is a mongo shell to my tickets database and so very similarly I can run

92
00:06:48,680 --> 00:06:54,410
right out a query right here that's can be executed against my tickets database for example DB tickets

93
00:06:54,410 --> 00:07:01,380
dot finds like so now for the purposes of this test I'm going to clear out and delete all the tickets

94
00:07:01,380 --> 00:07:07,080
I have inside of both databases I'll then run my script and we can run a query or two and figure out

95
00:07:07,320 --> 00:07:11,550
whether or not these kind of consistency issues are in play.

96
00:07:11,640 --> 00:07:16,620
So inside both my databases I'm going to delete all my different tickets so inside of orders I'll do

97
00:07:16,650 --> 00:07:26,950
a DV that ticket Scott remove and then over here inside of tickets I'll do a DV tickets dot remove guess.

98
00:07:26,970 --> 00:07:33,670
So now both databases 0 tickets so I'm now going to run my little script that's gonna go through and

99
00:07:33,670 --> 00:07:41,050
create those two hundred separate tickets so here we go.

100
00:07:41,080 --> 00:07:43,540
All right so we'll see just about 200 requests.

101
00:07:43,540 --> 00:07:44,750
I don't know I wasn't really counting.

102
00:07:44,770 --> 00:07:49,780
I don't know if you were but that made to 100 separate tickets and then issued a first update and a

103
00:07:49,780 --> 00:07:51,410
second update for each one.

104
00:07:51,460 --> 00:07:56,140
If I now go back over my scaffold window I'll see if there has been a tremendous number of events published

105
00:07:56,350 --> 00:07:57,880
and processed.

106
00:07:57,880 --> 00:07:58,520
Take a look at that.

107
00:07:58,520 --> 00:08:06,080
That's that is all the events over those 600 requests in total to create and then update twice.

108
00:08:06,280 --> 00:08:06,560
Yes.

109
00:08:06,570 --> 00:08:08,810
Now in theory everything has been processed.

110
00:08:08,820 --> 00:08:10,500
Now here's the moment of truth.

111
00:08:10,530 --> 00:08:12,670
We're gonna first take a look on the right hand side.

112
00:08:12,690 --> 00:08:18,420
We're going to take a look and try to count out the number of tickets with a price of 10 inside of the

113
00:08:18,420 --> 00:08:19,850
tickets database.

114
00:08:19,860 --> 00:08:25,100
Now hopefully we should see 0 tickets for the price of 10 inside the tickets database.

115
00:08:25,110 --> 00:08:25,740
That is the goal.

116
00:08:25,740 --> 00:08:27,090
That's what we're hoping for.

117
00:08:27,090 --> 00:08:30,330
And I have a pretty high degree of certainty that's gonna be the case.

118
00:08:30,940 --> 00:08:39,610
I'll do a DV tickets dot find a price of 10 and I'm gonna get the number of records that match that

119
00:08:39,610 --> 00:08:40,960
criteria right there.

120
00:08:40,960 --> 00:08:42,240
So I end up with zero.

121
00:08:42,280 --> 00:08:42,640
Perfect.

122
00:08:42,640 --> 00:08:43,720
That is what we want.

123
00:08:43,720 --> 00:08:48,430
That means that after we made these three requests to the ticket service there were no tickets inside

124
00:08:48,430 --> 00:08:50,270
of here that had a price of 10.

125
00:08:50,380 --> 00:08:54,310
If there were any that had a price of 10 that would mean that we have got some serious issues inside

126
00:08:54,310 --> 00:08:56,830
of our but we end up with zero.

127
00:08:56,830 --> 00:08:57,630
That of price 10.

128
00:08:57,640 --> 00:08:59,440
So everything looks good there.

129
00:08:59,530 --> 00:09:05,290
So we're gonna run that same query on the orders database and hopefully in a perfect world there are

130
00:09:05,290 --> 00:09:08,490
going to be no records with a price of 10.

131
00:09:08,560 --> 00:09:12,850
If there are no records of the price of 10 that means we processed all these different update events

132
00:09:12,940 --> 00:09:20,410
in the correct order so let's run a query find how many were marked with a price of 10 and go from there.

133
00:09:22,200 --> 00:09:30,510
I'm going to do a DB tickets find the price of 10 and I'll get the number that have a price at 10:00

134
00:09:31,030 --> 00:09:33,790
and you could see in this case I end up with just one.

135
00:09:33,840 --> 00:09:36,950
Now that's a little bit of a disappointment here.

136
00:09:36,960 --> 00:09:38,880
Well partial disappointment.

137
00:09:38,880 --> 00:09:43,320
So again in a perfect world we would have seen zero here zero would've meant that we'd processed all

138
00:09:43,320 --> 00:09:46,920
events in the correct order inside of our order service.

139
00:09:47,520 --> 00:09:53,040
But we saw just one that has a price of 10 that just you know as I run the script every single time

140
00:09:53,040 --> 00:09:56,580
I run it there seems a slightly different number that end up at the price of 10.

141
00:09:56,610 --> 00:10:00,780
The last time I ran it I actually end up with eight separate ones that had a price of 10.

142
00:10:00,870 --> 00:10:05,620
And so it was a little bit more compelling to say hey look we got a major concurrency issue here than

143
00:10:05,640 --> 00:10:06,330
just one.

144
00:10:06,330 --> 00:10:11,490
But even with one we can still say that one record out of two hundred so point five percent of the time

145
00:10:11,880 --> 00:10:17,040
we end up with some major data consistency issue between our orders database and the tickets database.

146
00:10:17,040 --> 00:10:21,150
And naturally I'm sure I could probably run the script another time yeah I'll make another two hundred

147
00:10:21,690 --> 00:10:26,340
and I'll bet you will see that we're going to end up with more than one additional record.

148
00:10:26,430 --> 00:10:28,170
So this looks like there were two now.

149
00:10:28,170 --> 00:10:30,830
So there was previously one with a price of 10.

150
00:10:30,840 --> 00:10:34,680
So in this latest run I now ended up with two additional records being out of sync.

151
00:10:34,680 --> 00:10:36,070
I'll do it just one more time.

152
00:10:36,300 --> 00:10:40,530
Hopefully I'll get some convincing evidence here.

153
00:10:40,540 --> 00:10:41,200
There we go.

154
00:10:41,200 --> 00:10:41,990
All right fantastic.

155
00:10:42,010 --> 00:10:48,480
So now we got seven records in that latest run seven out of 200 that are now out of sync.

156
00:10:48,520 --> 00:10:52,870
If I go and take a look at the ticket service there are still zero inside the ticket service that a

157
00:10:52,870 --> 00:10:56,710
price of 10 which is again very good hundred percent what we expect.

158
00:10:56,710 --> 00:11:02,880
But inside the orders database that's on occurring here on the left there are now ten out of 600 or

159
00:11:02,880 --> 00:11:08,280
in the latest run seven out of 200 that are out of sync so seven out two hundred.

160
00:11:08,290 --> 00:11:10,990
That's kind of bad odds in my opinion.

161
00:11:10,990 --> 00:11:18,220
I would not want to expect to see that seven out of every 200 series of requests to create an update

162
00:11:18,280 --> 00:11:23,080
update a ticket failed catastrophically like this where we end up with our data being out of sync.

163
00:11:23,080 --> 00:11:26,080
So I would say that right now 7 out of 200 that's pretty bad.

164
00:11:26,080 --> 00:11:28,400
And we definitely don't want to have that.

165
00:11:28,410 --> 00:11:32,860
So at this point time I apologize for the length this video but hopefully we can see that without too

166
00:11:32,860 --> 00:11:39,040
much financially we can really start to break some event handling or orders of events being processed

167
00:11:39,070 --> 00:11:40,630
inside of application.

168
00:11:40,630 --> 00:11:45,610
So it's pretty clear to me at least that we need to implement some kind of concurrency control here

169
00:11:45,620 --> 00:11:50,880
we need something to make sure that we process these events in the correct order.

170
00:11:50,970 --> 00:11:55,860
Last thing on mentioned really quickly even though I say that we didn't really have to find angle a

171
00:11:55,860 --> 00:11:57,050
lot to make this thing break.

172
00:11:57,120 --> 00:12:01,080
Just you know I really did have to actually mess around with this application a lot.

173
00:12:01,080 --> 00:12:05,910
I spent the last three hours before this video finally coming up with these series of requests you see

174
00:12:05,910 --> 00:12:06,780
right here.

175
00:12:07,170 --> 00:12:12,900
And also making a couple of changes to the infrastructure as well so specifically spooling up for copies

176
00:12:12,900 --> 00:12:13,760
of the order service.

177
00:12:13,830 --> 00:12:16,680
There are technically four copies that ticket service as well.

178
00:12:16,740 --> 00:12:19,910
So I really truly did have to try to break this stuff.

179
00:12:19,920 --> 00:12:24,840
So just you know it is not inherently flimsy or anything like that and using this event based pattern

180
00:12:25,110 --> 00:12:28,320
doesn't mean that we're going to automatically start to break every time.

181
00:12:28,320 --> 00:12:33,750
However at the same time I really just mean to say that there is the potential for stuff going wrong.

182
00:12:33,750 --> 00:12:38,370
And so it would behoove us to spend a little bit of time to try to figure out this concurrency stuff.

183
00:12:38,370 --> 00:12:40,030
So just a quick little note there.

184
00:12:40,110 --> 00:12:44,430
Don't feel like this stuff is horribly flimsy or anything like that because again I really had to try

185
00:12:44,430 --> 00:12:45,520
to break it.

186
00:12:45,540 --> 00:12:50,310
OK so enough talking at this point it is clear that there are some concurrency issues.

187
00:12:50,400 --> 00:12:55,410
We need something additional inside of our app to make sure that we process all these events in the

188
00:12:55,410 --> 00:12:56,150
correct order.

189
00:12:56,580 --> 00:12:58,740
So let's figure that step out in the next video.