I am really excited about this video because we finally get to see some really weird concurrency stuff
in action.
So let me tell you what we're going to do between the last video in this one I put together a little
test script.
So this is a little script I put together that didn't go through a very particular series of actions.
And as we start to run the script we're going to be firing off a ton of requests to our server once
the script runs.
We're going to then take a look at the consistency of data between our two databases and we're going
to see that in some very particular cases we can definitely get events being processed out of order.
So let me tell you a little bit more about this little test script I put together.
Now you do not have to run this test script.
I just put together very quickly and I did change a couple of other things by the scenes to get it to
work correctly.
So we're going to take a look at my setup rather than trying to replicate the script on your side.
So here's what the script does.
This script is going to try to create a ticket with a price of five.
It's then going to update a ticket to a price of 10.
And then finally update a ticket to a price of fifteen dollars.
And it's going to repeat that entire process to 100 times in a row.
What's more is that it's going to try to repeat this process all in parallel.
So in other words simultaneously as much as we can.
Anyways we're going to try to run all these requests to create and update tickets at the same time about
200 times in parallel.
Now technically because the way no J.S. works it's not truly in parallel but we can imagine that more
or less kind of somewhat happening at the same time all these requests being issued in parallel.
However the actual update requests will occur in series.
So we're going to first create the ticket then only after that request is completed will we attempt
to update it.
And then only after that request is completed will we attempt to update it a second time to a final
price of 15 let's think about how this is actually gonna play out on our back and servers so over here
on the top left hand side we've got these requests being issued.
Now I'm only going to show one set of requests right here.
So one series of create update update.
But again this is going to happen at two hundred times more or less in parallel so we're going to first
make the request to create a ticket with the price of five that's going to go directly to our ticket
service the ticket services are going to get then going to save a new record inside of its database
with the price of five.
So I would say maybe it gets an idea of C.C. q and a price of five and that's it.
That request has been promised to me processed then the same instant that we create that actual record
inside the database.
We're also going to emit an event from our ticket service of ticket created that event will then go
off to our ticket created channel and then eventually be assigned to a instance of the order service
for the purposes of this test.
I'm running four different copies of the order service.
Right now you are running one copy one instance of it inside your communities cluster.
I'm running four separate copies just to kind of instigate some little concurrency issues here so this
event will be processed by one of our order services maybe it gets processed by instance say and so
inside of our orders database we would replicate some data from that event into our orders database.
So we'd end up with a ticket idea here of Susie Q and A price of five then.
Next up we're gonna make the request to update the ticket to a price of 10 they're going to process
that update to 10 and we're going to update or make the request to try to update B ticket to a price
of 15 as well now because my script that I put together is running all these three requests in series
I'm going to expect my ticket database to always end up with every ticket having a price of fifteen
no two ways about it I really expect the ticket database to have some very consistent data should actually
have every single ticket with a price of 15 however we're going to see very quickly that because we
are creating so many different events here.
So for every single ticket we're creating a total of three events and all these events are going to
be thrown out to these four separate copies of the order service.
There might be some scenarios or some cases where these update events get processed slightly out of
order.
So this ticket update to a price of 10 might get as scientists say and this update for a price of 15
might get assigned to service D.
Now in theory we really really hope for the price 10 event to get processed first.
So what we really want to have happen is see this thing get updated to a price of 10.
So that thing is now processed and then this other event down here the second update event get processed
and have this record end up with a price of 15.
That is what we really really want to have happen is that would lead to our data being consistent between
the tickets database and the orders database however you're going to see very quickly that in some number
of cases because we are emitting so many events and they're going to so many instances of the order
service there might be some scenarios where we accidentally process the update to a price of 15 first
and then only after that process the this other update which it technically should have been process
second.
So process this one right here and end up with a price of 10 and so that would be a very bad scenario.
We would not want to see any tickets with a price or a final price of ten because then we would have
some inconsistent data between our orders database Andy tickets database.
So again I just want to make sure it's super clear after I run the script I'm going to expect that all
the tickets inside the tickets database have a price at 15.
And I feel pretty confident that was going to happen.
However I would really want to also double check and make sure that all the tickets inside the orders
database end up with a price of 15.
And if we see any with the price of 10 that is going to be a pretty bad sign that is going to be a sign
that there are some major concurrency issues inside of our app and we're going to have to somehow fix
those issues cats now that we understand this entire test setup.
We understand what we are really looking for at the end of the day.
Let me show you one last piece of this testing setup to actually figure out how many tickets have a
price of 10 and a price of 15 across these two services.
I set up to terminal windows right here the one on the right is running a mongo shell.
So this is a mongo shell to the orders DB.
So I can use this terminal window over here to essentially make queries directly to my orders database.
For example I can write out DV tickets dot fines now Natalie return all the tickets I've created over
time on the right hand side is a mongo shell to my tickets database and so very similarly I can run
right out a query right here that's can be executed against my tickets database for example DB tickets
dot finds like so now for the purposes of this test I'm going to clear out and delete all the tickets
I have inside of both databases I'll then run my script and we can run a query or two and figure out
whether or not these kind of consistency issues are in play.
So inside both my databases I'm going to delete all my different tickets so inside of orders I'll do
a DV that ticket Scott remove and then over here inside of tickets I'll do a DV tickets dot remove guess.
So now both databases 0 tickets so I'm now going to run my little script that's gonna go through and
create those two hundred separate tickets so here we go.
All right so we'll see just about 200 requests.
I don't know I wasn't really counting.
I don't know if you were but that made to 100 separate tickets and then issued a first update and a
second update for each one.
If I now go back over my scaffold window I'll see if there has been a tremendous number of events published
and processed.
Take a look at that.
That's that is all the events over those 600 requests in total to create and then update twice.
Yes.
Now in theory everything has been processed.
Now here's the moment of truth.
We're gonna first take a look on the right hand side.
We're going to take a look and try to count out the number of tickets with a price of 10 inside of the
tickets database.
Now hopefully we should see 0 tickets for the price of 10 inside the tickets database.
That is the goal.
That's what we're hoping for.
And I have a pretty high degree of certainty that's gonna be the case.
I'll do a DV tickets dot find a price of 10 and I'm gonna get the number of records that match that
criteria right there.
So I end up with zero.
Perfect.
That is what we want.
That means that after we made these three requests to the ticket service there were no tickets inside
of here that had a price of 10.
If there were any that had a price of 10 that would mean that we have got some serious issues inside
of our but we end up with zero.
That of price 10.
So everything looks good there.
So we're gonna run that same query on the orders database and hopefully in a perfect world there are
going to be no records with a price of 10.
If there are no records of the price of 10 that means we processed all these different update events
in the correct order so let's run a query find how many were marked with a price of 10 and go from there.
I'm going to do a DB tickets find the price of 10 and I'll get the number that have a price at 10:00
and you could see in this case I end up with just one.
Now that's a little bit of a disappointment here.
Well partial disappointment.
So again in a perfect world we would have seen zero here zero would've meant that we'd processed all
events in the correct order inside of our order service.
But we saw just one that has a price of 10 that just you know as I run the script every single time
I run it there seems a slightly different number that end up at the price of 10.
The last time I ran it I actually end up with eight separate ones that had a price of 10.
And so it was a little bit more compelling to say hey look we got a major concurrency issue here than
just one.
But even with one we can still say that one record out of two hundred so point five percent of the time
we end up with some major data consistency issue between our orders database and the tickets database.
And naturally I'm sure I could probably run the script another time yeah I'll make another two hundred
and I'll bet you will see that we're going to end up with more than one additional record.
So this looks like there were two now.
So there was previously one with a price of 10.
So in this latest run I now ended up with two additional records being out of sync.
I'll do it just one more time.
Hopefully I'll get some convincing evidence here.
There we go.
All right fantastic.
So now we got seven records in that latest run seven out of 200 that are now out of sync.
If I go and take a look at the ticket service there are still zero inside the ticket service that a
price of 10 which is again very good hundred percent what we expect.
But inside the orders database that's on occurring here on the left there are now ten out of 600 or
in the latest run seven out of 200 that are out of sync so seven out two hundred.
That's kind of bad odds in my opinion.
I would not want to expect to see that seven out of every 200 series of requests to create an update
update a ticket failed catastrophically like this where we end up with our data being out of sync.
So I would say that right now 7 out of 200 that's pretty bad.
And we definitely don't want to have that.
So at this point time I apologize for the length this video but hopefully we can see that without too
much financially we can really start to break some event handling or orders of events being processed
inside of application.
So it's pretty clear to me at least that we need to implement some kind of concurrency control here
we need something to make sure that we process these events in the correct order.
Last thing on mentioned really quickly even though I say that we didn't really have to find angle a
lot to make this thing break.
Just you know I really did have to actually mess around with this application a lot.
I spent the last three hours before this video finally coming up with these series of requests you see
right here.
And also making a couple of changes to the infrastructure as well so specifically spooling up for copies
of the order service.
There are technically four copies that ticket service as well.
So I really truly did have to try to break this stuff.
So just you know it is not inherently flimsy or anything like that and using this event based pattern
doesn't mean that we're going to automatically start to break every time.
However at the same time I really just mean to say that there is the potential for stuff going wrong.
And so it would behoove us to spend a little bit of time to try to figure out this concurrency stuff.
So just a quick little note there.
Don't feel like this stuff is horribly flimsy or anything like that because again I really had to try
to break it.
OK so enough talking at this point it is clear that there are some concurrency issues.
We need something additional inside of our app to make sure that we process all these events in the
correct order.
So let's figure that step out in the next video.