In the last couple of videos we've discussed the AK mode we've discussed.
Q groups and this kind of graceful shut down anytime a client is about to close down.
And throughout these videos he might've been saying Stephen this is way too in.
There's no way I need to know how to correctly close down a connection and stuff like that.
Well it turns out all this stuff is extra ordinarily important for some reasons we're going to lay out
inside this video.
To be honest with you this video is probably the most important one inside the entire course that's
going to help you understand how this entire asynchronous style of communication between micro services
and even micro services in general is really really hard to manage on the data side.
So we're going take a look at a couple of diagrams and it's going to be kind of crazy but whatever let's
just get through it.
OK so going to imagine that we're working on a totally different application for just a moment.
The example I'm going to give you is kind of a classic example of how concurrency or handling events
and whatnot can kind of go really wrong really quickly.
So we're going to imagine that we are handling some kind of a banking application in our banking application.
We're going to have a publisher that can image events of account deposits and account withdraw.
And as you can imagine well we're going to keep track of how much money a particular user has inside
their account.
So we're going to emit these two kinds of events.
They're going to go over to these two different channels that have been created inside of our nets straining
server we're going to have two services two copies of the same service called account service.
These two services are going to be members of the same queue group inside of each of these channels.
So whenever an event flows into that streaming service it's the event is only going to go to exactly
at one of these two instances these two instances of the Account Service are going to watch for these
incoming events and then depending upon whether it is a deposit or a withdrawal it is going to open
up a file so plain file on our harddrive and update the amount of money that a user has.
So by default this user will have zero dollars and we're going to increment or decrement that amount
over time.
Now for a real bank a real bank usually doesn't care that much.
If you go under zero dollars on your account balance they're just gonna give you a overdraft fee and
charge you some money for essentially borrowing money for some period of time where we're going to imagine
that our bank is a little bit different.
We're going to say that a user can never ever have less than zero dollars if they go below zero.
That is a critical error and it represents something going extremely wrong inside our application.
So that is a hard requirement.
Let's imagine how our app would work ideally in an ideal situation.
So maybe our publisher comes online and publishes an event of account deposit seventy dollars so that
would go over to this channel that streaming server would take a look at the members of the SKU group
and then send this event off to just one of those members.
So this case maybe a sense it off to this account service right here account service with an open up
that file increment to seventy dollars and that's it.
We're good to go.
Next up is 40 maybe that kids handled by this one.
We go to 110 and that's it all done.
And then finally maybe the user tries to withdraw some money so that will come down to this Q Group
that's channel down here maybe go off to this account service we withdraw one hundred dollars we still
have ten which means still good to go so that is the ideal situation but it turns out that there is
an almost infinite number of ways that this process can fail extremely easily.
Incredibly easy.
Just unbelievably easy.
So let's walk through a couple of different ways that this entire system can fail catastrophically so
the first issue we're going to consider is if a listener fails to process the incoming events so we're
going to imagine once again maybe this account deposit goes out gets assigned to this account service
right here and then this account service tries to process this incoming event.
So ideally this thing would open up some file on the harddrive and add in 70 and then save the file.
But what can go wrong with that process.
Well there's really an unbelievable number of things that can go wrong.
This file could be already locked in other words some other program can already have this file open
on the harddrive which would prevent us from opening it and making changes to it we could also have
some faulty logic inside of here maybe before depositing some money maybe we check to make sure that
the user has the ability to deposit some additional money maybe there's for example a weekly deposit
limit where we don't want any user deposits too much money.
So in that scenario well we might reject that event if the vial is locked like I just mentioned a moment
ago.
That would be rejected.
Maybe we've got some typo inside the file or something like that.
Maybe there's some totally unpredictable issue where this event just fails to be processed.
So whatever the issue is with our current setup remember if anything goes wrong inside of our listener
ideally we would not acknowledge the event and so eventually this event will be re processed but it
takes 30 seconds before and that streaming server decides to actually re process this event and send
it off to some other service like maybe this one over here.
So while we are waiting those 30 seconds for this thing to be processed again the publisher might go
ahead and publish the remaining two events so it might say OK let's do a deposit of 40 maybe that gets
handled down here and maybe it gets handled successfully.
And then after that couple of seconds later we try to do withdrawal gets handled down here and oh if
we try to withdraw one hundred dollars off 40 we're not going to go into the negatives and we have a
critical business air so if for whatever reason any event fails to be processed it can cause a catastrophic
error in our business logic of our program.
And as you saw in the last couple of videos is super easy for that to happen.
So what's the next case in which something can fail catastrophically.
Well if one listener runs more quickly than another let's imagine once again we send off 70 it gets
handled by this service and maybe this service for some reason has a backlog of events.
Maybe there's like 100 events that it's waiting to process because this virtual machine that that service
is running on is right now overloaded or who knows what so maybe this event gets sent over and we're
waiting for this thing to be acknowledged and in the meantime we send over another event to the same
service and we're now waiting for both these things to be processed and acknowledge now these things
have 30 seconds to be processed and it is entirely reasonable that the account service might process
them within that 30 second window.
But in the meantime as we are waiting for them to be processed we might also dispatch a withdrawal and
then maybe that gets sent to this other account service down here that is really really fast.
Maybe we just launched the thing and it has an open or no events to be processed in its backlog so in
that case this instance the account service is going to immediately take a look at that incoming event
tried to withdraw one hundred dollars and once again whoops we're in the negatives critical business
error so this is an entirely possible and likely situation we might successfully eventually process
these events but just because one event went to this service and the others went to this service well
we're totally out of luck.
So here's yet another scenario as we just saw in the last couple of videos Nats might have a client
shutdown but it won't actually consider that client to be dead for 10 20 seconds or so depending upon
those heartbeat settings.
So let's imagine that this service right here gets shut down without it being a graceful shutdown maybe
for whatever reason it just suddenly dies a hundred percent but for some window of time 10 20 seconds
or so Natsumi server is gonna think that thing is still alive.
So in that scenario once again maybe we take the 70 maybe Nats tries to allocate it to this dead service
because it thinks it's still running maybe the same with this event right here.
And then the hundred dollars get sent over to this service right here.
So once again these things are not going to be processed.
They will eventually after 30 seconds and that's doesn't get that acknowledgment and decides to reallocate
them or assign them to some other service.
But in that 30 second window well we're still going to be waiting.
We're still going to go ahead and process with this hundred dollar withdrawal.
And so once again we're going to try to withdraw a hundred dollars off a zero.
Boom everything fails yet again all right just one more little example here.
So in all the slides I've shown you so far we really made the assumption that we were going to do the
deposits and the withdrawal within absolute seconds of each other.
Some of the words these events were all going to be sent into streaming server at pretty much the same
time.
But let's now imagine for a second that well maybe a user is kind of following what a user actually
does.
They don't make two deposits in a row and then withdraw within seconds maybe in this scenario we say
that the first deposit happens on a Tuesday.
Then a another deposit on a Wednesday and then withdrawal on Thursday.
So maybe in this scenario there is a ton of time between each of these events being processed.
So let's now imagine that a user does the deposit goes over to the service maybe the deposit initially
fails maybe that stream server sends it or resigns it somewhere else.
Totally fine if that happens.
We've got a ton of time to actually process this event.
And so eventually even if this thing initially fails eventually we deposit seventy dollars we're good.
So then on Wednesday maybe we do the same thing.
It might get juggled a couple of times back and forth because it's failing to be processed but eventually
we get our money in there and now here comes Thursday and let's do the withdrawal.
So now with this withdrawal let's imagine for a second that the hard drive that we're storing this file
on is really laggy for some reason maybe it takes twenty nine point nine nine seconds to open this file
read the value out and then like another second to actually write the value end or update that value.
Let's imagine what would happen if it took us twenty nine point nine nine nine seconds to open up that
file off the harddrive so at twenty nine point nine nine we open up this value and we get the 110 inside
of application and then a millisecond later like a fraction of a second boom we just hit 30 seconds
and at 30 seconds now assumes that this service failed to process that event.
And so nasty side.
Okay well I better go ahead and try to reprocess this thing.
I'll send it to the other service.
But keep in mind this thing is still successfully processing the events and there's no actual time out
on the service to say stop processing after 30 seconds.
Our assumption is at 30 seconds this thing has totally failed and we don't really need to do any cleanup
that's what we are kind of assuming right now.
So then like two milliseconds later maybe at that point this service goes ahead and finally is able
to update that value.
They say OK we're gonna withdraw 100 dollars.
We're down to just 10 and we're gonna save that back into that file but at that point NATS is already
redistribute that event over to this other service.
And so this service is going to see the incoming event and say oh withdraw a hundred dollars.
Yeah no problem.
Okay.
I can open the file maybe now the harddrive is no longer laggy so it occurs instantaneously and we're
going to try to subtract on our dollars from it and oh once again it would take us below zero critical
error.
So at this point we've now gone through several scenarios that absolutely positively no two ways about
it can and probably will happen inside of application.
So we can have some issue with processing these things and tried to go out of order we might fail because
well there's going out of order because one is one instance of our services running slowly or quickly
we can have the very core issue of one these services crashing Nath tries to throw the event to some
service that isn't actually running then as we just saw.
Well even if we don't run into these kind of issues where everything is occurring at the same time we
might run into these kind of strange corner cases where we try to process these same event twice in
a row.
So at this point hopefully you understand the gravity of these problems.
These are core issues that we kind of have a really tough time to address.
And what's more they are almost guaranteed to happen at some point in time almost guaranteed even in
this scenario where we imagined that these events were occurring within days of each other as opposed
to milliseconds.
We still might run into issues if we tried to recede and process the same event twice or even three
times in a row.
So again these are unbelievable issues that are really challenging to solve.
And what's more these are issues that we can't just somehow solve by using a different event bus everything
we just saw is kind of typical of all event bus implementations that there's not really anything particular
about Napster string server that makes it harder or more challenging to deal with these problems.
Nonetheless we have to deal with them somehow.
So how are we gonna do that.
Well let's take a pause right here.
We'll come back the next video and take a look at some different ways that we're gonna solve all the
issues we just discussed.