In this video I want to show you three more possible solutions that will not work to actually solve
this concurrency stuff.
Now for some people listening to this video these concurrency discussions might be a bit confusing or
really vague or boring.
So if you do not want to hear anything else about concurrency pauses video right now and go on to the
next video.
Otherwise stick around and I'll show you three more possible solutions you might think of that are not
going to quite work out.
So if you're still here.
Let's get to it.
Here's another possible solution that is not going to quite work in this possible solution.
We're going to say that we once again have multiple instances of the Account Service A and B.
We're also going to say that A and B have some shared state between the two of them.
So this might be some shared Reddit server might be a shared database.
It might be direct network connections whatever it is.
It is some shared piece of information in the shared state.
We're going to record the sequence number of every processed event that ever gets processed by A and
B then whenever a or b receives an event to process it will look into the shared state and make sure
that the previous sequence number has already been processed.
So let's walk through what this would really look like.
So we're going to imagine that we get no one.
It goes over to service a service label immediately process this event.
So it's going to deposit 70 dollars.
It will then take that sequence number and put it into the shared store.
Like so next up sequence number two.
That's deposit.
Well imagine that goes over to B B is going to then look into this data store and see if the previous
event sequence number.
So in this case no one has already been processed.
No one has been processed because it is inside the state of store.
And that means that now service B can successfully or at least try to process.
Number two.
So in this case deposit 40 dollars.
No problem will then take that sequence number and put in the shared data store.
And then finally withdrawal goes over to a h checks to see if Number Two has been processed because
three minus one is too yet looks like two is already in there.
So we can successfully withdraw the hundred dollars right away.
So this is a good solution in that it makes sure that we're only ever going to process any event exactly
one time and we're always going to make sure that we only process any or as we all of our events in
the correct order.
That's the benefit to the solution.
The downside is that it requires due process all the events that comes into the services in essentially
a sequential fashion which has a really big performance penalty.
Think about what would happen here if we had actually two accounts instead of just one.
Let's say that we've got an account for user a.
That starts off at zero and user fee that starts off at zero.
So now this first event right here might be a deposit for user a that next one might be a deposit for
user B.
So let's now imagine this goes into a and this goes into B now at this point B.
Down here is gonna take a look at this shared store and I can say has no one been processed yet when
this scenario.
No we're still waiting.
And let's imagine that this event fails processing for whatever reasoning maybe service a does not have
a collection or a connection to the file store or maybe it's just laggy or who knows what's going on.
So you have to essentially wait for this deposit or this event right here to time out over the span
of 30 seconds and eventually be reissued by the NAT server.
And while we are waiting for that 30 seconds deposit for account B right here is going to be held up
even though this account right here account for user B is completely unrelated from user a.
So in other words we are always limited globally for all of our different accounts to only processing
exactly one update at a time which is clearly not a great thing it means every single update that we
want to make inside our application.
Only one update and that means that we are going to be severely constrained in how many or how much
processing or how many updates we can run at any given point in time.
So the solution looks good but it doesn't quite work.
Let's try to take some elements from the solution though and move it forward to another attempt.
OK.
So now in this attempt we're going to try to directly solve that issue we ran into you with the last
one.
So in this case we're going to say that we're going to track exactly which event or watch user each
event is intended to be processed for.
So in this case we're going to say that user gem it's sequence number one he's trying to deposit 70
dollars.
So this is going to go off to service say we're going to try to process for user Jim a deposit of 70.
So it's going to make that change.
And then inside the shared data store we will say that for user Jim we have processed his first event.
They'll store it like so so then the next event to come in is maybe sequence number one for user Mary.
And she is trying to deposit 40 dollars so because this is the first event for user Mary we'll just
go ahead and merely deposit that 40.
Everything looks good then inside the shared data store we'll also record.
We've got Mary's first event then user Jim is going to have 2 and 3.
We have processed both those and shared them inside this data store.
So this is very similar.
But now each user has their own sequence numbers.
How does this solve the issue.
We just ran into.
Well let's think through it really quickly I'm going to undo the diagram.
So we're back to event number one coming into service right here.
Let's say that we go ahead and process this event and we deposit seventy dollars into user Jim's account.
So now we're going to have this first event for Mary go over to service B.
And simultaneously we'll imagine that this deposit for Jim goes into service.
Now once again let's say that user Mary's deposit right here is having some big issues.
Maybe there is an error in processing maybe he cannot connect to the file storage whatever it is.
We cannot process this event with the previous solution we just looked at.
That would have held up Jim's deposit as well but because we are accounting sequence numbers separately
for different users this service day is going to take a look inside of our shared sequence numbers database
that's going to say okay for user Jim where'd you process Jim's first event.
Well this has an event number of 2.
So we are definitely safe in processing Jim's second event.
Let's go ahead and deposit 40 dollars.
Now Jim is up to one hundred and ten then Jim can issue is withdrawal will once again look in.
Okay.
User Jim we've done his first and second event now in its third.
Let's go and process it.
Let's withdraw a hundred dollars we're now down to 10.
Everything looks good.
And in this entire process we might still be waiting on user Mary's first deposit.
So with this kind of scenario we can make sure that one user does not affect another user.
Now we're talking about users here but of course this kind of applies to any kind of resource we can
essentially say that any requests is going to modify some type of resource we're going to put all the
events related to one kind of owner of that resource into its own kind of sequence pool.
Now this is a great solution right here but there's a little issue with the actual implementation we're
really talking about giving every user or every kind of owner of something here it's own personal little
bit of sequence it turns out that with Nat streaming server to get a restarted sequence we have to create
a totally different channel.
So if we want to do something like this we would have to have two channels reserved just for Jim and
we might give them a name like account deposit Jim an account withdrawal.
Jim we then have to come up with two identical looking channels dedicated just to Mary.
So deposit for Mary and withdrawal from Mary as well.
That's the only way out of the box with Nat streaming server that we can get those restarted sequence
numbers.
This definitely sounds like well hey whatever let's just create more channels.
Unfortunately that streaming server there's a decent amount of overhead in adding more channels to be
processed out of the box Nat string server is only going to allow you to have 1000 channels by default
we can increase that number of course or decrease it if we want to but as we create more channels there's
more processing overhead with each one that we create.
So we are not going to be able to create a separate set of channels for every single user or every type
of resource that needs its own sequence numbering.
So this is a solution that definitely looks good but it's not going to work with Nat streaming and as
a matter of fact it turns out that this idea of having these different pools or these different channels
where each one has its own little sequence it turns out that creating these different sequences with
any event plus ends up being a lot of processing overhead.
So you're going to see some resources online that advocate this kind of approach but unfortunately again
you're going to have a kind of tough time finding an event boss implementation that will allow each
of these to get their own sequence numbers.
So all right let's take a look at one more possible solution so in this possible solution we're going
to see something that's very similar to the one we just looked at what we're going to work around that
limitation of not being able to assign each of our different users or kind of owners of resource their
own separate sequence numbers.
Now the other thing I want to remind you is that I've been showing all these diagrams with the sequence
number being known somehow to the publisher.
In reality the sequence number is not automatically known to the publisher.
Instead our publisher is going to publish some event that's going to go over to the Natsumi server and
only then does it get assigned a sequence number.
And then after that it will be sent on to whatever listeners there are so in this approach we're going
to start off with this list of events once again and we're going to somehow store each of these events
at the publisher.
So when I say I started the event I literally mean the publishers can have some kind of database where
it is going to list all the different events that it has dispatched over time as it dispatches each
of these events.
We're going to hope or just kind of pray that after it gets sent over to that streaming server Nats
will send back a message and say Here's the sequence number of the event that you just dispatched.
So we throw this thing over.
It would be assigned number one and then we would hope that that streaming server will send that number
one back over to our publisher.
So then this publisher can say okay I dispatch an event it had sequence of number one and that was for
user Jim.
And here's what the actual event was
done in this event.
We'll go on to our services and then be processed.
So we'll say in this case okay I was going to posit many dollars into Jim's account but here's Jim right
here.
Deposit 70 and then simultaneously we're going to make sure that service a right here stores the I.D.
of the sequence number or the sequence of the sequence number of the event.
It just processed so it's store number one to say I just processed number one.
So now the publisher would move on to say number two.
Same thing would occur here.
We're going to dispatch that.
This thing will be assigned at number two because remember we can only get sequence numbers for the
overall events that go into a channel so hopefully once again this number two would be sent back over
to our publisher.
And it would save that number to write their.
This would go on to our service.
We would take a look at Mary.
We would deposit 40 dollars into her account and we would say the last idea that we process for the
thing was number two.
So now we move on to user Jim with user Jim.
We know that we are making a modification to the user Jim and we know that the last time we modified
user Jim was right here because we are storing all these events.
So we know that the last time we modified user Jim we had an event with a sequence number of number
one they put in the number one right there.
We would then throw this event once again over to Nats this thing would be assigned a sequence number
of number three.
We've had a record that back over here and then we would send this event on to one of our listeners.
And now here's the critical part when service a takes a look at this.
And it goes to process it it would take a look at what the last I.D. was or the last sequence number
40 event that modified user Jim service they could then look inside the database and make sure that
the most recent I.D. for Jim was number one as long as that number one right there and number one right
there are equal to each other than service say we'll say Okay I will process this event.
I will deposit 40 dollars taking us up to 110 and then record this things I.D. or sequence number as
the last sequence processed and now finally we'll move on so for user Jim right here.
We know this is about Jim.
So once again we take a look at the most recent event that they had a sequence number of three.
So we put that in right there send this off to Nats this would be given sequence number four and once
again this would go on to our listener.
Now we're making a modification to user Jim.
We'd make sure the last I.D. we processed our last sequence was number three.
Yep it was.
So let's go ahead and apply this update.
We're going to withdraw a hundred dollars.
We're now down to 10 and we'll update that sequence idea right there too for and that's the whole idea.
So we're going to essentially the keys here are to make sure that the publisher understands the ideas
or the sequence is applied to every given event that's going to store those and we'll know which ideas
or which sequences applied to which user.
Then whenever it dispatches a event tied to that user in the future it will also attach the sequence
for the previous event that modified it.
So now with this whole scenario Imagine what would happen if we simultaneously kind of rewind here a
little bit when we undo a little bit and let's say that we go back to the scenario where we've got a
deposit for Jim and simultaneously a withdrawal as well like so so in this case the deposit would have
a sequence of three and then four like.
So the most recent event for the withdrawal would have been the three.
Yeah.
There we go.
So let's imagine what would happen if we accidentally tried to process the withdrawal before the deposit.
In this case service B would take a look at the idea the last processed one it was number one but this
thing this event says that the last event or the last sequence number concern with this event was number
three.
1 in 3 are not equal.
So service B would refuse to process this event right here.
It would say you know what.
Number one as ers me number three has not yet been processed.
We're going to assume that's going to come in very shortly but until it does I'm going to refuse to
process this event entirely.
So then we were just essentially sit around and wait.
Eventually if this thing times out and get sent back or Jeanette's No problem we're just gonna keep
waiting around until eventually sequence number three gets processed one sequence number three gets
processed.
The one is equal to the last I.D. processed right here so we'll go ahead and process that we're up to
the 110.
And now finally we can handle this event right here.
OK so this one admittedly is pretty darn confusing.
Where does this work.
Well yes and no not quite the whole problem here you might have noticed when I drag this thing over.
I had said that the publisher doesn't actually know the sequence number and so we were gonna kind of
hope that Nat sent back the sequence number.
That does not actually occur when we send an event off to be published by not streaming.
It's kind of a one way operation.
We don't really get the ability to somehow reach out to Nats and say hey for that event we posted a
bit ago.
What was the sequence number on it.
It's something that gets published or known only by Nats and we have a pretty tough time to reach in
and figure out what that sequence number actually was.
So we can't really somehow reach back and say oh yeah this thing was given a sequence number of one.
So any now any future event tied to a user of Jim in this case would have been this one right here.
The last sequence should have been one.
That's the part that we can't really solve.
We can't really store what the last sequence number was for Jim nonetheless.
This does seem like it's getting even closer even though we are being held up by a little technical
limitation here.
OK.
So that's it for these three additional possible solutions.
Once again I apologize for the duration of this video.
This was an optional video and I just want to throw out these other possible solutions.
OK.
So quick pause right here.
We'll take a look at how we're going to actually solve this in the next video.