At this point in time we've been running these two listeners for the last couple of videos or so.
And in this last couple of videos you might have noticed that sometimes you have published an event
and not seen it immediately appear inside one of these listeners.
Let's see if I can replicate that really quickly.
So I'm going to first publish a normal event.
OK I see.
Number 70 right there.
I'm now going to restart both these listeners very very quickly and then published an event very quickly
after that.
And I bet you we are not going to immediately see event number 71 up here right away.
If I do see seventy one up here right away I'm going to restart the publisher yet again and then see
if events somebody to also shows up really quickly.
Let's just give it a shot get I can restart restart publish and you'll notice I do not see seventy one.
Now if we wait for about 30 seconds or so we might see seventy one up here and one is when windows I
can even do another published.
OK.
There's seventy two but without a doubt 71 has been skipped.
So where did events 71 go off to.
Why are we not seeing it appear anywhere.
Well we're going to do a little bit of digging.
I want you to get kind of a natural understanding of what's going on internally with Natsumi server
and hey there's some new one right there.
Why did it take so long right.
So again I want to get you out make sure you have a little bit better idea of what's going on internally
with Nat streaming server.
So it will require us to do a little bit of debugging the first thing I want to begin with.
You might recall that when we put our Nats deployment file together inside of our K directory a little
bit ago on our service we actually exposed to different ports.
So one was for connecting clients that was port for two two two.
There was another port named monitoring on 8 2 2 2.
So we can access the Nat string server on this monitoring ports.
And it's going to give us a lot of information about all the different subscriptions that have been
created all the different clients some incoming traffic statistics stuff like that.
So we're going to take a look at the information that is served to us by this monitoring stuff to get
a better idea of why we are sometimes missing events to do so the first thing we have to do is expose
port eight to two on our local machine so that we can access it and take a look at that monitoring information
so inside my terminal.
Once again I'm going to go back to that window where we had previously set up the port forwarding report
for two 2.
We're going to repeat that same process but for Port 8 2 2 2 instead.
So I can open up another terminal window.
We're going to do that same port forwarding stuff.
So remember we're going to first do a cube Seitel get pods I'm going to find my Nat's deployment pod.
I'm going to copy the name.
Well then do a cube Seitel port dash forward.
I can paste in the pod name and then 8 2 2 2 colon 8 to do 2
we can out open our browser and navigate to a local host 8 2 2 2 and that should print out some monitoring
information about our running and that's server.
Now we're gonna go to a very specific address instead of our browser.
I'm gonna put it to my editor here really quickly just you can read it very easily.
We're going to go to local host colon 8 2 2 2 streaming so I'm going to take that your l put it into
my browser and I'll see something like this right here.
So this is the Nat streaming server monitoring page.
We can click around to these different pages and see some stats or information about our running streaming
server now you'll notice that whenever I click one of these tabs or one these little links I see some
nicely formatted Jason data you might see just a big blob of text I see this nicely formatted because
I'm running a Chrome extension that automatically formats Jason inside my browser if you want to run
the same extension I would encourage you hop on the chrome extension store and just search for Jason
you'll find one very similar very easily so there's a couple of links on this page that I want to highlight
in particular.
First off is clients when I go to clients it's going to print out the information about every client
that is currently connected to my Nat streaming server and in particular it will show the I.D. of each
of them at the very bottom of this array.
I see one client that has an I.D. of ABC and you might recall that our publisher has a client idea of
ABC so this entry right here represents our publisher that is currently connected to Nat streaming server
these two right here represent the two listeners that we currently have running we can also take a look
at channels over here.
You'll notice that on the channels tab as a list of all the active channels currently inside of our
cluster.
So right now we have just one channel of ticket colon created now there's a little bit more information
we can use to kind of extract about these channels and unfortunately it is not super well documented.
In other words there's not really any link on this page to help you understand how to get here.
So I'm going to click on the 10 channels link and then inside the address bar I'm going to put in a
question mark and then subs equals one.
Like so now I'm going to navigate there.
So this is going to print out a lot more information about each of the channels that now streaming server
is currently running.
So now see OK we've got a channel with the name of ticket colon created and we can see that there are
two subscriptions for this channel.
So there are two subscriptions one for each of the two listeners that we are currently running.
They are both members of the same Q Group because they have the same Q Group name there is an ACH wait
property which is the number of seconds that that streaming server is going to wait after sending the
thing a message for that client or the subscription to act or acknowledged the message.
And then there's some other information out of here that you can kind of figure out on your own if you
want to.
So how is this going to help us diagnose this issue with somehow restarting these listeners and then
publishing events and not seeing the event immediately appear.
Well let me show you what's going on.
I want you to take note that inside of my subscription list right now I have two running subscriptions
and then going to go back over I'm going to restart one of the listeners get restarted and now I'm going
to go back over and refresh this page and as soon as I do you'll now see that there are three running
subscriptions.
It looks like they are none of them are online so they all have is offline a false so right now.
When I restarted that listener I without a doubt stopped a listener I stopped it.
I restarted one so the copy of that listener that client that I was running with the subscription it
was closed down and I created a new one.
But for a brief period of time Nat string server is just going to assume that maybe there is some momentary
interrupt in connection or communication with that client.
So for a very brief period of time that streaming service says oh you know what.
That that client that subscription just went offline but I'm sure it'll be back in just a moment or
two.
And so she's going to sit around and wait and wait and wait and wait until eventually it decides you
know what that connection that just went offline.
They're probably not coming back.
And so after some period of time the subscription will eventually be removed from this list.
So it's been about 30 seconds or so.
That's how long I've been yapping on.
So I got to now refresh the page and now it's back to to so that's what's going on behind the scenes.
That's why we are seeing some messages being temporary lost Nordstrom's server thinks that that client
is still around even though we just completely killed it.
So there's some period of time where Nordstrom's servers can say you know what.
I think that thing might be coming back any point time.
I'm just going to hold onto this event and send it to them when they come online.
And that's why it appears that we are losing some messages for some period of time now.
As you can imagine that is not super desirable.
So how can we help Natsumi server understand that when one of these clients goes off line it's not coming
back.
Well there's two things we can do.
The first thing we can do we kind of already did back inside of our Nats deployment file inside the
case directory you might recall that we had that long list of arguments inside of here.
We had arguments of hp hp TI and HP f HP stands for heartbeat a heartbeat is like a little request that
Nat streaming server is going to send to all of its different connected clients every so many seconds.
This is purely a little health check it makes sure that each of these clients is still up and running
HP is how often that's remote server is going to make a heartbeat request to each of its clients HP
TI is how long each client has to respond and SPF is the number of times that each client can fail before
that streaming server is going to assume that that connection is dead and gone.
So when we restart one of these listeners that streaming server is then at some in the next couple of
seconds like zero to five seconds whatever it listener we killed that streaming servers going to send
it a harpy request we're then going to have to wait for that killed process to fail the health check
or the heartbeat twice in a row before Nordstrom's server eventually says OK that thing must really
be dead I'm going to take it off this list of subscriptions.
So that's the first thing we can do to kind of fix this problem.
We can implement tighter harpy checks but even then we still have to wait for 10 seconds or so for this
thing to actually be cleaned up.
So there must be another way to kind of tell that stream server hey our client's going dead don't consider
us to be receiving any messages anymore.
Well let's take a look at that second method of helping Nats understand that a client is going offline
in the next video.