In the last section we spoke about the need to do some amount of pre-processing inside of our application
before we just assume that all these event objects that were that we are receiving from grade are completely
valid and need to be processed.
So in this section we're going to talk about exactly how we are going to approach this processing step.
Now I want to say before we do this that absolutely we could probably just dive right into our request
handler write a for loop that iterates over this list of event objects and then do some just very manual
very tedious logic inside of that for a loop to check every single event and blah blah blah do all this
very you know really tedious logic but rather than taking that kind of very basic and straightforward
approach.
I want to apply a lot of very fancy logic that you'll commonly see inside of express applications.
So we're going to use some helper libraries and whatnot to make writing all this pre-processing very
easy and very straightforward.
Now I say easy and straightforward.
Let me clarify and say easy and straightforward if you understand these helper libraries.
So we're going to have some code that might look a little bit weird but it's going to be very compact
and very concise compared to how it might look if we started to do all these processing with a simple
for loop.
OK.
All right.
So let's talk about exactly how this is going to work.
So the first thing I want to do is kind of consider a response or a kind of a request that we would
get from our Send grid web book.
So here is the entire request right here and here is inside of it for different events that the send
grid API might be passing to us.
So in this case out of these very four distinct events right here there's only one event that we really
care about and that's the one in green right here.
Notice that this one in green has an event of click it has an e-mail defined and it has a u r l with
API surveys the survey ID and then slash.
Yes.
So this is going to be the only one the only event that we really care about.
I wouldn't really care about this event right here because it has in the event of bounce.
We only care about clicks.
I wouldn't care about this one right here because even though it has an event of click it has a u r
l that we don't really care about.
And then on this one down here it has an event of click.
It has a correct u r l and has a valid email but you'll notice that the e-mail the survey ID and the
choice are completely identical to the one over here.
So the words out of these four possible events that we might be getting from the API we only really
care about one.
OK so now we understand that.
I want to show you a diagram that's going to imagine that we received this list of events right here
and kind of walk through the processing that we're going to do to narrow this down to just the one event
that we care about.
This one if you're on the top left.
OK.
So let's walk through this.
Now this diagram is pretty big.
But again we're going to walk through it step by step and really understand exactly how we are handling
this processing.
OK.
So hopefully if I go to 150 that's a little bit too large.
So list 125 for the zoom level.
Always remember that you can plot these diagrams from the Course repo yourself and take a glance at
them at the text on here is too small.
OK.
All right so let's go through this.
So we're going to say that whenever we receive this list of events right here we're going to run a map
function over the list of events.
Remember what map does a map function takes an iterator function that iterator is going to have some
amount of logic to process every element in the original array.
It will make some change or will do something to every element.
It will return some value and then after that process is done for every element in the array the map
function returns a new array with all these new values inside of it.
So we're going to take the list of events we're going to map over it.
And for every single element in that array we're going to do this sequence of steps that we're about
to go through.
OK.
So let's talk about it step by step.
So the first thing that we're going to do with every single event object that we receive is going to
look at its new RL property.
If the event has a U R L So it's like an actual click event.
You know it's an event that we really care about.
We're going to extract just the route portion of the URL.
So in other words if a given event had like localhost Kolin three thousand API blah blah blah.
The only part of your L that you and I really care about here is the route portion.
So that would be API surveys 5 9 7 1 or whatever the survey ID slash Yes.
For the purposes of this processing step we don't care about the domain of the URL at all.
We just care about the route.
So once we extract just the path or just the route portion of the URL we will then attempt to extract
the survey ID and the choice out of the URL so in other words if we were to imagine API slash surveys
slash 5 9 7 1 slash Yes I would want to extract an object that has a key of survey ID of the survey
ID and a property of choice that contains either yes or no.
Now after we've gone through every single one of these elements and extracted the data that we really
care about then we'll start to filter down our resultset and start tossing out elements that we don't
really want to deal with.
So after we extract that piece of data we'll return just the surveyor d the email and the choice for
every record and then if any record does not have a survey ID and a choice.
Some of the words if it kind of fits in this bucket right here where it does have a L but it doesn't
have a survey ID or a choice then we're going to discard it and say hey I don't care about this record.
So if we can imagine running this first series of steps right here on this list of events our list of
records would now look a little something like this.
So we would now have an object with an e-mail a survey ID and a choice.
This one would still have an email survey ID and a choice.
And then the other two events that we have would now be showing up as undefined because remember how
a map function works the map function is going to run for every element and then whatever you return
is be placed into this new array.
And so if we start discarding records that we don't care about because hey they don't have a survey
ID and they don't have a choice then the map function is going to automatically inject undefined in
that LMM for that position inside the array.
So now we're down to just these two records right here.
So the next step is going to be to get rid of that those actual records those kind of elements in the
array that are now undefined.
So we'll dump those two undefined records right here and now will be left with just needs to duplicate
once.
So remember we can very easily have duplicate records inside of this event array.
So we want to make sure that we only have one copy of every unique click from each actual user.
Each person received the e-mail.
So after we're down to this list of events we'll then look through it.
We will remove any duplicate records and then at the very end of the day we end up with just this one
event right here where we have the email survey I.D. and the choice I'll get.
So that's pretty much the flow here.
We're going to take the entire list of events.
We're going to attempt to extract some information if we can't successfully extract the information
that will discard the records will then compact the list of records in the array.
Getting rid of anything that's undefined and then is the very last step we're going to remove any duplicate
records.
So again we're going to use a little bit of fancy logic and fancy syntax as we go through this but I'm
definitely 100 percent comfortable in believing that we will both understand exactly what's going on.
So with this kind of preview out of the way let's continue in the next section we're going to install
a library or two to help us with this entire processing flow and we'll start to implement all this logic.
So quick break and I'll see you in the next section.