1
00:00:00,660 --> 00:00:05,550
In the last section we spoke about the need to do some amount of pre-processing inside of our application

2
00:00:05,700 --> 00:00:10,980
before we just assume that all these event objects that were that we are receiving from grade are completely

3
00:00:10,980 --> 00:00:13,500
valid and need to be processed.

4
00:00:13,500 --> 00:00:19,180
So in this section we're going to talk about exactly how we are going to approach this processing step.

5
00:00:19,200 --> 00:00:24,810
Now I want to say before we do this that absolutely we could probably just dive right into our request

6
00:00:24,810 --> 00:00:32,250
handler write a for loop that iterates over this list of event objects and then do some just very manual

7
00:00:32,250 --> 00:00:38,120
very tedious logic inside of that for a loop to check every single event and blah blah blah do all this

8
00:00:38,130 --> 00:00:44,340
very you know really tedious logic but rather than taking that kind of very basic and straightforward

9
00:00:44,340 --> 00:00:45,240
approach.

10
00:00:45,240 --> 00:00:51,870
I want to apply a lot of very fancy logic that you'll commonly see inside of express applications.

11
00:00:51,870 --> 00:00:58,080
So we're going to use some helper libraries and whatnot to make writing all this pre-processing very

12
00:00:58,140 --> 00:01:00,110
easy and very straightforward.

13
00:01:00,120 --> 00:01:01,980
Now I say easy and straightforward.

14
00:01:02,340 --> 00:01:06,410
Let me clarify and say easy and straightforward if you understand these helper libraries.

15
00:01:06,420 --> 00:01:11,100
So we're going to have some code that might look a little bit weird but it's going to be very compact

16
00:01:11,130 --> 00:01:17,370
and very concise compared to how it might look if we started to do all these processing with a simple

17
00:01:17,400 --> 00:01:18,100
for loop.

18
00:01:18,210 --> 00:01:18,980
OK.

19
00:01:19,470 --> 00:01:19,710
All right.

20
00:01:19,710 --> 00:01:22,890
So let's talk about exactly how this is going to work.

21
00:01:22,890 --> 00:01:28,200
So the first thing I want to do is kind of consider a response or a kind of a request that we would

22
00:01:28,200 --> 00:01:30,950
get from our Send grid web book.

23
00:01:30,960 --> 00:01:37,530
So here is the entire request right here and here is inside of it for different events that the send

24
00:01:37,530 --> 00:01:40,520
grid API might be passing to us.

25
00:01:40,530 --> 00:01:46,470
So in this case out of these very four distinct events right here there's only one event that we really

26
00:01:46,470 --> 00:01:48,950
care about and that's the one in green right here.

27
00:01:48,990 --> 00:01:56,010
Notice that this one in green has an event of click it has an e-mail defined and it has a u r l with

28
00:01:56,160 --> 00:01:59,270
API surveys the survey ID and then slash.

29
00:01:59,280 --> 00:01:59,950
Yes.

30
00:01:59,970 --> 00:02:03,720
So this is going to be the only one the only event that we really care about.

31
00:02:03,960 --> 00:02:08,120
I wouldn't really care about this event right here because it has in the event of bounce.

32
00:02:08,190 --> 00:02:09,730
We only care about clicks.

33
00:02:09,870 --> 00:02:15,150
I wouldn't care about this one right here because even though it has an event of click it has a u r

34
00:02:15,150 --> 00:02:17,170
l that we don't really care about.

35
00:02:17,280 --> 00:02:20,020
And then on this one down here it has an event of click.

36
00:02:20,220 --> 00:02:25,920
It has a correct u r l and has a valid email but you'll notice that the e-mail the survey ID and the

37
00:02:25,920 --> 00:02:29,940
choice are completely identical to the one over here.

38
00:02:29,940 --> 00:02:35,910
So the words out of these four possible events that we might be getting from the API we only really

39
00:02:35,910 --> 00:02:36,830
care about one.

40
00:02:36,990 --> 00:02:39,160
OK so now we understand that.

41
00:02:39,180 --> 00:02:45,000
I want to show you a diagram that's going to imagine that we received this list of events right here

42
00:02:45,390 --> 00:02:51,180
and kind of walk through the processing that we're going to do to narrow this down to just the one event

43
00:02:51,180 --> 00:02:52,000
that we care about.

44
00:02:52,050 --> 00:02:53,780
This one if you're on the top left.

45
00:02:54,180 --> 00:02:54,550
OK.

46
00:02:54,600 --> 00:02:56,470
So let's walk through this.

47
00:02:56,630 --> 00:02:58,160
Now this diagram is pretty big.

48
00:02:58,530 --> 00:03:02,790
But again we're going to walk through it step by step and really understand exactly how we are handling

49
00:03:02,790 --> 00:03:05,050
this processing.

50
00:03:05,070 --> 00:03:05,290
OK.

51
00:03:05,310 --> 00:03:10,440
So hopefully if I go to 150 that's a little bit too large.

52
00:03:10,440 --> 00:03:12,590
So list 125 for the zoom level.

53
00:03:12,600 --> 00:03:17,160
Always remember that you can plot these diagrams from the Course repo yourself and take a glance at

54
00:03:17,160 --> 00:03:18,720
them at the text on here is too small.

55
00:03:18,800 --> 00:03:19,380
OK.

56
00:03:19,740 --> 00:03:21,100
All right so let's go through this.

57
00:03:21,120 --> 00:03:27,690
So we're going to say that whenever we receive this list of events right here we're going to run a map

58
00:03:27,690 --> 00:03:30,000
function over the list of events.

59
00:03:30,000 --> 00:03:36,450
Remember what map does a map function takes an iterator function that iterator is going to have some

60
00:03:36,450 --> 00:03:41,400
amount of logic to process every element in the original array.

61
00:03:41,430 --> 00:03:44,780
It will make some change or will do something to every element.

62
00:03:44,880 --> 00:03:50,700
It will return some value and then after that process is done for every element in the array the map

63
00:03:50,730 --> 00:03:54,890
function returns a new array with all these new values inside of it.

64
00:03:55,200 --> 00:03:58,230
So we're going to take the list of events we're going to map over it.

65
00:03:58,500 --> 00:04:02,580
And for every single element in that array we're going to do this sequence of steps that we're about

66
00:04:02,580 --> 00:04:03,710
to go through.

67
00:04:04,290 --> 00:04:04,510
OK.

68
00:04:04,530 --> 00:04:06,470
So let's talk about it step by step.

69
00:04:06,480 --> 00:04:11,850
So the first thing that we're going to do with every single event object that we receive is going to

70
00:04:11,850 --> 00:04:14,050
look at its new RL property.

71
00:04:14,550 --> 00:04:19,920
If the event has a U R L So it's like an actual click event.

72
00:04:19,920 --> 00:04:21,640
You know it's an event that we really care about.

73
00:04:21,870 --> 00:04:26,690
We're going to extract just the route portion of the URL.

74
00:04:26,910 --> 00:04:32,530
So in other words if a given event had like localhost Kolin three thousand API blah blah blah.

75
00:04:32,580 --> 00:04:36,940
The only part of your L that you and I really care about here is the route portion.

76
00:04:36,960 --> 00:04:42,840
So that would be API surveys 5 9 7 1 or whatever the survey ID slash Yes.

77
00:04:43,020 --> 00:04:48,080
For the purposes of this processing step we don't care about the domain of the URL at all.

78
00:04:48,090 --> 00:04:50,180
We just care about the route.

79
00:04:50,310 --> 00:04:55,860
So once we extract just the path or just the route portion of the URL we will then attempt to extract

80
00:04:56,160 --> 00:05:04,170
the survey ID and the choice out of the URL so in other words if we were to imagine API slash surveys

81
00:05:04,170 --> 00:05:11,550
slash 5 9 7 1 slash Yes I would want to extract an object that has a key of survey ID of the survey

82
00:05:11,550 --> 00:05:17,340
ID and a property of choice that contains either yes or no.

83
00:05:17,430 --> 00:05:23,070
Now after we've gone through every single one of these elements and extracted the data that we really

84
00:05:23,070 --> 00:05:28,170
care about then we'll start to filter down our resultset and start tossing out elements that we don't

85
00:05:28,170 --> 00:05:30,730
really want to deal with.

86
00:05:30,780 --> 00:05:37,710
So after we extract that piece of data we'll return just the surveyor d the email and the choice for

87
00:05:37,710 --> 00:05:43,210
every record and then if any record does not have a survey ID and a choice.

88
00:05:43,230 --> 00:05:48,480
Some of the words if it kind of fits in this bucket right here where it does have a L but it doesn't

89
00:05:48,480 --> 00:05:54,630
have a survey ID or a choice then we're going to discard it and say hey I don't care about this record.

90
00:05:54,810 --> 00:06:01,800
So if we can imagine running this first series of steps right here on this list of events our list of

91
00:06:01,800 --> 00:06:04,590
records would now look a little something like this.

92
00:06:04,800 --> 00:06:10,110
So we would now have an object with an e-mail a survey ID and a choice.

93
00:06:10,110 --> 00:06:13,620
This one would still have an email survey ID and a choice.

94
00:06:13,770 --> 00:06:19,560
And then the other two events that we have would now be showing up as undefined because remember how

95
00:06:19,590 --> 00:06:25,140
a map function works the map function is going to run for every element and then whatever you return

96
00:06:25,320 --> 00:06:27,160
is be placed into this new array.

97
00:06:27,510 --> 00:06:32,610
And so if we start discarding records that we don't care about because hey they don't have a survey

98
00:06:32,620 --> 00:06:39,030
ID and they don't have a choice then the map function is going to automatically inject undefined in

99
00:06:39,030 --> 00:06:42,080
that LMM for that position inside the array.

100
00:06:42,090 --> 00:06:45,750
So now we're down to just these two records right here.

101
00:06:45,840 --> 00:06:50,220
So the next step is going to be to get rid of that those actual records those kind of elements in the

102
00:06:50,220 --> 00:06:52,050
array that are now undefined.

103
00:06:52,380 --> 00:06:57,540
So we'll dump those two undefined records right here and now will be left with just needs to duplicate

104
00:06:57,540 --> 00:06:58,530
once.

105
00:06:58,560 --> 00:07:04,470
So remember we can very easily have duplicate records inside of this event array.

106
00:07:04,470 --> 00:07:10,670
So we want to make sure that we only have one copy of every unique click from each actual user.

107
00:07:10,680 --> 00:07:12,490
Each person received the e-mail.

108
00:07:12,840 --> 00:07:16,370
So after we're down to this list of events we'll then look through it.

109
00:07:16,420 --> 00:07:22,350
We will remove any duplicate records and then at the very end of the day we end up with just this one

110
00:07:22,350 --> 00:07:28,990
event right here where we have the email survey I.D. and the choice I'll get.

111
00:07:29,010 --> 00:07:30,360
So that's pretty much the flow here.

112
00:07:30,390 --> 00:07:33,080
We're going to take the entire list of events.

113
00:07:33,270 --> 00:07:38,160
We're going to attempt to extract some information if we can't successfully extract the information

114
00:07:38,380 --> 00:07:42,510
that will discard the records will then compact the list of records in the array.

115
00:07:42,510 --> 00:07:48,330
Getting rid of anything that's undefined and then is the very last step we're going to remove any duplicate

116
00:07:48,330 --> 00:07:50,030
records.

117
00:07:50,040 --> 00:07:57,620
So again we're going to use a little bit of fancy logic and fancy syntax as we go through this but I'm

118
00:07:57,630 --> 00:08:03,190
definitely 100 percent comfortable in believing that we will both understand exactly what's going on.

119
00:08:03,600 --> 00:08:08,520
So with this kind of preview out of the way let's continue in the next section we're going to install

120
00:08:08,550 --> 00:08:14,750
a library or two to help us with this entire processing flow and we'll start to implement all this logic.

121
00:08:14,760 --> 00:08:17,190
So quick break and I'll see you in the next section.