1 00:00:01,070 --> 00:00:03,010 We've now seen the basics of micro surfaces. 2 00:00:03,020 --> 00:00:08,000 We've learned a little bit about Docker Cooper NeTi is scaffold and now it's time to take all this knowledge 3 00:00:08,060 --> 00:00:10,520 and kick off our next big project. 4 00:00:10,520 --> 00:00:15,410 This next project is going to be significantly larger significantly larger than the one we just worked 5 00:00:15,410 --> 00:00:17,050 on before we get started on it. 6 00:00:17,050 --> 00:00:22,830 However I want to go over a couple of big notes or important things we learned about this first application. 7 00:00:23,090 --> 00:00:25,270 So a couple of different lessons here. 8 00:00:25,270 --> 00:00:29,860 First I want to reflect on is that the big challenge that we really ran into ran all this micro services 9 00:00:29,870 --> 00:00:34,960 stuff was handling data that really was the biggest challenge in this first application. 10 00:00:35,000 --> 00:00:38,870 It's only going to get worse as we start to work with the more complicated data model. 11 00:00:38,870 --> 00:00:44,000 So if we have anything beyond say just a blog in comments working with that data between multiple different 12 00:00:44,000 --> 00:00:49,820 services is going to be start to become even more challenging we learned that there are more than one 13 00:00:49,820 --> 00:00:52,020 way to share data between services. 14 00:00:52,040 --> 00:00:55,730 We spoke about the differences between sync and acing communication. 15 00:00:55,790 --> 00:00:59,780 There really is a sliding scale on a sync versus acing communication. 16 00:01:00,050 --> 00:01:01,460 So we can have fully sync. 17 00:01:01,460 --> 00:01:02,720 We can the async. 18 00:01:02,720 --> 00:01:05,140 We can also have a combination of the two. 19 00:01:05,360 --> 00:01:09,840 You and I are going to focus on async communication between services on this course. 20 00:01:09,920 --> 00:01:13,970 The reason for this and the reason we're doing this over sync communication is that to be honest with 21 00:01:13,970 --> 00:01:19,580 you sync communication is generally a little bit easier or possibly I should say a lot easier to implement 22 00:01:19,970 --> 00:01:25,720 even though it kind of has some downsides in the actual production environment because it is easy. 23 00:01:25,730 --> 00:01:30,060 I figured hey it's a course let's focus on how to do the hard things right. 24 00:01:30,080 --> 00:01:34,220 You want to have a guide or something to tell you some things to be aware of when you're going through 25 00:01:34,220 --> 00:01:40,830 hard stuff and so I want to show you hard stuff but I want to show you the easy stuff async communication 26 00:01:40,920 --> 00:01:46,920 focuses on communicating changes to our data whenever something happens using events. 27 00:01:46,920 --> 00:01:53,280 So we take these events which can be as we saw a object can be an array a string a number any message 28 00:01:53,280 --> 00:01:54,750 we can possibly imagine. 29 00:01:54,870 --> 00:02:00,330 We take these events and we send them to any event bus it is then up to that event bus to take that 30 00:02:00,330 --> 00:02:03,940 event and send it around to all the different services inside of our application. 31 00:02:04,110 --> 00:02:10,170 And each of those different services can react to that event in some way one of the really nice things 32 00:02:10,170 --> 00:02:15,590 about async communication is that every service generally ends up being a rather self-sufficient. 33 00:02:15,600 --> 00:02:18,150 Now of course as I mentioned multiple times. 34 00:02:18,420 --> 00:02:22,980 No these things are not truly self sufficient because they are relying upon events coming in from outside 35 00:02:22,980 --> 00:02:23,770 sources. 36 00:02:23,910 --> 00:02:29,460 But if any other service starts to go down we can generally still have the rest of our application work 37 00:02:29,490 --> 00:02:35,210 because we do not have strict dependencies between our different services. 38 00:02:35,220 --> 00:02:36,710 We also learned a little bit about Docker. 39 00:02:36,720 --> 00:02:40,710 We saw that it's really easy to use Docker to package up all these services. 40 00:02:40,710 --> 00:02:42,420 And we also saw that Cuban. 41 00:02:42,630 --> 00:02:48,300 Yeah it was definitely a pain to set up going through the basics of pods deployment services. 42 00:02:48,300 --> 00:02:51,640 I'm sure through some of that you were saying Steven why are we doing this. 43 00:02:51,690 --> 00:02:53,880 And I know that really was a slog to get through. 44 00:02:53,880 --> 00:02:59,160 But at the end of the day communities is going to make it a lot easier to deploy our micro services 45 00:02:59,430 --> 00:03:01,700 and scale them up over time as well. 46 00:03:01,740 --> 00:03:06,660 Without a doubt Cuban that is is the way to go even if it is a little bit of a pain to get started with 47 00:03:08,940 --> 00:03:09,140 all right. 48 00:03:09,150 --> 00:03:12,540 So these were some of our big lessons some big things that we said okay. 49 00:03:12,540 --> 00:03:14,300 Hey this is something to pay attention to. 50 00:03:14,430 --> 00:03:18,480 But in addition to lessons we also had some really big pains. 51 00:03:18,510 --> 00:03:22,230 There was a lot of kind of painful things we had to deal with as we were working through this first 52 00:03:22,230 --> 00:03:23,370 application. 53 00:03:23,370 --> 00:03:26,140 So I want to focus on some of these different painful things. 54 00:03:26,210 --> 00:03:31,760 Now let's take a look at some solutions that we're going to start to implement in this next application. 55 00:03:31,770 --> 00:03:33,720 Now this is not an exhaustive list right here. 56 00:03:33,720 --> 00:03:36,390 There are some other things that are pains beyond what I'm just saying. 57 00:03:36,510 --> 00:03:40,440 I'm really just highlighting some the big ticket items because these are going to be items that we're 58 00:03:40,440 --> 00:03:44,170 going to handle much differently in the second application. 59 00:03:44,220 --> 00:03:48,470 So first off we technically had a decent amount of duplicated code. 60 00:03:48,660 --> 00:03:50,740 Kind of not really I guess. 61 00:03:51,120 --> 00:03:54,440 But we had a lot of very similar looking code in each of our services. 62 00:03:54,450 --> 00:03:58,890 We had all the same kind of express setup all the same looking root handlers. 63 00:03:58,890 --> 00:04:04,560 We ended up having kind of duplicated logic around sending down posts or comments between query and 64 00:04:04,560 --> 00:04:06,190 the post service and common services. 65 00:04:06,330 --> 00:04:10,180 Without a doubt there was some amount of duplicated code. 66 00:04:10,240 --> 00:04:15,010 Next up it was definitely a very challenging to picture the flow of events between all these different 67 00:04:15,010 --> 00:04:15,980 services. 68 00:04:16,000 --> 00:04:21,340 So at some point in time I showed you some diagrams to introduce that idea of the moderation service. 69 00:04:21,340 --> 00:04:25,730 And I laid out three different options for how to handle some events flowing between each one. 70 00:04:26,260 --> 00:04:29,770 And I'll be honest with you when I was putting those videos together even I was watching these videos 71 00:04:29,770 --> 00:04:31,640 and saying wait what. 72 00:04:31,650 --> 00:04:33,090 How are how is this stuff going on. 73 00:04:33,090 --> 00:04:35,610 Why is this happening why is what. 74 00:04:35,620 --> 00:04:40,570 And so I think that understanding the flow of events between these services and understanding when events 75 00:04:40,600 --> 00:04:43,070 get dispatched is kind of challenging. 76 00:04:43,090 --> 00:04:46,330 And so I think that we need to figure out a better way to handle that. 77 00:04:46,350 --> 00:04:51,090 It was also really challenging to remember what properties and event should have. 78 00:04:51,100 --> 00:04:55,780 So we were working between multiple different projects and we would have inside of one project we would 79 00:04:56,230 --> 00:05:00,620 emit any event and then we would attempt to receive that in another service. 80 00:05:00,670 --> 00:05:06,040 So we were essentially sharing information across a boundary and we didn't have any good documentation 81 00:05:06,070 --> 00:05:11,800 in place or anything like that to understand the different properties or even the names of those properties 82 00:05:11,890 --> 00:05:17,010 or even the types of data of those properties we're referring to in our events. 83 00:05:17,020 --> 00:05:24,110 And so it was incredibly easy to make a typo we might emit an event in one service and give it a property 84 00:05:24,110 --> 00:05:29,660 with the name of like user name and then maybe on another service we tried to receive that event and 85 00:05:29,660 --> 00:05:35,150 accidentally wrote out a property name in there to off the incoming event referencing like name instead 86 00:05:35,150 --> 00:05:36,470 of user name. 87 00:05:36,470 --> 00:05:41,660 Without a doubt remembering those properties really challenging even if it doesn't really seem so right 88 00:05:41,660 --> 00:05:44,840 now finally will not. 89 00:05:44,840 --> 00:05:48,620 Finally we've got a couple more on here but the next one it was kind of hard to test some these different 90 00:05:48,620 --> 00:05:49,560 event floats. 91 00:05:49,600 --> 00:05:53,170 So for example that moderation thing to test out moderation. 92 00:05:53,170 --> 00:05:58,630 We first had to create a post we then had to create a comment we had to then refresh the page. 93 00:05:58,810 --> 00:06:00,660 We then had to go through the other possibilities. 94 00:06:00,670 --> 00:06:07,180 So a comment with orange in the comments without orange maybe a different spelling of orange maybe orange 95 00:06:07,180 --> 00:06:08,280 capitalized. 96 00:06:08,380 --> 00:06:12,520 There was some even flows where maybe it wasn't really hard but it was certainly tedious to test out 97 00:06:12,520 --> 00:06:13,920 the event flows. 98 00:06:14,120 --> 00:06:18,100 So we had to refresh multiple times we had to think about all these different edge cases all that kind 99 00:06:18,100 --> 00:06:18,680 of stuff. 100 00:06:18,770 --> 00:06:24,380 I can say that was kind of painful now to other things that you might have encountered. 101 00:06:24,550 --> 00:06:29,170 You might have had your computer start to get kind of laggy while running Cuban eddies and scaffold 102 00:06:29,470 --> 00:06:34,210 and maybe having your browser open and all this other stuff so you might be at a point right now where 103 00:06:34,210 --> 00:06:39,290 you're saying gee am I gonna be able to develop all these services locally like on my own machine. 104 00:06:39,460 --> 00:06:43,960 And then finally something that I mentioned very briefly during this first application there is kind 105 00:06:43,960 --> 00:06:49,910 of a interesting case if you really started to think through the flow of events inside your application. 106 00:06:49,930 --> 00:06:54,640 In other words looking at this point right here what if someone create a comment after editing five 107 00:06:54,640 --> 00:07:00,170 others and then editing another post and then while at the same time creating a post and I just want 108 00:07:00,170 --> 00:07:06,280 you to imagine that maybe there is some kind of like magic series of events where if we emitted some 109 00:07:06,280 --> 00:07:11,020 series of events very quickly in cyber application we might actually be able to break our data model 110 00:07:11,410 --> 00:07:15,880 for example maybe some events would flow around our application out of order. 111 00:07:15,880 --> 00:07:22,330 Imagine a case in which maybe someone read it a post and then simultaneously in the same microsecond 112 00:07:22,740 --> 00:07:28,120 read it a comment attached that post we were making some really big assumptions inside this application 113 00:07:28,180 --> 00:07:34,180 about the order of events flowing around what would happen if someone created a post and a common associate 114 00:07:34,180 --> 00:07:35,530 with that at the same time. 115 00:07:35,710 --> 00:07:41,110 And then maybe for some reason the comment creation event showed up instead of our common service before 116 00:07:41,110 --> 00:07:47,210 post creation our combat service was assuming that there was always going to be a post inside it's little 117 00:07:47,210 --> 00:07:50,840 in memory data structure to associate an incoming comment with. 118 00:07:50,840 --> 00:07:58,760 So if we for some reason received a comment creation event before a post creation our entire application 119 00:07:58,760 --> 00:07:59,150 would break. 120 00:07:59,150 --> 00:08:05,040 And so this note right here is meant to say what is there are some really weird series of events wild 121 00:08:05,150 --> 00:08:10,410 some really crazy scenario is going on that might potentially break our application. 122 00:08:10,460 --> 00:08:11,990 How do we figure those things out. 123 00:08:11,990 --> 00:08:13,840 How do we write code to deal with them. 124 00:08:13,880 --> 00:08:19,560 That's something that I kind of alluded to but we never really answered for all these painful things 125 00:08:20,010 --> 00:08:22,890 go solution before I show you the solutions however. 126 00:08:22,980 --> 00:08:29,090 Just a quick note for this next project We're going to work on we're gonna make some really big changes 127 00:08:29,090 --> 00:08:35,600 to our development process to accommodate some of these different painful things and I kind of suspect 128 00:08:35,870 --> 00:08:40,340 that some people watching this course might really dislike some of the decisions or changes that we're 129 00:08:40,340 --> 00:08:42,220 gonna make to our development process. 130 00:08:42,260 --> 00:08:43,120 You might not like them. 131 00:08:43,130 --> 00:08:47,000 You might totally disagree with what we're about to do you might say Oh Steven I don't want to do that. 132 00:08:47,000 --> 00:08:51,710 That sounds like a pain and I got to be honest with you I would not be making these changes to our developing 133 00:08:51,710 --> 00:08:56,900 process if I did not really really think that this was the right way to build micro services. 134 00:08:56,900 --> 00:09:01,060 Seriously this next project we're really not cutting a lot of corners we're cutting a couple of corners 135 00:09:01,070 --> 00:09:06,140 I'll be honest with you and I gonna very much point out where those corners are but we're really not 136 00:09:06,470 --> 00:09:08,100 going to be cutting a lot of corners and I want it. 137 00:09:08,120 --> 00:09:14,220 So I want to show you the right way to do as many things as possible so then mind. 138 00:09:14,220 --> 00:09:15,600 Here's how we gonna solve these problems. 139 00:09:15,600 --> 00:09:16,400 Here's how we're gonna do. 140 00:09:16,410 --> 00:09:17,470 And I know this is a long video. 141 00:09:17,510 --> 00:09:20,240 It's going to kind of wrap this up rather quickly. 142 00:09:20,260 --> 00:09:20,430 All right. 143 00:09:20,440 --> 00:09:23,640 So first off what's duplicated code to solve this. 144 00:09:23,650 --> 00:09:25,220 Let's zoom in over here on the Reagan side. 145 00:09:26,980 --> 00:09:34,170 We are going to build a central library as a NPM module and we're gonna throw in a ton of code into 146 00:09:34,170 --> 00:09:37,900 that library that we're gonna share between all of our different services. 147 00:09:37,950 --> 00:09:41,940 So we are going to be building out a collection of different express servers for this next application 148 00:09:42,210 --> 00:09:46,920 and a lot of these express applications are going to have a lot of common code inside them rather than 149 00:09:46,920 --> 00:09:48,670 duplicating that code all over the place. 150 00:09:48,720 --> 00:09:51,510 We're going to build one central library. 151 00:09:51,510 --> 00:09:53,950 We're gonna throw in a lot of express related code. 152 00:09:53,970 --> 00:09:56,620 A lot of other associated code as well. 153 00:09:56,640 --> 00:10:00,730 And then we're going to install that library inside of all of our services. 154 00:10:00,750 --> 00:10:05,730 So this is going to be really great because we're gonna have a lot of fantastic code reuse the downside 155 00:10:05,730 --> 00:10:10,980 to this however is that we might end up having to work on that library at the very start of this project. 156 00:10:10,980 --> 00:10:15,540 And so there will be some scenarios where I'm going to tell you oh yeah we need a middleware to handle 157 00:10:15,540 --> 00:10:18,510 authentication and so it should look like this. 158 00:10:18,510 --> 00:10:21,140 You might say wait how do you know it should look like this Steven. 159 00:10:21,180 --> 00:10:22,980 Or where are we going to use this middleware. 160 00:10:23,040 --> 00:10:26,520 And you're just gonna have to trust me and understand we're going to use this somewhere. 161 00:10:26,520 --> 00:10:30,420 We're just writing this code upfront so we have it and we don't have to worry about this really tedious 162 00:10:30,450 --> 00:10:37,800 express related stuff when we're trying to really focus on handling data and micro services next up. 163 00:10:37,950 --> 00:10:40,950 Kind of hard to picture the flow of events between these different services. 164 00:10:40,950 --> 00:10:46,010 So our solution or the solution I wrote right here doesn't quite answer this or solve it but suffice 165 00:10:46,020 --> 00:10:50,110 to say we're going to define all of our events in this shared library. 166 00:10:50,370 --> 00:10:54,370 So we have a very precise definition of what all these different events are. 167 00:10:54,450 --> 00:10:58,380 We're gonna make sure that all the very all the different properties are very crystal clear in their 168 00:10:58,380 --> 00:11:02,790 spelling their type all that kind of stuff and this is going to go a long way to helping us understand 169 00:11:03,090 --> 00:11:08,490 what different events we have going between our services and possibly in some cases the order in which 170 00:11:08,490 --> 00:11:11,620 they're flowing around as well. 171 00:11:11,620 --> 00:11:16,420 Next up speaking of hard to remember properties moving forward and this is the first item where I think 172 00:11:16,420 --> 00:11:17,780 you might say wait what. 173 00:11:17,800 --> 00:11:18,910 We're doing this. 174 00:11:18,910 --> 00:11:21,100 Yeah we're gonna start to write everything in typescript. 175 00:11:21,310 --> 00:11:23,650 If you don't know typescript that's entirely possible. 176 00:11:23,650 --> 00:11:25,870 It is OK at the very end of this course. 177 00:11:25,870 --> 00:11:30,520 I have another appendix that's gonna give you an overview of typescript certainly more than enough to 178 00:11:30,520 --> 00:11:33,060 get through the rest of this course and to be honest with you. 179 00:11:33,060 --> 00:11:38,110 I think that possibly a lot of the value in this course is going to be are kind of around learning TypeScript 180 00:11:38,410 --> 00:11:43,870 and applying it in a kind of micros services sort of world is we're gonna write some extremely reusable 181 00:11:43,930 --> 00:11:45,660 and flexible code here. 182 00:11:45,670 --> 00:11:51,790 I seriously put in a tremendous amount of time on really polishing the code quality or the second application 183 00:11:51,970 --> 00:11:57,250 so you might be unhappy that we're using typescript on the second have this course but again I can't 184 00:11:57,250 --> 00:12:01,690 say enough it really is the right way to do this and I have a couple more things I want to say about 185 00:12:01,690 --> 00:12:03,040 that but again long video. 186 00:12:03,040 --> 00:12:05,950 So I'm just going to keep going. 187 00:12:06,060 --> 00:12:06,440 Next up. 188 00:12:06,450 --> 00:12:08,250 It was really hard to test them even flows. 189 00:12:08,250 --> 00:12:12,420 I mean maybe not really half this application but trust me for this next one it is gonna be really hard 190 00:12:12,420 --> 00:12:13,200 to test and flows. 191 00:12:13,200 --> 00:12:17,280 There's going to be very precise series of steps we have to go through in the browser that's going to 192 00:12:17,280 --> 00:12:22,920 be kind of a nightmare kind of annoying to have to do and having to test this stuff manually. 193 00:12:22,920 --> 00:12:24,150 It only goes so far. 194 00:12:24,150 --> 00:12:29,280 So moving forward we're going to start to write tests or a lot of the functionality would be put together 195 00:12:29,670 --> 00:12:34,620 really around a lot of the data and around some of the different express things but not first some the 196 00:12:34,620 --> 00:12:37,170 more common middleware so we're going to put together. 197 00:12:37,170 --> 00:12:40,980 So we're not going to totally overboard with tests but we're going to write a lot of tests moving forward 198 00:12:41,070 --> 00:12:47,160 as opposed to immediately trying to just click around stuff in the browser to test. 199 00:12:47,270 --> 00:12:53,150 Next up if your machine is starting to get laggy running Cuba needs I am going to show you fairly briefly 200 00:12:53,240 --> 00:12:58,100 how to run a Cuban 80s cluster in the cloud in a sort of development environment. 201 00:12:58,220 --> 00:13:03,110 So this is not going to be a production style cluster that is going to be a cluster specifically intended 202 00:13:03,290 --> 00:13:07,070 to allow you to do development on a large Cuban aunties cluster. 203 00:13:07,190 --> 00:13:11,930 In the case that you might be running too many services to really execute on your own local machine 204 00:13:12,680 --> 00:13:17,090 so you can kind of imagine that if you're working at a company that has dozens of different micro services 205 00:13:17,090 --> 00:13:21,260 you might need to have some kind of testing or development environment that works with all those different 206 00:13:21,260 --> 00:13:28,140 services and maybe a dozen or two dozen three dozen services is too much to run on some old laptop. 207 00:13:28,160 --> 00:13:33,290 So the way we solve that is by creating a cluster in the cloud a development style cluster and then 208 00:13:33,290 --> 00:13:37,340 we're going to use the same kind of scaffold of workflow with that remote cluster. 209 00:13:37,340 --> 00:13:41,750 So the real goal of all this or this line item right here is to solve any performance issues you might 210 00:13:41,750 --> 00:13:42,610 have. 211 00:13:42,690 --> 00:13:47,120 Now I do want you know right away chances are if you are going to create a development cluster in the 212 00:13:47,120 --> 00:13:47,900 cloud. 213 00:13:47,900 --> 00:13:50,760 Very likely that you'll end up having to pay money for it. 214 00:13:50,900 --> 00:13:55,760 You might fit underneath a free tier or it might be entirely free but it is very much possible that 215 00:13:55,760 --> 00:13:56,680 you will have to spend money. 216 00:13:56,690 --> 00:13:58,280 Just want throw that out there right now. 217 00:14:00,060 --> 00:14:06,690 Then finally handling this really crazy data stuff where who knows if we look on this and then create 218 00:14:06,840 --> 00:14:10,510 some other resource and then edit it at the exact same instant. 219 00:14:10,650 --> 00:14:15,170 Again that's something we kind of touched on but we didn't really have any good solution for it. 220 00:14:15,210 --> 00:14:19,920 I've really no solution whatsoever we had nothing to mitigate problems like that so that's gonna be 221 00:14:19,920 --> 00:14:22,430 a huge focus in this next application. 222 00:14:22,540 --> 00:14:27,990 We're going to have on ending focus on data consistency and making sure that if events come into our 223 00:14:27,990 --> 00:14:34,020 application in some out of order out of sequence or who knows what we are going to have some code or 224 00:14:34,020 --> 00:14:37,380 some idea of how to deal with that. 225 00:14:37,570 --> 00:14:43,690 So again if you are not happy about typescript if you are not happy about writing tests I apologize 226 00:14:44,170 --> 00:14:44,950 to some degree. 227 00:14:45,610 --> 00:14:50,050 But again I would not be saying Yeah we got to use TypeScript and tests if I did not really think it 228 00:14:50,050 --> 00:14:51,450 was the right way to do this. 229 00:14:51,490 --> 00:14:56,740 And again I've got some more notes or thoughts on TypeScript and test with micro services but too much 230 00:14:56,740 --> 00:14:59,400 for an already long video or going to college. 231 00:14:59,410 --> 00:14:59,740 Good. 232 00:14:59,770 --> 00:15:01,600 Right now we can take a pause right here. 233 00:15:01,630 --> 00:15:05,770 We can come back next video and we're gonna start to take a look at a couple of mockups for the next 234 00:15:05,770 --> 00:15:10,120 application we're going to start and we're going to start to think about the overall architecture of 235 00:15:10,120 --> 00:15:11,010 it as well.