We've now seen the basics of micro surfaces.
We've learned a little bit about Docker Cooper NeTi is scaffold and now it's time to take all this knowledge
and kick off our next big project.
This next project is going to be significantly larger significantly larger than the one we just worked
on before we get started on it.
However I want to go over a couple of big notes or important things we learned about this first application.
So a couple of different lessons here.
First I want to reflect on is that the big challenge that we really ran into ran all this micro services
stuff was handling data that really was the biggest challenge in this first application.
It's only going to get worse as we start to work with the more complicated data model.
So if we have anything beyond say just a blog in comments working with that data between multiple different
services is going to be start to become even more challenging we learned that there are more than one
way to share data between services.
We spoke about the differences between sync and acing communication.
There really is a sliding scale on a sync versus acing communication.
So we can have fully sync.
We can the async.
We can also have a combination of the two.
You and I are going to focus on async communication between services on this course.
The reason for this and the reason we're doing this over sync communication is that to be honest with
you sync communication is generally a little bit easier or possibly I should say a lot easier to implement
even though it kind of has some downsides in the actual production environment because it is easy.
I figured hey it's a course let's focus on how to do the hard things right.
You want to have a guide or something to tell you some things to be aware of when you're going through
hard stuff and so I want to show you hard stuff but I want to show you the easy stuff async communication
focuses on communicating changes to our data whenever something happens using events.
So we take these events which can be as we saw a object can be an array a string a number any message
we can possibly imagine.
We take these events and we send them to any event bus it is then up to that event bus to take that
event and send it around to all the different services inside of our application.
And each of those different services can react to that event in some way one of the really nice things
about async communication is that every service generally ends up being a rather self-sufficient.
Now of course as I mentioned multiple times.
No these things are not truly self sufficient because they are relying upon events coming in from outside
sources.
But if any other service starts to go down we can generally still have the rest of our application work
because we do not have strict dependencies between our different services.
We also learned a little bit about Docker.
We saw that it's really easy to use Docker to package up all these services.
And we also saw that Cuban.
Yeah it was definitely a pain to set up going through the basics of pods deployment services.
I'm sure through some of that you were saying Steven why are we doing this.
And I know that really was a slog to get through.
But at the end of the day communities is going to make it a lot easier to deploy our micro services
and scale them up over time as well.
Without a doubt Cuban that is is the way to go even if it is a little bit of a pain to get started with
all right.
So these were some of our big lessons some big things that we said okay.
Hey this is something to pay attention to.
But in addition to lessons we also had some really big pains.
There was a lot of kind of painful things we had to deal with as we were working through this first
application.
So I want to focus on some of these different painful things.
Now let's take a look at some solutions that we're going to start to implement in this next application.
Now this is not an exhaustive list right here.
There are some other things that are pains beyond what I'm just saying.
I'm really just highlighting some the big ticket items because these are going to be items that we're
going to handle much differently in the second application.
So first off we technically had a decent amount of duplicated code.
Kind of not really I guess.
But we had a lot of very similar looking code in each of our services.
We had all the same kind of express setup all the same looking root handlers.
We ended up having kind of duplicated logic around sending down posts or comments between query and
the post service and common services.
Without a doubt there was some amount of duplicated code.
Next up it was definitely a very challenging to picture the flow of events between all these different
services.
So at some point in time I showed you some diagrams to introduce that idea of the moderation service.
And I laid out three different options for how to handle some events flowing between each one.
And I'll be honest with you when I was putting those videos together even I was watching these videos
and saying wait what.
How are how is this stuff going on.
Why is this happening why is what.
And so I think that understanding the flow of events between these services and understanding when events
get dispatched is kind of challenging.
And so I think that we need to figure out a better way to handle that.
It was also really challenging to remember what properties and event should have.
So we were working between multiple different projects and we would have inside of one project we would
emit any event and then we would attempt to receive that in another service.
So we were essentially sharing information across a boundary and we didn't have any good documentation
in place or anything like that to understand the different properties or even the names of those properties
or even the types of data of those properties we're referring to in our events.
And so it was incredibly easy to make a typo we might emit an event in one service and give it a property
with the name of like user name and then maybe on another service we tried to receive that event and
accidentally wrote out a property name in there to off the incoming event referencing like name instead
of user name.
Without a doubt remembering those properties really challenging even if it doesn't really seem so right
now finally will not.
Finally we've got a couple more on here but the next one it was kind of hard to test some these different
event floats.
So for example that moderation thing to test out moderation.
We first had to create a post we then had to create a comment we had to then refresh the page.
We then had to go through the other possibilities.
So a comment with orange in the comments without orange maybe a different spelling of orange maybe orange
capitalized.
There was some even flows where maybe it wasn't really hard but it was certainly tedious to test out
the event flows.
So we had to refresh multiple times we had to think about all these different edge cases all that kind
of stuff.
I can say that was kind of painful now to other things that you might have encountered.
You might have had your computer start to get kind of laggy while running Cuban eddies and scaffold
and maybe having your browser open and all this other stuff so you might be at a point right now where
you're saying gee am I gonna be able to develop all these services locally like on my own machine.
And then finally something that I mentioned very briefly during this first application there is kind
of a interesting case if you really started to think through the flow of events inside your application.
In other words looking at this point right here what if someone create a comment after editing five
others and then editing another post and then while at the same time creating a post and I just want
you to imagine that maybe there is some kind of like magic series of events where if we emitted some
series of events very quickly in cyber application we might actually be able to break our data model
for example maybe some events would flow around our application out of order.
Imagine a case in which maybe someone read it a post and then simultaneously in the same microsecond
read it a comment attached that post we were making some really big assumptions inside this application
about the order of events flowing around what would happen if someone created a post and a common associate
with that at the same time.
And then maybe for some reason the comment creation event showed up instead of our common service before
post creation our combat service was assuming that there was always going to be a post inside it's little
in memory data structure to associate an incoming comment with.
So if we for some reason received a comment creation event before a post creation our entire application
would break.
And so this note right here is meant to say what is there are some really weird series of events wild
some really crazy scenario is going on that might potentially break our application.
How do we figure those things out.
How do we write code to deal with them.
That's something that I kind of alluded to but we never really answered for all these painful things
go solution before I show you the solutions however.
Just a quick note for this next project We're going to work on we're gonna make some really big changes
to our development process to accommodate some of these different painful things and I kind of suspect
that some people watching this course might really dislike some of the decisions or changes that we're
gonna make to our development process.
You might not like them.
You might totally disagree with what we're about to do you might say Oh Steven I don't want to do that.
That sounds like a pain and I got to be honest with you I would not be making these changes to our developing
process if I did not really really think that this was the right way to build micro services.
Seriously this next project we're really not cutting a lot of corners we're cutting a couple of corners
I'll be honest with you and I gonna very much point out where those corners are but we're really not
going to be cutting a lot of corners and I want it.
So I want to show you the right way to do as many things as possible so then mind.
Here's how we gonna solve these problems.
Here's how we're gonna do.
And I know this is a long video.
It's going to kind of wrap this up rather quickly.
All right.
So first off what's duplicated code to solve this.
Let's zoom in over here on the Reagan side.
We are going to build a central library as a NPM module and we're gonna throw in a ton of code into
that library that we're gonna share between all of our different services.
So we are going to be building out a collection of different express servers for this next application
and a lot of these express applications are going to have a lot of common code inside them rather than
duplicating that code all over the place.
We're going to build one central library.
We're gonna throw in a lot of express related code.
A lot of other associated code as well.
And then we're going to install that library inside of all of our services.
So this is going to be really great because we're gonna have a lot of fantastic code reuse the downside
to this however is that we might end up having to work on that library at the very start of this project.
And so there will be some scenarios where I'm going to tell you oh yeah we need a middleware to handle
authentication and so it should look like this.
You might say wait how do you know it should look like this Steven.
Or where are we going to use this middleware.
And you're just gonna have to trust me and understand we're going to use this somewhere.
We're just writing this code upfront so we have it and we don't have to worry about this really tedious
express related stuff when we're trying to really focus on handling data and micro services next up.
Kind of hard to picture the flow of events between these different services.
So our solution or the solution I wrote right here doesn't quite answer this or solve it but suffice
to say we're going to define all of our events in this shared library.
So we have a very precise definition of what all these different events are.
We're gonna make sure that all the very all the different properties are very crystal clear in their
spelling their type all that kind of stuff and this is going to go a long way to helping us understand
what different events we have going between our services and possibly in some cases the order in which
they're flowing around as well.
Next up speaking of hard to remember properties moving forward and this is the first item where I think
you might say wait what.
We're doing this.
Yeah we're gonna start to write everything in typescript.
If you don't know typescript that's entirely possible.
It is OK at the very end of this course.
I have another appendix that's gonna give you an overview of typescript certainly more than enough to
get through the rest of this course and to be honest with you.
I think that possibly a lot of the value in this course is going to be are kind of around learning TypeScript
and applying it in a kind of micros services sort of world is we're gonna write some extremely reusable
and flexible code here.
I seriously put in a tremendous amount of time on really polishing the code quality or the second application
so you might be unhappy that we're using typescript on the second have this course but again I can't
say enough it really is the right way to do this and I have a couple more things I want to say about
that but again long video.
So I'm just going to keep going.
Next up.
It was really hard to test them even flows.
I mean maybe not really half this application but trust me for this next one it is gonna be really hard
to test and flows.
There's going to be very precise series of steps we have to go through in the browser that's going to
be kind of a nightmare kind of annoying to have to do and having to test this stuff manually.
It only goes so far.
So moving forward we're going to start to write tests or a lot of the functionality would be put together
really around a lot of the data and around some of the different express things but not first some the
more common middleware so we're going to put together.
So we're not going to totally overboard with tests but we're going to write a lot of tests moving forward
as opposed to immediately trying to just click around stuff in the browser to test.
Next up if your machine is starting to get laggy running Cuba needs I am going to show you fairly briefly
how to run a Cuban 80s cluster in the cloud in a sort of development environment.
So this is not going to be a production style cluster that is going to be a cluster specifically intended
to allow you to do development on a large Cuban aunties cluster.
In the case that you might be running too many services to really execute on your own local machine
so you can kind of imagine that if you're working at a company that has dozens of different micro services
you might need to have some kind of testing or development environment that works with all those different
services and maybe a dozen or two dozen three dozen services is too much to run on some old laptop.
So the way we solve that is by creating a cluster in the cloud a development style cluster and then
we're going to use the same kind of scaffold of workflow with that remote cluster.
So the real goal of all this or this line item right here is to solve any performance issues you might
have.
Now I do want you know right away chances are if you are going to create a development cluster in the
cloud.
Very likely that you'll end up having to pay money for it.
You might fit underneath a free tier or it might be entirely free but it is very much possible that
you will have to spend money.
Just want throw that out there right now.
Then finally handling this really crazy data stuff where who knows if we look on this and then create
some other resource and then edit it at the exact same instant.
Again that's something we kind of touched on but we didn't really have any good solution for it.
I've really no solution whatsoever we had nothing to mitigate problems like that so that's gonna be
a huge focus in this next application.
We're going to have on ending focus on data consistency and making sure that if events come into our
application in some out of order out of sequence or who knows what we are going to have some code or
some idea of how to deal with that.
So again if you are not happy about typescript if you are not happy about writing tests I apologize
to some degree.
But again I would not be saying Yeah we got to use TypeScript and tests if I did not really think it
was the right way to do this.
And again I've got some more notes or thoughts on TypeScript and test with micro services but too much
for an already long video or going to college.
Good.
Right now we can take a pause right here.
We can come back next video and we're gonna start to take a look at a couple of mockups for the next
application we're going to start and we're going to start to think about the overall architecture of
it as well.