So enough with the announcements. 1
I want to get to a topic that I've been working on and focused on from my Node.js course. 2
And that is environment variables so like raise your hand in chat and you can by the way in chat you 3
can do a colon wave colon and then it will actually do a little wave hand wave your hand if you're dealing 4
with environment variables in your apps and you'd like easier ways that you're not exactly sure if you're 5
doing the best way for environment variables. 6
Because I struggled with that for quite a while. 7
I mean not just in containers but in all of distributed computing making sure that all of my apps have 8
at least some consistent way to do environment variables. 9
Getting information secrets your you know server your server DNS names you know you name it all that 10
stuff into your containers 11
so all right. 12
So it looks like a few of you are interested in this topic. 13
So this will be part of my node.js course. 14
And then I plan to take some of that and figure out how to release that information to the Web. 15
There's a couple of ways that you can go about it but the basics of this come out of 12 factor apps. 16
And if you haven't heard about a factor app it's been around for a decade or more and it's a concept 17
about 12 factors that make a distributed app more successful essentially and it's from a company that 18
learned early on in cloud and on the Internet how they needed to manage all the different points of 19
their app and how they needed to know the different concerns they had around best practices essentially 20
for writing distributed apps. 21
And chances are if you've got a database on one server and an API over here and a Web site over here 22
and a worker job over there but lets a distributed app right there's lots of definitions for distributed 23
that but generally most of us today if you're concerned about containers and you that's part of your 24
job and you're running them on servers you're probably dealing with some distributed apps. 25
So the interesting part about this is the config state. 26
So if you go back to the twelve factor dot net paid to throw this into chat you can definitely skim 27
through these later and then dive into each topic when you have to deal with that in your app. 28
But the one that I wanted to talk about today was the config and config was about storing the settings 29
for your environment. 30
So as you learn if you're taking my dog or Mastery course you know that you never want your your your 31
images or your apps themselves to have hard coded settings that would be different for each environment 32
and the environment might mean you're like a machine that the dev test server or the staging server 33
or the production server or if you like a lot of us you have multiple levels of production and multiple 34
copies of production like all these things. 35
If there's anything that might change in there that might be memory settings in Java that might be different 36
passwords for an API. 37
Different keys that might be storage keys for your cloud. 38
It could be anything from just the word of maybe those in your app is a little flag that indicates which 39
part of the system you're on and you need that set in a simple little environment variable. 40
All that stuff needs to be pulled out of your app and one of the rules of twelve factor is if strict 41
separation of config from code right. 42
And we've all seen that we've all probably done it putting you know hard hardcoded value like local 43
host because your app is always running on the same server as your database but that's before containers 44
then you might see a hardcoded app that says local host in code right. 45
And that's unfortunate because that makes them data that makes them very rigid and hard to deal with 46
because now you got to change code just a changed environment so you don't want to do that. 47
And one of the ideas of this is that twelve factor apps and if you're following the twelve factor principles 48
you tend to have a very easy time making them into containers the twelve factor apps ideally are using 49
everything through environment variables. 50
The reason is is because every OS and every system all supports that right. 51
If you have a Java app over here and you've got a no GSM over there and then you've got a database app 52
back here that's storing your databases and you've got a MIM caching all these things. 53
Every one of those has a different way of consuming its values its environment variables right. 54
The settings it needs the config it needs to get start up. 55
Those are all different. 56
And so that can be quite frustrating. 57
So the only thing that's consistent between all of them is environment variables. 58
Every shell supports it. 59
Windows Mac Linux they all support the same functionality. 60
It might be slightly different the way you set an environment variable or clear it but the key value 61
is the same and all of them. 62
Right. 63
So that's a core premise and that's what you can see throughout the doctor and the container ecosystem 64
as we've all adopted that as our standard. 65
Now what I means is is like if you have for example an application that has a demo file or an I and 66
I or some other sort of config file that the values need to get to then you have to write a middle layer 67
between the environment variables and then where they're going to go into the app. 68
Right. 69
And I have an example of this in my not so complete BHP app. 70
So over on GitHub in my PSP app images show you a quick literal example. 71
And then we're gonna get to your questions. 72
So in here what I have is a docker file and this sacrifice there's a lot of things but the things I 73
want to focus on is in this level app I have defaults that I set in the docker file for environment 74
variables. 75
Now the reason and the way that I'm setting them here is I prefer that the settings here default to 76
working in a production environment up to you whether you say hey look these defaults are just gonna 77
be the average production values or they might you might say hey this is gonna be the average developer 78
values I prefer to them sort of defaulting a little bit to the production side of things that way I 79
can override them with a compose file locally in my local machine no problem but it's up to you how 80
you do that but you definitely want to create consistent ones here that makes sense from the get go. 81
Now you're gonna overwrite a lot of these with other ways of setting it either through the docker run 82
command or through deploying a swarm or a Cuban network deployment or in your Ducker compose GMO file 83
you will overwrite those. 84
Right. 85
You can even overwrite these at the shell before you run those commands and it will suck those in. 86
You can also you using something like Docker compose you can set a dot EMV file that in the PSP world 87
is actually a standard that it will work in such that in the gamble as well. 88
So you can see that I have these here but obviously some of these go into the app they might go into 89
different files and let's back up a second 90
and I'll show you the next part. 91
So those are embedded in the doctor file when the image is built right now. 92
When I want to look at my doctor compose file this is going to be used for local settings right. 93
When I'm doing development and then I can look through here and you can see that I'm setting them all 94
and maybe either setting them differently. 95
I don't have to override them if I don't want to. 96
But in this case I'm choosing to set similar ones as an example. 97
If I didn't need to override the ones in the image I might just leave a lot of these alone. 98
So there's that. 99
All right. 100
So I'm setting them here and now these will be used on container startup. 101
So if I wanted to change them I have to restart the container. 102
I don't have to delete the container right. 103
I just had to restart it. 104
So you can use Dr. composed to do that then the next step is maybe let's see 105
maybe on my local machine or even in production I need those to be in specific files. 106
Right. 107
So when my kid my application starts up those environment variables need to be somewhere else. 108
They can't just live in environment variables for some apps right. 109
I would call these legacy apps or traditional apps where they're not looking to environment variables 110
by default. 111
Now if I was creating a new app my goal would be that that looks to environment variables and maybe 112
there's some config file or something that the app uses but I'm overriding those with enviroment variables 113
automatically. 114
So the idea is my app is smart enough to say hey if there isn't an environment variable for this I will 115
use this default from this file. 116
But if there's already an enviroment variable set I will use that. 117
Now if you're familiar with swarm and Cuban net is you know that once we get into secrets and configure 118
we can start overwriting those entire files and providing passwords and secrets as individual files. 119
So you also might want your app to be aware of those. 120
But if your app if you're not building your own custom app or you're using maybe stuff from Docker Hub 121
or official open source then what you need to look at is providing an entry point script. 122
Now the way that gets run is in your image you set the entry point script you want it to run and it 123
will run that at startup. 124
And what you can see here is a very simple example of me writing the violent variables and some other 125
hard coded data into a specific file on startup simply using a bash script. 126
Right. 127
Does have to be fancy. 128
You can use other methods to do this as well but it tends to be in the docker official images. 129
This app tends to be the standard. 130
All right. 131
So in this case I'm taking environment variables that were set either in the docker file or the compose 132
file which overrode the image Docker file it'll take those Passim into here and then this will echo 133
that stuff out to a file. 134
It's about image. 135
So this is for a legacy HP app that expects a p HP FP M config file and I need to set certain config 136
values for NPM that are unique to each environment in this case. 137
These are actually this how how much I want the FGM system to grow and how much how many sub processes 138
I want to spawn stuff like that. 139
Right. 140
So that would be different on the local development versus production on production I want it to be 141
to grow a lot. 142
Right. 143
So you would change that stuff here and this is how you take those legacy apps and make them work. 144
Now you can see this stuff in official images because that's how a lot of my stuff I learn from. 145
So if I just went into the my sequel you can see how they did it which is exactly the same way my sequel 146
does needs things in a config file. 147
Right. 148
It needs settings that are dynamic at runtime. 149
So if you looked in the docker file for the official my sequel and you scroll down you'll see that yes 150
it does set some environment variables here but it also has an entry point script that it copies in 151
it links it to a specific location and then it as it is the entry point. 152
So the command here is seeing the CMB you can't really see it. 153
The CMB is for just my sequel the entry point technically runs is before that and then it passes execution 154
to the CMT. 155
Once that script is done running now the way that I find that script in here is I just go up one level 156
and get hub and I click on this eight point so and then I will see an environment variable 157
see 158
and now we have the shell script that's run every time my sequel starts all right. 159
And in here it does things like take environment variables and make them available for the app. 160
In fact it's smart enough that if you add a little underscore file on the end of an environment variable 161
it will take whatever is in that file and turn it into the environment variable which is great for all 162
the applications that still need secrets an API keys and all that stuff. 163
And if you're using swarm secrets or Cuban entities secrets those things are provided as they can be 164
provided as files. 165
Now you can always use stuff like vault and other fancy systems that honestly file based but in containers 166
we tend to use things as files and Linux in general we then use things as files. 167
So you're passing these things in as individual files. 168
And then this little bit of shell script which is open source which means you can copy it and put it 169
into your app you can use that to copy the environment variables out of the text file and then stick 170
each one of them in an environment variable named appropriately and that way the app down later can 171
have environment variables that are written in to config files for the app and everything's wonderful. 172
Right. 173
And this would change on every start of the app because the inherent variables might be different. 174
And then it would rewrite those and that's how that works. 175
So hopefully understand those steps. 176
There's a lot more to it. 177
There's actually a let me get you real quick. 178
There is a great little summary of dealing more with environment variables inside of compose it's article 179
I wrote even get the link here. 180
Yeah this is over on the Riley site. 181
And I did a nice blog article for them. 182
Well I think it's nice. 183
But you know it's up to you to say give us lies. 184
And it talks about Dr. Campos because a lot of what we're dealing with for developers especially is 185
the local config. 186
How do I make my config specific for me. 187
And one of those things is an override file which I unfortunately don't have in this HP example I don't 188
think I don't have an override file but I should. 189
That would be another thing to add into here. 190
But in here I will put that in check. 191
Check that O'Reilly link out to talk about three different things and compose that will really help 192
you up your game on managing configuration and reducing replicated parts right templating means you 193
can add specific parts in the top your compose file and then repeat without having to repeat those over 194
and over again so if you have lots of the same environment variables because you're doing micro services 195
and you have maybe 20 micro services in a compose file templating is for you. 196
This is another thing that will really save you some time and you can have all the same parts that need 197
to be repeated over and over again at the very top of your compose file and then it's this is just a 198
gamble thing really the templating and then you can then insert that on each service at the appropriate 199
time. 200
The way that I do here there's other stuff in here as well I encourage you to check that out including 201
how to use its environment variables inside of composed files not just for setting environment variables 202
in the image but also anything else you want to change in your config file at runtime so check that 203
stuff out. 204
I hope you like it Michael. 205
What's up Michael. 206
Michael Ahrens on the call. 207
Call me. 208
What is this conference call there. 209
Dr. Captain. 210
Often what should not go into environment variables. 211
Who are you baiting. 212
The question. 213
To see what people come up with. 214
What should not go into environment variables. 215
Well so the run of the reasons that Docker and Cuban entities and other tools have created these ways 216
for us to put secrets in the files is so that they don't have to be an environment variables. 217
So ideally if you're creating your own apps for managing your own apps you're keeping those secrets 218
out of environment variables not because necessarily environment variables are inherently insecure it's 219
because we with environment variables we tend to leak them right. 220
They tend to they tend to show up in logs because maybe we do an environment variable dump somewhere 221
in our config or we do. 222
If you don't like a P HP was it there's a p HP test command PSP EMV or I came in what it's called anymore 223
someone I'm sure on here is and remember and tell me but where it'll dump your configuration to a web 224
page there's stuff like that and then if there's any secrets in any environment variables they will 225
be exposed there and you may not intend that especially they leak into logs or in the monitoring system 226
somehow and you. 227
If you have errors sometimes you know if you've ever had applications where if you set it to debug and 228
errors out this is the true thing in dot net where it will then put a whole bunch of variation on the 229
screen and might leak environment variables on the screen because you forgot to lock it down into production 230
mode there's all sorts of ways environment variables can end up in the wrong hands. 231
So a good a good first step on not putting stuff into environment variables would be the secrets and 232
with swarm and communities you can keep those in files and that way you can teach your app how to just 233
get them out of the files. 234
The next step there is you if you can get them from your secrets and config files into the files that 235
the app needs. 236
So instead of storing an environment variable that then eventually gets like I should example eventually 237
gets written to a file you could use can figs and store the configure and secrets and files and then 238
they go directly to the files that maybe my sequel or your app need. 239
Right. 240
So they never technically get into environment variables. 241
So at least not on the server itself. 242
Right. 243
So that's a great topic I'm glad you brought that up. 244
But the thing is is I could probably write an entire course on config secrets environment variables 245
but basically all the things you need for an environment and how to deal with them. 246
Right. 247
And I think we're even if you go to stuff like vault for you and now you're talking about a secret API 248
that you're calling it even gets more complicated. 249
So definitely there's lots to talk about there. 250
And if anyone has good resources that they like feel free to throw and chat or throw them in the comments 251
of the YouTube video for later. 252
All right good question Michael good to see you. 253
Why would would you. 254
Stephen says Would you Ducker file not be in source control though I don't favor having production creds 255
such as user name password and source control. 256
That's why those values are just defaults inside of your doctor files Steven. 257
You wouldn't put real secrets in that file. 258
Right. 259
As you can see the one in my example it's just over here everything in the dagger file is just examples 260
or samples. 261
Now if it's a simple number or something that just needs to be set and there's a default a sort of reasonable 262
default like these right here these p p p.m. settings then sure put in the production ones that you 263
prefer and then you can overwrite them as needed. 264
That way you don't have to set them every single time manually. 265
But yeah if you go down here to any of these ones that are unique to a specific app or a specific environment 266
I'm just putting in dummy values here. 267
I've definitely don't recommend that inside of your code repo. 268
You put in your secrets and your your URLs and environment you know the domain name settings and stuff 269
like that. 270
All right. 271
I agree but your Ducker file itself would be in with your app. 272
Yeah that's where you want to keep it. 273
All right. 274
Good question. 275
Good question yeah. 276
I mean unless they're encrypted right you could do hashing on them if your app accepts that sort of 277
thing. 278
But I still wouldn't keep them in there. 279
I don't I keep environment settings elsewhere but you do need defaults. 280
And what I tend to do in a compose file with Docker compose is I set defaults that work with a local 281
developer out of the box. 282
So Docker compose up would just work for them. 283
Now it may mean they need to get a dot in V file from somewhere else maybe a secure store where you 284
have sort of developer secrets you know that sort of thing. 285
But I really work hard to also not include that like ideally if you're local developer you have everything 286
you need locally. 287
You don't require remote api as remote things and you hopefully have sample data and things that can 288
just be on your local machine and you don't need a lot of secrets that you have to worry about you just 289
have dummy values for everything and you can just get buy locally. 290
Obviously not all of our apps can do that as sort of a utopia. 291
So you're going to come up with a different way to store those secrets and that's why there's so many 292
these ways to set environment variables at runtime so that you can keep them elsewhere and then just 293
get them in at the last second through the environment variables all right good questions on environment 294
verbs. 295
I'm glad we've got this topic today. 296
Let's see 297
what is the simplest way to implement the E and V file. 298
As well I showed you the example of the my sequel app right. 299
And in here they take this script and they there is no EMV file. 300
What this is designed for is this is when you use secrets or config. 301
So if you're using the native secrets and config features of Docker swarm or config maps and secrets 302
in communities you would maybe use this as a way to get those the information out of those files specifically 303
the secrets really and get them into environment variables. 304
If you need them right you may not need to get them out into environment variables but if you did then 305
you could use this little bit of shell script to do that. 306
I hope that answers your question but I wouldn't be. 307
There's no need to create a specific file. 308
Exactly. 309
If you're talking about how to do swarm secrets because swarm secrets create files for you then check 310
out my Docker mastery course at the URL. 311
Brett Fisher dot com slash talker which is down in the description. 312
I talked about that there as well in my swarm course.