Environment Variables in Container vs. Docker Compose File

Lately, I’ve changed a service to be deployed on a Docker container instead of RPM packaging. As part of this migration, and based on The Twelve-Factor App, I used environment variables to pass the service some data that vary between different environments (development environment, QA, production). During this process, I encountered some problems and solving them encouraged me to share my insights.

In this post, I’m going to talk about environment variables usage in a Docker container and Compose files, the way we can pass those variables, and how to validate their values. Then, I’ll present a few examples for defining variables as mandatory, optional, and with default values.

Photo by Humberto Braojos on Unsplash

Why Should We Use Environment Variables?

Environment variables (often shortened to env vars or env) allow us to:

  1. Change service behavior easily: Services often use parametrized values such as a DB hostname, ports, protocol type, credentials, and more. It’s a good practice to manage those values in a way where we can also easily maintain and change them (for example, use different values per each environment) without having to change also the code.
  2. Secure our code: We must handle sensitive and private data in a secure way, and avoid saving it in a source control system (which is shared among others). Using environment variables allows us to define them outside our codebase, and instead, pass them to a running service.

Another known practice is to save those values in a configuration file. However, when running the service in a container, by using environment variables instead of a config file, we can avoid saving sensitive data in a file located in the host machine. Instead, we can pass private values directly to a Docker container on start time. Having said that, it’s important to realize that a user with access and sudo privileges to the host machine running the Docker container, can execute the command docker inspect and discover all the environment variables passed to it.

Env Vars in Docker Container

  • Locally: HOST_NAME=localhost
  • QA: HOST_NAME=qa_service
  • Production: HOST_NAME=prod_service

To run our service locally, I’ll pass the container the “localhost” value. I can do that:

  • By adding ENV definition in DockerFile. It means it will be the default value for this env variable in the image.
  • By using the env option (or the shorter option e) in “docker run” command, we can pass a value and also override it, in case it was previously defined in DockerFile. i.e:
docker run -e HOST_NAME=qa_service DockerFile

To read more about “docker run” command please see here.

Docker Compose File

What is Compose file?

Compose is a tool for defining and running multi-container Docker applications. With Compose, you use a YAML file to configure your application’s services. Then, with a single command, you create and start all the services from your configuration. (taken from https://docs.docker.com/compose/)

We can see that the Docker Compose file meets our previous requirement.
Therefore, in our projects that are deployed via Docker, we have a Compose file which is part of the project repository. Based on this Yaml file, we can learn about the project itself and its dependencies.
For a detailed explanation about a Compose file you can read more here.

Env Vars in Docker Compose File

1. Env Vars for Docker Container Usage

  • If we use “docker-compose run” we can add the — env option.
  • If we use “docker-compose up” we can add to the Yaml file environment or env_file definitions.
  • We can set the required env vars in a file and specify its name under the env_file section in the Compose file.

For a more detailed explanation about the methods of population env vars to containers via Compose files, you can read here.

Example: In the following example 3 variables are passed to the container using the env_file and environment options: HOST_NAME, ELASTICSEARCH_PORT, ELASTICSEARCH_USER_NAME

version: '3.5'
services:
service:
image: "service:latest"
volumes:
- "./logs:/usr/local/var/service/logs"
ports:
- "8001:8000"
environment:
- HOST_NAME=localhost
env_file:
- service-variables.env

Here their values are inserted to the env_file named service-variables.env:

$ echo ELASTICSEARCH_PORT=9200 >> service-variables.env
$ echo ELASTICSEARCH_USER_NAME="elasticsearch" >> service-variables.env
$ cat service-variables.env
ELASTICSEARCH_PORT=9200
ELASTICSEARCH_USER_NAME=elasticsearch

After starting the container, you can run the command “docker inspect container-name” and see all env vars found in it under Config->ENV section, i.e.

{
...
"Mounts": [...],
"Config": {
...
"Env": [
"ELASTICSEARCH_PORT=9200",
"ELASTICSEARCH_USER_NAME=elasticsearch",
"HOST_NAME=localhost",
...
],
...
}
...
}

2. Env Vars for Docker Compose File Usage

version: '3.5'
services:
service:
image: "service:${TAG}"
volumes:
- "${LOGS_PATH}:/usr/local/var/service/logs"
ports:
- "${DEBUG_PORT}:8000"
environment:
- HOST_NAME=localhost

Those 3 vars are for the Docker Compose usage, and not for the containers. As opposed to the HOST_NAME value mentioned before, those 3 new values are evaluated during the pre-processing step managed by Docker Compose, and not on the container start\running time:

  • TAG — the Compose file uses the TAG value to decide which image tag will be processed, but it’s not a value that is passed as an environment variable to the container service.
  • LOGS_PATH — the Compose file uses the LOGS_PATH value to mount an internal folder from the container itself outside to the host machine. The container is unaware of the LOGS_PATH value.
  • DEBUG_PORT — port 8000 is exposed to the host machine on the given DEBUG_PORT. The container doesn’t use the DEBUG_PORT value.

Those values allow us to avoid hard-coded values in the Compose file, and make it portable between different execution environments. This way each developer assigns the relevant value per each KEY in the current running environment.

But wait, how should we set those values? Exactly as we pass env vars to containers?

The answer to that is NO. After I’ve spent some time trying to solve a few errors, I realized this important fact: the way to pass an environment variable to Compose file usage is different than the methods we have to pass vars to containers.

Population Methods of Environment Variables to Docker Compose Files

  1. .env file: Adding to .env the required environment variables.
    echo TAG=latest > .env

During my experience with env variables and Docker, I found this option a bit confusing.
Here are some important notes:

  1. The .env file must be found at the same folder we execute the command “docker-compose up”.
  2. When passing variables to a Docker Compose file, we can specify a different env file by using the option --env-file. Please make sure docker-compose is updated since in old versions this option is not supported (for more information, you can read https://github.com/docker/compose/issues/6170).

Example: In the following example 3 variables are passed to the Compose file, using the .env file and shell environment variables options: TAG, DEBUG_PORT, LOGS_PATH

$ echo TAG=latest >> .env
$ echo DEBUG_PORT=8001 >> .env
$ cat .env
TAG=latest
DEBUG_PORT=8001
$ LOGS_PATH=./logs docker-compose up

Those variables saved in .env or defined locally, are not passed to the container, and we can’t see them in the output of the command “docker inspect container-name”. Try it by yourself :)

Test Your Env Vars Values in Compose File

Example:

We’ll use the Docker Compose file we saw above.

version: '3.5'
services:
service:
image: "service:${TAG}"
volumes:
- "${LOGS_PATH}:/usr/local/var/service/logs"
ports:
- "${DEBUG_PORT}:8000"

When we only set DEBUG_PORT (in this case using shell environment variable): In the following example we can see 2 warning messages for each missing variables and the final generated configuration.

$ DEBUG_PORT=8080 docker-compose --file docker-compose-example.yaml config
WARNING: The TAG variable is not set. Defaulting to a blank string.
WARNING: The LOGS_PATH variable is not set. Defaulting to a blank string.
services:
service:
image: 'service:'
ports:
- published: 8080
target: 8000
volumes:
- .:/usr/local/var/service/logs:rw
version: '3.5'

Env Vars Definitions in Compose File

1.Default Value

${VARIABLE:-default} set the variable value to default if VARIABLE is unset or empty.
${VARIABLE-default} set the variable value to default if VARIABLE is unset.

2.Mandatory Value

By default, a missing value will cause a warning message while running “docker-compose config” or “docker-compose up”. By adding “?err_msg” to the syntax, we make the variable mandatory instead of an optional value. A missing value now will produce an error.

${VARIABLE:?err_msg} exits with an error message containing err_msg if VARIABLE is unset or empty.
${VARIABLE?err_msg} exits with an error message containing err_msg if VARIABLE is unset.

Examples:

We’ll use the same example from before but change TAG and LOGS_PATH to be mandatory variables:

version: '3.5'
services:
service:
image: "service:${TAG:?TAG is not set or empty}"
volumes:
- "${LOGS_PATH:?LOGS_PATH is not set}:/usr/local/var/service/logs"
ports:
- "${DEBUG_PORT}:8000"

Following are docker-compose config calls, with different env vars definitions:

$ DEBUG_PORT=8080 docker-compose --file docker-compose-example.yaml config
ERROR: Missing mandatory value for "image" option in service "service": TAG is not set or empty
$ DEBUG_PORT=8080 TAG="" docker-compose --file docker-compose-example.yaml config
ERROR: Missing mandatory value for "image" option in service "service": TAG is not set or empty
$ DEBUG_PORT=8080 TAG="latest" docker-compose --file docker-compose-example.yaml config
ERROR: Missing mandatory value for "volumes" option in service "service": LOGS_PATH is not set
$ DEBUG_PORT=8080 TAG="latest" LOGS_PATH="" docker-compose --file docker-compose-example.yaml config
services:
service:
image: service:latest
ports:
- published: 8080
target: 8000
volumes:
- .:/usr/local/var/service/logs:rw
version: '3.5'

Conclusion

Additional Resources

Special thanks to Ophir Harpaz and my manager Gadi Katsovich for their review and feedback.

Experienced Software Developer @ IBM Trusteer | An alumnus of the 9920 unit

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store