Environment Variables in Container vs. Docker Compose File

8 min readJan 3, 2021

Lately, I’ve changed a service to be deployed on a Docker container instead of RPM packaging. As part of this migration, and based on The Twelve-Factor App, I used environment variables to pass the service some data that vary between different environments (development environment, QA, production). During this process, I encountered some problems and solving them encouraged me to share my insights.

In this post, I’m going to talk about environment variables usage in a Docker container and Compose files, the way we can pass those variables, and how to validate their values. Then, I’ll present a few examples for defining variables as mandatory, optional, and with default values.

Why Should We Use Environment Variables?

An environment variable is a key-value pair that affects the way a process runs in different environments.

Environment variables (often shortened to env vars or env) allow us to:

Change service behavior easily: Services often use parametrized values such as a DB hostname, ports, protocol type, credentials, and more. It’s a good practice to manage those values in a way where we can also easily maintain and change them (for example, use different values per each environment) without having to change also the code.
Secure our code: We must handle sensitive and private data in a secure way, and avoid saving it in a source control system (which is shared among others). Using environment variables allows us to define them outside our codebase, and instead, pass them to a running service.

Another known practice is to save those values in a configuration file. However, when running the service in a container, by using environment variables instead of a config file, we can avoid saving sensitive data in a file located in the host machine. Instead, we can pass private values directly to a Docker container on start time. Having said that, it’s important to realize that a user with access and sudo privileges to the host machine running the Docker container, can execute the command docker inspect and discover all the environment variables passed to it.

Env Vars in Docker Container

Env vars usage in a docker container allows us to pass the container values that vary in different environments and also secret data (i.e DB hostname, ports, credentials).
Let’s say our service connects to an external service. In each working environment (i.e. my local machine, QA, Production), its hostname will be different. Thus, saving its value as env var allows us to specify its unique value in each environment and makes the service portable.
For example, if we use the key “HOST_NAME” to contain this host name value:

Locally: HOST_NAME=localhost
QA: HOST_NAME=qa_service
Production: HOST_NAME=prod_service

To run our service locally, I’ll pass the container the “localhost” value. I can do that:

By adding ENV definition in DockerFile. It means it will be the default value for this env variable in the image.
By using the env option (or the shorter option e) in “docker run” command, we can pass a value and also override it, in case it was previously defined in DockerFile. i.e:

docker run -e HOST_NAME=qa_service DockerFile

To read more about “docker run” command please see here.

Docker Compose File

During my work, I need to run a few containers simultaneously. For example, we have a service that consumes messages from RMQ and then creates records on Couchbase and Elasticsearch. While testing this service locally, I need to launch also a container per each service.
Instead of executing multiple “docker run” commands, with the relevant parameters per each of the services, I can use a Compose file that contains all of this data.

What is Compose file?

Compose is a tool for defining and running multi-container Docker applications. With Compose, you use a YAML file to configure your application’s services. Then, with a single command, you create and start all the services from your configuration. (taken from https://docs.docker.com/compose/)

We can see that the Docker Compose file meets our previous requirement.
Therefore, in our projects that are deployed via Docker, we have a Compose file which is part of the project repository. Based on this Yaml file, we can learn about the project itself and its dependencies.
For a detailed explanation about a Compose file you can read more here.

Env Vars in Docker Compose File

1. Env Vars for Docker Container Usage

After we understand what is the benefit of using a Compose file instead of running each container separately, let’s see how we pass env vars to each container.
As you remember, we talked at the beginning of this post about the “docker run” command and its ENV option. In Docker Compose we have similar definitions.

If we use “docker-compose run” we can add the — env option.
If we use “docker-compose up” we can add to the Yaml file environment or env_file definitions.
We can set the required env vars in a file and specify its name under the env_file section in the Compose file.

For a more detailed explanation about the methods of population env vars to containers via Compose files, you can read here.

Example: In the following example 3 variables are passed to the container using the env_file and environment options: HOST_NAME, ELASTICSEARCH_PORT, ELASTICSEARCH_USER_NAME

version: '3.5'
services:
  service:
    image: "service:latest"
    volumes:
      - "./logs:/usr/local/var/service/logs"
    ports:
      - "8001:8000"
    environment:
      - HOST_NAME=localhost
    env_file:
      - service-variables.env

Here their values are inserted to the env_file named service-variables.env:

$ echo ELASTICSEARCH_PORT=9200 >> service-variables.env
$ echo ELASTICSEARCH_USER_NAME="elasticsearch" >> service-variables.env
$ cat service-variables.env 
ELASTICSEARCH_PORT=9200
ELASTICSEARCH_USER_NAME=elasticsearch

After starting the container, you can run the command “docker inspect container-name” and see all env vars found in it under Config->ENV section, i.e.

{
...
"Mounts": [...],
"Config": {
       ...
       "Env": [
            "ELASTICSEARCH_PORT=9200",
            "ELASTICSEARCH_USER_NAME=elasticsearch",
            "HOST_NAME=localhost",
             ...
            ],
       ...
}
...
}

2. Env Vars for Docker Compose File Usage

Following is an example of a simple Compose file. As you can notice, it contains 3 env vars values(TAG, LOGS_PATH, DEBUG_PORT) and a single setting for env var (HOST_NAME).

version: '3.5'
services:
  service:
    image: "service:${TAG}"
    volumes:
      - "${LOGS_PATH}:/usr/local/var/service/logs"
    ports:
      - "${DEBUG_PORT}:8000"
    environment:
      - HOST_NAME=localhost

Those 3 vars are for the Docker Compose usage, and not for the containers. As opposed to the HOST_NAME value mentioned before, those 3 new values are evaluated during the pre-processing step managed by Docker Compose, and not on the container start\running time:

TAG — the Compose file uses the TAG value to decide which image tag will be processed, but it’s not a value that is passed as an environment variable to the container service.
LOGS_PATH — the Compose file uses the LOGS_PATH value to mount an internal folder from the container itself outside to the host machine. The container is unaware of the LOGS_PATH value.
DEBUG_PORT — port 8000 is exposed to the host machine on the given DEBUG_PORT. The container doesn’t use the DEBUG_PORT value.

Those values allow us to avoid hard-coded values in the Compose file, and make it portable between different execution environments. This way each developer assigns the relevant value per each KEY in the current running environment.

But wait, how should we set those values? Exactly as we pass env vars to containers?

The answer to that is NO. After I’ve spent some time trying to solve a few errors, I realized this important fact: the way to pass an environment variable to Compose file usage is different than the methods we have to pass vars to containers.

Population Methods of Environment Variables to Docker Compose Files

Shell environment variables: Defining them prior to running docker-compose, or at the same execution command in the shell, i.e:
TAG=latest docker-compose up
.env file: Adding to .env the required environment variables.
echo TAG=latest > .env

During my experience with env variables and Docker, I found this option a bit confusing.
Here are some important notes:

The .env file must be found at the same folder we execute the command “docker-compose up”.
When passing variables to a Docker Compose file, we can specify a different env file by using the option --env-file. Please make sure docker-compose is updated since in old versions this option is not supported (for more information, you can read https://github.com/docker/compose/issues/6170).

Example: In the following example 3 variables are passed to the Compose file, using the .env file and shell environment variables options: TAG, DEBUG_PORT, LOGS_PATH

$ echo TAG=latest >> .env
$ echo DEBUG_PORT=8001 >> .env
$ cat .env 
TAG=latest
DEBUG_PORT=8001
$ LOGS_PATH=./logs docker-compose up

Those variables saved in .env or defined locally, are not passed to the container, and we can’t see them in the output of the command “docker inspect container-name”. Try it by yourself :)

Test Your Env Vars Values in Compose File

A very useful command is the docker-compose config, which generates the final configuration docker-compose will use. On any missing values, it will return a relevant warning message.

Example:

We’ll use the Docker Compose file we saw above.

version: '3.5'
services:
  service:
    image: "service:${TAG}"
    volumes:
      - "${LOGS_PATH}:/usr/local/var/service/logs"
    ports:
      - "${DEBUG_PORT}:8000"

When we only set DEBUG_PORT (in this case using shell environment variable): In the following example we can see 2 warning messages for each missing variables and the final generated configuration.

$ DEBUG_PORT=8080 docker-compose --file docker-compose-example.yaml config
WARNING: The TAG variable is not set. Defaulting to a blank string.
WARNING: The LOGS_PATH variable is not set. Defaulting to a blank string.
services:
  service:
    image: 'service:'
    ports:
    - published: 8080
      target: 8000
    volumes:
    - .:/usr/local/var/service/logs:rw
version: '3.5'

Env Vars Definitions in Compose File

Docker Compose file allows us to specify default values and mandatory variables.

1.Default Value

${VARIABLE:-default} set the variable value to default if VARIABLE is unset or empty.
${VARIABLE-default} set the variable value to default if VARIABLE is unset.

2.Mandatory Value

By default, a missing value will cause a warning message while running “docker-compose config” or “docker-compose up”. By adding “?err_msg” to the syntax, we make the variable mandatory instead of an optional value. A missing value now will produce an error.

${VARIABLE:?err_msg} exits with an error message containing err_msg if VARIABLE is unset or empty.
${VARIABLE?err_msg} exits with an error message containing err_msg if VARIABLE is unset.

Examples:

We’ll use the same example from before but change TAG and LOGS_PATH to be mandatory variables:

version: '3.5'
services:
  service:
    image: "service:${TAG:?TAG is not set or empty}"
    volumes:
      - "${LOGS_PATH:?LOGS_PATH is not set}:/usr/local/var/service/logs"
    ports:
      - "${DEBUG_PORT}:8000"

Following are docker-compose config calls, with different env vars definitions:

$ DEBUG_PORT=8080 docker-compose --file docker-compose-example.yaml config
ERROR: Missing mandatory value for "image" option in service "service": TAG is not set or empty
$ DEBUG_PORT=8080 TAG="" docker-compose --file docker-compose-example.yaml config
ERROR: Missing mandatory value for "image" option in service "service": TAG is not set or empty
$ DEBUG_PORT=8080 TAG="latest" docker-compose --file docker-compose-example.yaml config
ERROR: Missing mandatory value for "volumes" option in service "service": LOGS_PATH is not set
$ DEBUG_PORT=8080 TAG="latest" LOGS_PATH="" docker-compose --file docker-compose-example.yaml config
services:
  service:
    image: service:latest
    ports:
    - published: 8080
      target: 8000
    volumes:
    - .:/usr/local/var/service/logs:rw
version: '3.5'

Conclusion

In this post, we talked about environment variables’ importance and the difference between a Docker container and a Docker Compose file usage. We also reviewed the useful docker inspect and docker-compose config commands that allow us to examine our files and the env vars definitions we specified. Hopefully, you’ll find this information useful for you.

Additional Resources

Special thanks to Ophir Harpaz and my manager Gadi Katsovich for their review and feedback.