Settings management

One of pydantic's most useful applications is settings management.

If you create a model that inherits from BaseSettings, the model initialiser will attempt to determine the values of any fields not passed as keyword arguments by reading from the environment. (Default values will still be used if the matching environment variable is not set.)

This makes it easy to:

• Create a clearly-defined, type-hinted application configuration class
• Automatically read modifications to the configuration from environment variables
• Manually override specific settings in the initialiser where desired (e.g. in unit tests)

For example:

Python 3.7 and abovePython 3.9 and above

from typing import Set

from pydantic import (
    BaseModel,
    BaseSettings,
    PyObject,
    RedisDsn,
    PostgresDsn,
    AmqpDsn,
    Field,
)


class SubModel(BaseModel):
    foo = 'bar'
    apple = 1


class Settings(BaseSettings):
    auth_key: str
    api_key: str = Field(..., env='my_api_key')

    redis_dsn: RedisDsn = 'redis://user:pass@localhost:6379/1'
    pg_dsn: PostgresDsn = 'postgres://user:pass@localhost:5432/foobar'
    amqp_dsn: AmqpDsn = 'amqp://user:pass@localhost:5672/'

    special_function: PyObject = 'math.cos'

    # to override domains:
    # export my_prefix_domains='["foo.com", "bar.com"]'
    domains: Set[str] = set()

    # to override more_settings:
    # export my_prefix_more_settings='{"foo": "x", "apple": 1}'
    more_settings: SubModel = SubModel()

    class Config:
        env_prefix = 'my_prefix_'  # defaults to no prefix, i.e. ""
        fields = {
            'auth_key': {
                'env': 'my_auth_key',
            },
            'redis_dsn': {
                'env': ['service_redis_dsn', 'redis_url']
            }
        }


print(Settings().dict())
"""
{
    'auth_key': 'xxx',
    'api_key': 'xxx',
    'redis_dsn': RedisDsn('redis://user:pass@localhost:6379/1', ),
    'pg_dsn': PostgresDsn('postgres://user:pass@localhost:5432/foobar', ),
    'amqp_dsn': AmqpDsn('amqp://user:pass@localhost:5672/', scheme='amqp',
user='user', password='pass', host='localhost', host_type='int_domain',
port='5672', path='/'),
    'special_function': <built-in function cos>,
    'domains': set(),
    'more_settings': {'foo': 'bar', 'apple': 1},
}
"""

from pydantic import (
    BaseModel,
    BaseSettings,
    PyObject,
    RedisDsn,
    PostgresDsn,
    AmqpDsn,
    Field,
)


class SubModel(BaseModel):
    foo = 'bar'
    apple = 1


class Settings(BaseSettings):
    auth_key: str
    api_key: str = Field(..., env='my_api_key')

    redis_dsn: RedisDsn = 'redis://user:pass@localhost:6379/1'
    pg_dsn: PostgresDsn = 'postgres://user:pass@localhost:5432/foobar'
    amqp_dsn: AmqpDsn = 'amqp://user:pass@localhost:5672/'

    special_function: PyObject = 'math.cos'

    # to override domains:
    # export my_prefix_domains='["foo.com", "bar.com"]'
    domains: set[str] = set()

    # to override more_settings:
    # export my_prefix_more_settings='{"foo": "x", "apple": 1}'
    more_settings: SubModel = SubModel()

    class Config:
        env_prefix = 'my_prefix_'  # defaults to no prefix, i.e. ""
        fields = {
            'auth_key': {
                'env': 'my_auth_key',
            },
            'redis_dsn': {
                'env': ['service_redis_dsn', 'redis_url']
            }
        }


print(Settings().dict())
"""
{
    'auth_key': 'xxx',
    'api_key': 'xxx',
    'redis_dsn': RedisDsn('redis://user:pass@localhost:6379/1', ),
    'pg_dsn': PostgresDsn('postgres://user:pass@localhost:5432/foobar', ),
    'amqp_dsn': AmqpDsn('amqp://user:pass@localhost:5672/', scheme='amqp',
user='user', password='pass', host='localhost', host_type='int_domain',
port='5672', path='/'),
    'special_function': <built-in function cos>,
    'domains': set(),
    'more_settings': {'foo': 'bar', 'apple': 1},
}
"""

(This script is complete, it should run "as is")

Environment variable names

The following rules are used to determine which environment variable(s) are read for a given field:

• By default, the environment variable name is built by concatenating the prefix and field name.

  • For example, to override special_function above, you could use:

      export my_prefix_special_function='foo.bar'
    

  • Note 1: The default prefix is an empty string.
  • Note 2: Field aliases are ignored when building the environment variable name.

• Custom environment variable names can be set in two ways:
  • Config.fields['field_name']['env'] (see auth_key and redis_dsn above)
  • Field(..., env=...) (see api_key above)
• When specifying custom environment variable names, either a string or a list of strings may be provided.
  • When specifying a list of strings, order matters: the first detected value is used.
  • For example, for redis_dsn above, service_redis_dsn would take precedence over redis_url.

Warning

Since v1.0 pydantic does not consider field aliases when finding environment variables to populate settings models, use env instead as described above.

To aid the transition from aliases to env, a warning will be raised when aliases are used on settings models without a custom env var name. If you really mean to use aliases, either ignore the warning or set env to suppress it.

Case-sensitivity can be turned on through the Config:

Python 3.7 and above

from pydantic import BaseSettings


class Settings(BaseSettings):
    redis_host = 'localhost'

    class Config:
        case_sensitive = True

(This script is complete, it should run "as is")

When case_sensitive is True, the environment variable names must match field names (optionally with a prefix), so in this example redis_host could only be modified via export redis_host. If you want to name environment variables all upper-case, you should name attribute all upper-case too. You can still name environment variables anything you like through Field(..., env=...).

In Pydantic v1 case_sensitive is False by default and all variable names are converted to lower-case internally. If you want to define upper-case variable names on nested models like SubModel you have to set case_sensitive=True to disable this behaviour.

Note

On Windows, Python's os module always treats environment variables as case-insensitive, so the case_sensitive config setting will have no effect - settings will always be updated ignoring case.

Parsing environment variable values

For most simple field types (such as int, float, str, etc.), the environment variable value is parsed the same way it would be if passed directly to the initialiser (as a string).

Complex types like list, set, dict, and sub-models are populated from the environment by treating the environment variable's value as a JSON-encoded string.

Another way to populate nested complex variables is to configure your model with the env_nested_delimiter config setting, then use an env variable with a name pointing to the nested module fields. What it does is simply explodes your variable into nested models or dicts. So if you define a variable FOO__BAR__BAZ=123 it will convert it into FOO={'BAR': {'BAZ': 123}} If you have multiple variables with the same structure they will be merged.

With the following environment variables:

# your environment
export V0=0
export SUB_MODEL='{"v1": "json-1", "v2": "json-2"}'
export SUB_MODEL__V2=nested-2
export SUB_MODEL__V3=3
export SUB_MODEL__DEEP__V4=v4

You could load a settings module thus:

Python 3.7 and above

from pydantic import BaseModel, BaseSettings


class DeepSubModel(BaseModel):
    v4: str


class SubModel(BaseModel):
    v1: str
    v2: bytes
    v3: int
    deep: DeepSubModel


class Settings(BaseSettings):
    v0: str
    sub_model: SubModel

    class Config:
        env_nested_delimiter = '__'


print(Settings().dict())
"""
{
    'v0': '0',
    'sub_model': {
        'v1': 'json-1',
        'v2': b'nested-2',
        'v3': 3,
        'deep': {'v4': 'v4'},
    },
}
"""

(This script is complete, it should run "as is")

env_nested_delimiter can be configured via the Config class as shown above, or via the _env_nested_delimiter keyword argument on instantiation.

JSON is only parsed in top-level fields, if you need to parse JSON in sub-models, you will need to implement validators on those models.

Nested environment variables take precedence over the top-level environment variable JSON (e.g. in the example above, SUB_MODEL__V2 trumps SUB_MODEL).

You may also populate a complex type by providing your own parsing function to the parse_env_var classmethod in the Config object.

Python 3.7 and abovePython 3.9 and above

import os
from typing import Any, List

from pydantic import BaseSettings


class Settings(BaseSettings):
    numbers: List[int]

    class Config:
        @classmethod
        def parse_env_var(cls, field_name: str, raw_val: str) -> Any:
            if field_name == 'numbers':
                return [int(x) for x in raw_val.split(',')]
            return cls.json_loads(raw_val)


os.environ['numbers'] = '1,2,3'
print(Settings().dict())
#> {'numbers': [1, 2, 3]}

import os
from typing import Any

from pydantic import BaseSettings


class Settings(BaseSettings):
    numbers: list[int]

    class Config:
        @classmethod
        def parse_env_var(cls, field_name: str, raw_val: str) -> Any:
            if field_name == 'numbers':
                return [int(x) for x in raw_val.split(',')]
            return cls.json_loads(raw_val)


os.environ['numbers'] = '1,2,3'
print(Settings().dict())
#> {'numbers': [1, 2, 3]}

(This script is complete, it should run "as is")

Dotenv (.env) support

Note

dotenv file parsing requires python-dotenv to be installed. This can be done with either pip install python-dotenv or pip install pydantic[dotenv].

Dotenv files (generally named .env) are a common pattern that make it easy to use environment variables in a platform-independent manner.

A dotenv file follows the same general principles of all environment variables, and looks something like:

# ignore comment
ENVIRONMENT="production"
REDIS_ADDRESS=localhost:6379
MEANING_OF_LIFE=42
MY_VAR='Hello world'

Once you have your .env file filled with variables, pydantic supports loading it in two ways:

1. setting env_file (and env_file_encoding if you don't want the default encoding of your OS) on Config in a BaseSettings class:

class Settings(BaseSettings):
    ...

    class Config:
        env_file = '.env'
        env_file_encoding = 'utf-8'

2. instantiating a BaseSettings derived class with the _env_file keyword argument (and the _env_file_encoding if needed):

settings = Settings(_env_file='prod.env', _env_file_encoding='utf-8')

In either case, the value of the passed argument can be any valid path or filename, either absolute or relative to the current working directory. From there, pydantic will handle everything for you by loading in your variables and validating them.

Note

If a filename is specified for env_file, Pydantic will only check the current working directory and won't check any parent directories for the .env file.

Even when using a dotenv file, pydantic will still read environment variables as well as the dotenv file, environment variables will always take priority over values loaded from a dotenv file.

Passing a file path via the _env_file keyword argument on instantiation (method 2) will override the value (if any) set on the Config class. If the above snippets were used in conjunction, prod.env would be loaded while .env would be ignored.

If you need to load multiple dotenv files, you can pass the file paths as a list or tuple.

Later files in the list/tuple will take priority over earlier files.

from pydantic import BaseSettings

class Settings(BaseSettings):
    ...

    class Config:
        # `.env.prod` takes priority over `.env`
        env_file = '.env', '.env.prod'

You can also use the keyword argument override to tell Pydantic not to load any file at all (even if one is set in the Config class) by passing None as the instantiation keyword argument, e.g. settings = Settings(_env_file=None).

Because python-dotenv is used to parse the file, bash-like semantics such as export can be used which (depending on your OS and environment) may allow your dotenv file to also be used with source, see python-dotenv's documentation for more details.

Secret Support

Placing secret values in files is a common pattern to provide sensitive configuration to an application.

A secret file follows the same principal as a dotenv file except it only contains a single value and the file name is used as the key. A secret file will look like the following:

/var/run/database_password:

super_secret_database_password

Once you have your secret files, pydantic supports loading it in two ways:

1. setting secrets_dir on Config in a BaseSettings class to the directory where your secret files are stored:

class Settings(BaseSettings):
    ...
    database_password: str

    class Config:
        secrets_dir = '/var/run'

2. instantiating a BaseSettings derived class with the _secrets_dir keyword argument:

settings = Settings(_secrets_dir='/var/run')

In either case, the value of the passed argument can be any valid directory, either absolute or relative to the current working directory. Note that a non existent directory will only generate a warning. From there, pydantic will handle everything for you by loading in your variables and validating them.

Even when using a secrets directory, pydantic will still read environment variables from a dotenv file or the environment, a dotenv file and environment variables will always take priority over values loaded from the secrets directory.

Passing a file path via the _secrets_dir keyword argument on instantiation (method 2) will override the value (if any) set on the Config class.

Use Case: Docker Secrets

Docker Secrets can be used to provide sensitive configuration to an application running in a Docker container. To use these secrets in a pydantic application the process is simple. More information regarding creating, managing and using secrets in Docker see the official Docker documentation.

First, define your Settings

class Settings(BaseSettings):
    my_secret_data: str

    class Config:
        secrets_dir = '/run/secrets'

Note

By default Docker uses /run/secrets as the target mount point. If you want to use a different location, change Config.secrets_dir accordingly.

Then, create your secret via the Docker CLI

printf "This is a secret" | docker secret create my_secret_data -

Last, run your application inside a Docker container and supply your newly created secret

docker service create --name pydantic-with-secrets --secret my_secret_data pydantic-app:latest

Field value priority

In the case where a value is specified for the same Settings field in multiple ways, the selected value is determined as follows (in descending order of priority):

1. Arguments passed to the Settings class initialiser.
2. Environment variables, e.g. my_prefix_special_function as described above.
3. Variables loaded from a dotenv (.env) file.
4. Variables loaded from the secrets directory.
5. The default field values for the Settings model.

Customise settings sources

If the default order of priority doesn't match your needs, it's possible to change it by overriding the customise_sources method on the Config class of your Settings .

customise_sources takes three callables as arguments and returns any number of callables as a tuple. In turn these callables are called to build the inputs to the fields of the settings class.

Each callable should take an instance of the settings class as its sole argument and return a dict.

Changing Priority

The order of the returned callables decides the priority of inputs; first item is the highest priority.

Python 3.7 and abovePython 3.9 and above

from typing import Tuple
from pydantic import BaseSettings, PostgresDsn
from pydantic.env_settings import SettingsSourceCallable


class Settings(BaseSettings):
    database_dsn: PostgresDsn

    class Config:
        @classmethod
        def customise_sources(
            cls,
            init_settings: SettingsSourceCallable,
            env_settings: SettingsSourceCallable,
            file_secret_settings: SettingsSourceCallable,
        ) -> Tuple[SettingsSourceCallable, ...]:
            return env_settings, init_settings, file_secret_settings


print(Settings(database_dsn='postgres://postgres@localhost:5432/kwargs_db'))
#> database_dsn=PostgresDsn('postgres://postgres@localhost:5432/env_db', )

from pydantic import BaseSettings, PostgresDsn
from pydantic.env_settings import SettingsSourceCallable


class Settings(BaseSettings):
    database_dsn: PostgresDsn

    class Config:
        @classmethod
        def customise_sources(
            cls,
            init_settings: SettingsSourceCallable,
            env_settings: SettingsSourceCallable,
            file_secret_settings: SettingsSourceCallable,
        ) -> tuple[SettingsSourceCallable, ...]:
            return env_settings, init_settings, file_secret_settings


print(Settings(database_dsn='postgres://postgres@localhost:5432/kwargs_db'))
#> database_dsn=PostgresDsn('postgres://postgres@localhost:5432/env_db', )

(This script is complete, it should run "as is")

By flipping env_settings and init_settings, environment variables now have precedence over __init__ kwargs.

Adding sources

As explained earlier, pydantic ships with multiples built-in settings sources. However, you may occasionally need to add your own custom sources, customise_sources makes this very easy:

Python 3.7 and abovePython 3.9 and above

import json
from pathlib import Path
from typing import Dict, Any

from pydantic import BaseSettings


def json_config_settings_source(settings: BaseSettings) -> Dict[str, Any]:
    """
    A simple settings source that loads variables from a JSON file
    at the project's root.

    Here we happen to choose to use the `env_file_encoding` from Config
    when reading `config.json`
    """
    encoding = settings.__config__.env_file_encoding
    return json.loads(Path('config.json').read_text(encoding))


class Settings(BaseSettings):
    foobar: str

    class Config:
        env_file_encoding = 'utf-8'

        @classmethod
        def customise_sources(
            cls,
            init_settings,
            env_settings,
            file_secret_settings,
        ):
            return (
                init_settings,
                json_config_settings_source,
                env_settings,
                file_secret_settings,
            )


print(Settings())
#> foobar='spam'

import json
from pathlib import Path
from typing import Any

from pydantic import BaseSettings


def json_config_settings_source(settings: BaseSettings) -> dict[str, Any]:
    """
    A simple settings source that loads variables from a JSON file
    at the project's root.

    Here we happen to choose to use the `env_file_encoding` from Config
    when reading `config.json`
    """
    encoding = settings.__config__.env_file_encoding
    return json.loads(Path('config.json').read_text(encoding))


class Settings(BaseSettings):
    foobar: str

    class Config:
        env_file_encoding = 'utf-8'

        @classmethod
        def customise_sources(
            cls,
            init_settings,
            env_settings,
            file_secret_settings,
        ):
            return (
                init_settings,
                json_config_settings_source,
                env_settings,
                file_secret_settings,
            )


print(Settings())
#> foobar='spam'

(This script is complete, it should run "as is")

Removing sources

You might also want to disable a source:

Python 3.7 and abovePython 3.9 and above

from typing import Tuple

from pydantic import BaseSettings
from pydantic.env_settings import SettingsSourceCallable


class Settings(BaseSettings):
    my_api_key: str

    class Config:
        @classmethod
        def customise_sources(
            cls,
            init_settings: SettingsSourceCallable,
            env_settings: SettingsSourceCallable,
            file_secret_settings: SettingsSourceCallable,
        ) -> Tuple[SettingsSourceCallable, ...]:
            # here we choose to ignore arguments from init_settings
            return env_settings, file_secret_settings


print(Settings(my_api_key='this is ignored'))
#> my_api_key='xxx'

from pydantic import BaseSettings
from pydantic.env_settings import SettingsSourceCallable


class Settings(BaseSettings):
    my_api_key: str

    class Config:
        @classmethod
        def customise_sources(
            cls,
            init_settings: SettingsSourceCallable,
            env_settings: SettingsSourceCallable,
            file_secret_settings: SettingsSourceCallable,
        ) -> tuple[SettingsSourceCallable, ...]:
            # here we choose to ignore arguments from init_settings
            return env_settings, file_secret_settings


print(Settings(my_api_key='this is ignored'))
#> my_api_key='xxx'

(This script requires MY_API_KEY env variable to be set, e.g. export MY_API_KEY=xxx)