Credstash values with Pydantic Settings Management

At JBS Dev we value the lessons taught in The 12 Factor App guide. In particular, factor III states that config should be contained in the environment. Using this approach let us separate config from code, manage secrets appropriately, and mange local dev environment config in exactly the same way as live environments. Because we value encryption for our secrets and deploy primarily to AWS, we use Credstash to store secrets in DynamoDB with encryption from KMS. And we have found the Pydantic settings module to be useful for building strongly typed configuration models which read from the environment.

The code below uses a preprocessing validator to replace environment values like %%credstash.key%% with the value of that key from credstash. In this way we can refer to any secret value by name only and defer retrieval of the value to run time. Additionally using the SecretStr field type provided by Pydantic allows us to be sure that the value will not accidentally appear in logs or serialization.

import re

from credstash import getSecret
from pydantic import validator, BaseSettings, SecretStr


# A value wrapped in %% will be looked up in Credstash.
# EG: %%app.db_pass.dev%%
CREDSTASH_KEY_RE = re.compile(r'%%([^$]*)%%')


def credstash_lookup(cls, value):
    try:
        # Because values can be populated by kwargs to an instance of the Settings
        # class, we need to ensure this value is a string before doing regex operations.
        if isinstance(value, str):
            value = CREDSTASH_KEY_RE.sub(
                # When the regex matches, the first capture group
                # is sent to this callable, which looks up the
                # replacement value.
                lambda m: getSecret(m.group(1)),
                value,
            )
    except Exception:
        raise ValueError(f'Failed to fetch {value} from credstash')
    return value


class Settings(BaseSettings):
    db_pass: SecretStr = None
    db_host: str = None
    db_port: int

    # Include this line in your Settings class to apply the
    # credstash lookup on all fields.
    _credstash_lookup = validator('*', pre=True)(credstash_lookup)