Environment Reference

ETLPlus prefers environment-injected runtime configuration over repository-local files.

Use profile.env for documented defaults and placeholders, then override those values from the invoking shell, CI job, container runtime, or scheduler.

Precedence

For config substitution, ETLPlus currently resolves values in this order:

  1. vars from the pipeline config

  2. profile.env from the pipeline config

  3. environment variables provided by the invoking process

That means external environment variables override profile.env.

Runtime Knobs

Variable

Purpose

Notes

ETLPLUS_LOG_LEVEL

Override the default CLI logging level.

Applies across supported commands; --quiet and --verbose still remain available.

ETLPLUS_STATE_DIR

Change the machine-local ETLPlus state directory.

Affects local run history and scheduler state.

ETLPLUS_TELEMETRY_ENABLED

Enable or disable the optional telemetry bridge.

Use true or false.

ETLPLUS_TELEMETRY_EXPORTER

Select the telemetry exporter bridge.

Current supported values are opentelemetry and none.

ETLPLUS_TELEMETRY_SERVICE_NAME

Set the service name used by the telemetry bridge.

Useful when the same workstation or runner hosts multiple ETLPlus workloads.

Secret References

For pipeline config values that should not be committed, ETLPlus supports additive secret:... tokens during config substitution.

  • Use secret:NAME for the stable environment-first form. It resolves NAME from the effective runtime environment.

  • Use secret:env:NAME when you want the provider name to be explicit. It is equivalent to secret:NAME.

  • Use secret:file:path.to.key only for local-development compatibility. It reads a JSON or YAML mapping file selected by ETLPLUS_SECRETS_FILE.

Encrypted local secret files and cloud secret-manager backends are intentionally deferred until the provider interface is stable. In shared, scheduled, or production-like deployments, prefer environment injection or platform identity over repository-local secret files.

Storage and Provider Credentials

etlplus check --readiness --config pipeline.yml now reports provider-specific environment gaps for common remote-storage and cloud-database paths.

Use this page as the default runtime posture for shared or scheduled pipelines: keep provider metadata in config, inject secrets through the process environment or platform identity, and reserve local files or localhost services for development fixtures.

AWS S3

Common credential-chain hints checked by readiness:

  • AWS_ACCESS_KEY_ID

  • AWS_SECRET_ACCESS_KEY

  • AWS_SESSION_TOKEN

  • AWS_PROFILE

  • AWS_DEFAULT_PROFILE

  • AWS_ROLE_ARN

  • AWS_WEB_IDENTITY_TOKEN_FILE

  • AWS_CONTAINER_CREDENTIALS_RELATIVE_URI

  • AWS_CONTAINER_CREDENTIALS_FULL_URI

  • AWS_SHARED_CREDENTIALS_FILE

  • AWS_CONFIG_FILE

Typical approaches:

  • Explicit key pair via AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY

  • Shared config/profile via AWS_PROFILE

  • Workload or instance credentials via the normal AWS SDK chain

Azure Blob Storage and ADLS Gen2

Combined Azure environment hint set used by readiness reporting:

  • AZURE_STORAGE_CONNECTION_STRING

  • AZURE_STORAGE_ACCOUNT_URL

  • AZURE_STORAGE_CREDENTIAL

Bootstrap variables checked by readiness:

  • AZURE_STORAGE_CONNECTION_STRING

  • AZURE_STORAGE_ACCOUNT_URL

Explicit credential variable checked by readiness:

  • AZURE_STORAGE_CREDENTIAL

If your azure-blob://... or abfs://... path already embeds the account host, ETLPlus can use that as bootstrap context. Readiness still evaluates Azure in two phases: bootstrap first, then explicit credentials for non-public targets.

BigQuery

Install support first:

pip install "etlplus[database-bigquery]"

Common Google Cloud credential hints checked by readiness:

  • GOOGLE_APPLICATION_CREDENTIALS

  • GOOGLE_CLOUD_PROJECT

  • GCLOUD_PROJECT

  • CLOUDSDK_CONFIG

Recommended posture:

  • Keep connector metadata in the pipeline config with provider: bigquery, project, and dataset

  • Inject credentials through Application Default Credentials or other platform identity mechanisms

Snowflake

Install support first:

pip install "etlplus[database-snowflake]"

Common Snowflake credential hints checked by readiness:

  • SNOWFLAKE_USER

  • SNOWFLAKE_PASSWORD

  • SNOWFLAKE_AUTHENTICATOR

  • SNOWFLAKE_PRIVATE_KEY_PATH

  • SNOWFLAKE_PRIVATE_KEY

Recommended posture:

  • Keep account, database, schema, and warehouse metadata in the connector

  • Inject user/password, key-based auth, or SSO-specific environment at runtime

  • Use a full connection_string when you already have a managed DSN surface

Example Shell Setup

export AWS_PROFILE=etlplus-dev
export AZURE_STORAGE_ACCOUNT_URL="https://acct.blob.core.windows.net"
export AZURE_STORAGE_CREDENTIAL="${AZURE_STORAGE_CREDENTIAL}"
export GOOGLE_APPLICATION_CREDENTIALS="$HOME/.config/gcloud/etlplus.json"
export SNOWFLAKE_USER="etlplus"
export SNOWFLAKE_PASSWORD="${SNOWFLAKE_PASSWORD}"
etlplus check --readiness --config examples/configs/pipeline.yml