> ## Documentation Index
> Fetch the complete documentation index at: https://docs.lerian.studio/llms.txt
> Use this file to discover all available pages before exploring further.
# Fetcher configuration
> The environment variables that shape a Fetcher deployment — encryption, MongoDB, RabbitMQ, object storage, multi-tenancy, telemetry, and internal datasources.
Fetcher configures itself entirely from the environment. Each service reads its own set. The Manager and the Worker share most variables, and each one owns a few of its own.
For local work, `make set-env` copies each component's `.env.example` to `.env`. In production, set the variables through your orchestrator.
**Both services need the same `APP_ENC_KEY`.** The Worker uses it to decrypt datasource credentials and to check the signature on every message the Manager sends. A missing or short key stops the process at startup. The log reads `master key too short: got 0 bytes, minimum 32 required`, and the service never binds a port.
## Both services
***
### Application
| Variable | Description | Default | Required |
| ------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | -------- |
| `APP_ENC_KEY` | Base64-encoded 32-byte master key. Generate it with `make generate-master-key`. Fetcher derives four independent keys from it. | — | Yes |
| `APP_ENC_KEY_VERSION` | Key version recorded against every credential Fetcher encrypts. Increment it when you change the master key. | `1` | No |
| `ENV_NAME` | Environment label. It also selects the logger profile: `production`, `staging`, `uat`, `development`, or `local`. An unset or unknown value selects the `local` profile. The shipped example sets `development`. | — | No |
| `LOG_LEVEL` | Log verbosity. Unset, it follows the environment profile: `debug` for `local` and `development`, `info` for the rest. | — | No |
| `VERSION` | Version string reported by `/version`. The shipped example sets `v1.0.0`. | `0.0.0` | No |
| `DEPLOYMENT_MODE` | `saas`, `byoc`, or `local`. Tags the `/readyz` response and drives SaaS TLS enforcement. `local` also relaxes license enforcement. | `local` | No |
| `ALLOW_INSECURE_TLS` | Permits plaintext connections to MongoDB, Redis, PostgreSQL, and RabbitMQ. Leave it unset in production. | unset | No |
| `READYZ_DRAIN_DELAY_SEC` | Drain window after `SIGTERM`, in seconds. Minimum `1`. | `12` | No |
### MongoDB
MongoDB holds Fetcher's own metadata: connection records and job records.
| Variable | Description | Default | Required |
| ------------------- | ------------------------------------------------------------------------------------------------------------------------------- | ------- | -------- |
| `MONGO_URI` | Connection scheme: `mongodb` or `mongodb+srv`. Fetcher composes the connection URI from it. The shipped example sets `mongodb`. | — | Yes |
| `MONGO_HOST` | MongoDB host. | — | Yes |
| `MONGO_PORT` | MongoDB port. | — | Yes |
| `MONGO_NAME` | MongoDB database name. Repository access resolves this name in lower case. | — | Yes |
| `MONGO_USER` | Database user. | — | Yes |
| `MONGO_PASSWORD` | Database password. Fetcher URL-escapes it. | — | Yes |
| `MONGO_PARAMETERS` | Extra query-string parameters for the URI. | — | No |
| `MONGO_TLS_CA_CERT` | Base64-encoded PEM CA certificate for TLS. | — | No |
### RabbitMQ
| Variable | Description | Default | Required |
| ----------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | -------- |
| `RABBITMQ_URI` | AMQP scheme. Fetcher composes the broker URL from it. The shipped example sets `amqp`. | — | Yes |
| `RABBITMQ_HOST` | Broker host. | — | Yes |
| `RABBITMQ_PORT_AMQP` | AMQP port. | — | Yes |
| `RABBITMQ_PORT_HOST` | Management port. | — | No |
| `RABBITMQ_DEFAULT_USER` | Broker user. | — | Yes |
| `RABBITMQ_DEFAULT_PASS` | Broker password. | — | Yes |
| `RABBITMQ_HEALTH_CHECK_URL` | Management URL the readiness probe calls. The shipped example builds it from `RABBITMQ_HOST` and `RABBITMQ_PORT_HOST`. | — | No |
| `RABBITMQ_FETCHER_WORK_QUEUE` | Queue that carries extraction jobs from the Manager to the Worker. Both services must name the same queue. The shipped example uses `fetcher.extract-external-data.queue`. | — | Yes |
| `RABBITMQ_TLS` | Enables TLS to the broker. | `false` | No |
### Multi-tenancy
| Variable | Description | Default | Required |
| ------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | ------------------ |
| `MULTI_TENANT_ENABLED` | Turns on database-per-tenant isolation. It also activates the host-safety guard on tenant-supplied datasource hosts. | `false` | No |
| `MULTI_TENANT_URL` | Tenant Manager service URL. | — | With multi-tenancy |
| `MULTI_TENANT_SERVICE_API_KEY` | Default API key for Tenant Manager lookups. | — | With multi-tenancy |
| `MULTI_TENANT_SERVICE_API_KEY_` | Per-service key. The suffix is the service name, upper-cased, with hyphens turned into underscores (`plugin-crm` becomes `PLUGIN_CRM`). It falls back to the default key. | — | No |
| `MULTI_TENANT_REDIS_HOST` | Redis host for tenant Pub/Sub. | — | With multi-tenancy |
| `MULTI_TENANT_REDIS_PORT` | Tenant Pub/Sub Redis port. | `6379` | No |
| `MULTI_TENANT_REDIS_PASSWORD` | Tenant Pub/Sub Redis password. | — | No |
| `MULTI_TENANT_REDIS_TLS` | TLS for tenant Pub/Sub Redis. The client trusts the runtime trust store, so install a private CA there. | `false` | No |
| `MULTI_TENANT_MAX_TENANT_POOLS` | Ceiling on concurrent per-tenant connection pools. | `100` | No |
| `MULTI_TENANT_IDLE_TIMEOUT_SEC` | Idle timeout before a tenant pool is released. | `300` | No |
| `MULTI_TENANT_CACHE_TTL_SEC` | TTL of the cached tenant configuration. The shipped example sets `120`. | — | No |
| `MULTI_TENANT_TIMEOUT` | Timeout for Tenant Manager calls, in seconds. | `30` | No |
| `MULTI_TENANT_CIRCUIT_BREAKER_THRESHOLD` | Consecutive failures before the tenant circuit breaker opens. | `5` | No |
| `MULTI_TENANT_CIRCUIT_BREAKER_TIMEOUT_SEC` | How long the breaker stays open before a reset attempt. | `30` | No |
| `MULTI_TENANT_ALLOW_INSECURE_HTTP` | Permits plaintext HTTP to the Tenant Manager. | `false` | No |
**Startup stops in multi-tenant mode.** `MULTI_TENANT_ENABLED=true` requires `MULTI_TENANT_URL`, `MULTI_TENANT_SERVICE_API_KEY`, and `MULTI_TENANT_REDIS_HOST`. A missing value aborts startup on both services, and the error names the variable. Two `MULTI_TENANT_SERVICE_API_KEY_` variables that normalize to the same token also abort startup, and the error names the token. One service's credential never silently overwrites another's.
### Telemetry
| Variable | Description | Default | Required |
| -------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------- | ------- | -------------- |
| `ENABLE_TELEMETRY` | Turns on OpenTelemetry traces and metrics. | `false` | No |
| `OTEL_EXPORTER_OTLP_ENDPOINT` | OTLP collector endpoint. | — | With telemetry |
| `OTEL_INSECURE_EXPORTER` | Allows a plaintext exporter connection. | `false` | No |
| `OTEL_RESOURCE_SERVICE_NAME` | Service name in telemetry. The shipped examples set `fetcher` for the Manager and `fetcher-worker` for the Worker. | — | No |
| `OTEL_LIBRARY_NAME` | Instrumentation library name. The shipped example sets `github.com/LerianStudio/fetcher/v2`. | — | No |
| `OTEL_RESOURCE_SERVICE_VERSION` | Service version attribute. `/readyz` reports this value, and reports `unknown` when it is unset. The shipped example points it at `VERSION`. | — | No |
| `OTEL_RESOURCE_DEPLOYMENT_ENVIRONMENT` | Deployment environment attribute. The shipped example points it at `ENV_NAME`. | — | No |
## Manager only
***
| Variable | Description | Default | Required |
| --------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | --------- |
| `SERVER_ADDRESS` | Listen address for the HTTP API. The same listener serves `/health`, `/readyz`, `/metrics`, and `/version`. The shipped example builds it from `SERVER_PORT`, which it sets to `4006`. | — | Yes |
| `SWAGGER_ENABLED` | Serves the API reference at `/swagger/docs` and the OpenAPI 3.1 contract at `/swagger/openapi.json` and `/swagger/openapi.yaml`. | `false` | No |
| `PLUGIN_AUTH_ENABLED` | Turns on Access Manager authentication. Multi-tenant mode requires it. | `false` | No |
| `PLUGIN_AUTH_ADDRESS` | Access Manager address. | — | With auth |
| `SCHEMA_CACHE_TTL_SECONDS` | TTL for a cached datasource schema. An empty or invalid value falls back to 5 minutes. | `300` | No |
| `REDIS_HOST` | Valkey or Redis host. It backs the schema cache and the rate limiter on the connection test. | — | No |
| `REDIS_PORT` | Redis port. | — | No |
| `REDIS_PASSWORD` | Redis password. | — | No |
| `REDIS_DB` | Redis database index. | `0` | No |
| `REDIS_TLS` | Enables TLS to Redis. | `false` | No |
| `REDIS_CA_CERT` | Base64-encoded PEM CA used when `REDIS_TLS=true`. | — | No |
| `MAX_PAGINATION_LIMIT` | Ceiling on the `limit` query parameter of every list operation. | `100` | No |
| `MAX_PAGINATION_MONTH_DATE_RANGE` | Ceiling on the creation-date window of every list operation, in months. | `1` | No |
**The router refuses to build on two security mismatches.** It reports `tenant middleware requires effective authentication` when multi-tenant mode runs with authentication off. It reports `auth middleware is enabled but its address is empty` when `PLUGIN_AUTH_ENABLED=true` and `PLUGIN_AUTH_ADDRESS` is blank. In both cases the Manager does not start.
## Worker only
***
### Runtime and events
| Variable | Description | Default | Required |
| -------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------- | ----------- | -------- |
| `HEALTH_PORT` | Port for the Worker's health micro-server. It serves `/health`, `/readyz`, `/readyz/tenant/:id`, and `/metrics`. The Worker has no other HTTP server. | `4007` | No |
| `RABBITMQ_NUMBERS_OF_WORKERS` | How many jobs the Worker processes in parallel. | `5` | No |
| `MONGO_MAX_POOL_SIZE` | Mongo client pool size for the metadata store. The shipped example sets `1000`. | `100` | No |
| `STREAMING_ENABLED` | Must be `true`. Job terminal events are a product contract, so the Worker refuses to start without streaming. | `false` | Yes |
| `STREAMING_BROKERS` | Bootstrap broker list for the streaming layer, as comma-separated `host:port` pairs. The shipped example sets `localhost:9092`. | — | Yes |
| `STREAMING_CLOUDEVENTS_SOURCE` | CloudEvents source stamped on emitted Worker events. The shipped example sets `//lerian.fetcher/worker`. | — | Yes |
| `STREAMING_CLIENT_ID` | Streaming client identifier. | — | No |
| `STREAMING_COMPRESSION` | Compression codec: `snappy`, `lz4`, `zstd`, `gzip`, or `none`. | `lz4` | No |
| `STREAMING_REQUIRED_ACKS` | Producer acknowledgement level: `all`, `leader`, or `none`. | `all` | No |
| `RABBITMQ_JOB_EVENTS_EXCHANGE` | Exchange that carries `job.completed` and `job.failed`. The shipped example uses `fetcher.job.events`. | — | Yes |
| `MULTI_TENANT_RECONCILE_INTERVAL_SEC` | Interval at which the Worker reconciles per-tenant consumers against the active tenants. | `60` | No |
| `ENGINE_MAX_RESULT_BYTES` | Overrides the engine's serialized-result ceiling, as a byte count. Zero or negative keeps the default. | `268435456` | No |
| `CRYPTO_ENCRYPT_SECRET_KEY_PLUGIN_CRM` | Encryption key for the CRM compatibility extraction path. | — | No |
| `CRYPTO_HASH_SECRET_KEY_PLUGIN_CRM` | Hash key for the CRM compatibility extraction path. | — | No |
**The Worker fails closed on event configuration.** `STREAMING_ENABLED` unset or `false` aborts startup with `STREAMING_ENABLED=true is required for mandatory job event notifications`. A blank `RABBITMQ_JOB_EVENTS_EXCHANGE`, an empty `STREAMING_BROKERS`, and an empty `STREAMING_CLOUDEVENTS_SOURCE` each abort startup the same way, as does a `STREAMING_COMPRESSION` or `STREAMING_REQUIRED_ACKS` outside its accepted set. The Worker never falls back to a silent no-op emitter.
### Object storage
The Worker writes every stored result to S3-compatible object storage. The endpoint scheme controls TLS, so `http://` turns it off.
| Variable | Description | Default | Required |
| ------------------------------- | --------------------------------------------------------------------------------------------------------------------------- | ----------- | -------- |
| `OBJECT_STORAGE_ENDPOINT` | S3-compatible endpoint, for example `http://minio:9000`. Leave it unset to target AWS S3. | — | No |
| `OBJECT_STORAGE_REGION` | S3 region. | `us-east-1` | No |
| `OBJECT_STORAGE_BUCKET` | Bucket for stored results. The Worker does not start without it. | — | Yes |
| `OBJECT_STORAGE_KEY_PREFIX` | Key prefix applied to stored objects. | — | No |
| `OBJECT_STORAGE_ACCESS_KEY_ID` | Static access key ID. Set it with `OBJECT_STORAGE_SECRET_KEY`, or leave both unset to use the ambient AWS credential chain. | — | No |
| `OBJECT_STORAGE_SECRET_KEY` | Static secret access key. Set it with `OBJECT_STORAGE_ACCESS_KEY_ID`. Supplying only one of the two aborts startup. | — | No |
| `OBJECT_STORAGE_USE_PATH_STYLE` | Path-style addressing. Set it to `true` for MinIO and SeaweedFS. | `false` | No |
Set result expiry with a lifecycle policy on the bucket. See [Deployment](/en/fetcher/fetcher-deployment).
### Message signature compatibility
| Variable | Description | Default | Required |
| ----------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | -------- |
| `RABBITMQ_ALLOW_LEGACY_BODY_SIGNATURE_FALLBACK` | Compatibility flag for a rolling upgrade. It also accepts body-only signatures on consumed messages. A body-only signature does not bind the message to its tenant, job, exchange, or routing key. It ships off. Turn it on only while a fleet rolls forward, then turn it off. | `false` | No |
## Internal datasources
***
Internal datasources are the databases of other Lerian products. Fetcher resolves them without an API-registered connection. You declare each one with a `DATASOURCE_{NAME}_*` group, where `{NAME}` is a prefix you choose.
The accepted `_CONFIG_NAME` values are a fixed registry: `midaz_onboarding`, `midaz_transaction`, and `plugin_crm`. Fetcher skips any other name and logs a warning that names both the rejected value and the accepted set. For your own databases, register a connection through the API instead.
| Variable | Description | Example |
| ------------------------------- | ------------------------------------------------------ | ------------------------- |
| `DATASOURCE_{NAME}_CONFIG_NAME` | Registry name of the datasource. Required. | `midaz_onboarding` |
| `DATASOURCE_{NAME}_TYPE` | Datasource type, matched case-insensitively. Required. | `POSTGRESQL` |
| `DATASOURCE_{NAME}_HOST` | Host. Required. | `db.internal.example.com` |
| `DATASOURCE_{NAME}_PORT` | Port. Required. | `5432` |
| `DATASOURCE_{NAME}_DATABASE` | Database name. Required. | `onboarding` |
| `DATASOURCE_{NAME}_USER` | Username. | `fetcher_ro` |
| `DATASOURCE_{NAME}_PASSWORD` | Password. Supply it from your secret manager. | — |
| `DATASOURCE_{NAME}_SSLMODE` | TLS mode. This is the TLS control on this path. | `require` |
| `DATASOURCE_{NAME}_OPTIONS` | Query-string options. MongoDB only. | `authSource=admin` |
**A bad value skips the datasource, loudly.** An invalid `_TYPE`, an invalid `_SSLMODE`, or a missing `_HOST` or `_DATABASE` makes Fetcher skip that datasource and log a warning that names the config name and the offending value. Fetcher never downgrades a TLS setting to reach a database.
## Next steps
***
Dependencies, queues, storage, scaling, and startup checks.
Master key, derived keys, signing, and host validation.
Probes, drain behavior, metrics, and tracing.
A first extraction, with or without infrastructure.