The environment variables that shape a Fetcher deployment — encryption, MongoDB, RabbitMQ, object storage, multi-tenancy, telemetry, and internal datasources.
Fetcher configures itself entirely from the environment. Each service reads its own set. The Manager and the Worker share most variables, and each one owns a few of its own.For local work, make set-env copies each component’s .env.example to .env. In production, set the variables through your orchestrator.
Both services need the same APP_ENC_KEY. The Worker uses it to decrypt datasource credentials and to check the signature on every message the Manager sends. A missing or short key stops the process at startup. The log reads master key too short: got 0 bytes, minimum 32 required, and the service never binds a port.
Base64-encoded 32-byte master key. Generate it with make generate-master-key. Fetcher derives four independent keys from it.
—
Yes
APP_ENC_KEY_VERSION
Key version recorded against every credential Fetcher encrypts. Increment it when you change the master key.
1
No
ENV_NAME
Environment label. It also selects the logger profile: production, staging, uat, development, or local. An unset or unknown value selects the local profile. The shipped example sets development.
—
No
LOG_LEVEL
Log verbosity. Unset, it follows the environment profile: debug for local and development, info for the rest.
—
No
VERSION
Version string reported by /version. The shipped example sets v1.0.0.
0.0.0
No
DEPLOYMENT_MODE
saas, byoc, or local. Tags the /readyz response and drives SaaS TLS enforcement. local also relaxes license enforcement.
local
No
ALLOW_INSECURE_TLS
Permits plaintext connections to MongoDB, Redis, PostgreSQL, and RabbitMQ. Leave it unset in production.
unset
No
READYZ_DRAIN_DELAY_SEC
Drain window after SIGTERM, in seconds. Minimum 1.
AMQP scheme. Fetcher composes the broker URL from it. The shipped example sets amqp.
—
Yes
RABBITMQ_HOST
Broker host.
—
Yes
RABBITMQ_PORT_AMQP
AMQP port.
—
Yes
RABBITMQ_PORT_HOST
Management port.
—
No
RABBITMQ_DEFAULT_USER
Broker user.
—
Yes
RABBITMQ_DEFAULT_PASS
Broker password.
—
Yes
RABBITMQ_HEALTH_CHECK_URL
Management URL the readiness probe calls. The shipped example builds it from RABBITMQ_HOST and RABBITMQ_PORT_HOST.
—
No
RABBITMQ_FETCHER_WORK_QUEUE
Queue that carries extraction jobs from the Manager to the Worker. Both services must name the same queue. The shipped example uses fetcher.extract-external-data.queue.
Turns on database-per-tenant isolation. It also activates the host-safety guard on tenant-supplied datasource hosts.
false
No
MULTI_TENANT_URL
Tenant Manager service URL.
—
With multi-tenancy
MULTI_TENANT_SERVICE_API_KEY
Default API key for Tenant Manager lookups.
—
With multi-tenancy
MULTI_TENANT_SERVICE_API_KEY_<SERVICE>
Per-service key. The suffix is the service name, upper-cased, with hyphens turned into underscores (plugin-crm becomes PLUGIN_CRM). It falls back to the default key.
—
No
MULTI_TENANT_REDIS_HOST
Redis host for tenant Pub/Sub.
—
With multi-tenancy
MULTI_TENANT_REDIS_PORT
Tenant Pub/Sub Redis port.
6379
No
MULTI_TENANT_REDIS_PASSWORD
Tenant Pub/Sub Redis password.
—
No
MULTI_TENANT_REDIS_TLS
TLS for tenant Pub/Sub Redis. The client trusts the runtime trust store, so install a private CA there.
false
No
MULTI_TENANT_MAX_TENANT_POOLS
Ceiling on concurrent per-tenant connection pools.
100
No
MULTI_TENANT_IDLE_TIMEOUT_SEC
Idle timeout before a tenant pool is released.
300
No
MULTI_TENANT_CACHE_TTL_SEC
TTL of the cached tenant configuration. The shipped example sets 120.
—
No
MULTI_TENANT_TIMEOUT
Timeout for Tenant Manager calls, in seconds.
30
No
MULTI_TENANT_CIRCUIT_BREAKER_THRESHOLD
Consecutive failures before the tenant circuit breaker opens.
5
No
MULTI_TENANT_CIRCUIT_BREAKER_TIMEOUT_SEC
How long the breaker stays open before a reset attempt.
30
No
MULTI_TENANT_ALLOW_INSECURE_HTTP
Permits plaintext HTTP to the Tenant Manager.
false
No
Startup stops in multi-tenant mode.MULTI_TENANT_ENABLED=true requires MULTI_TENANT_URL, MULTI_TENANT_SERVICE_API_KEY, and MULTI_TENANT_REDIS_HOST. A missing value aborts startup on both services, and the error names the variable. Two MULTI_TENANT_SERVICE_API_KEY_<SERVICE> variables that normalize to the same token also abort startup, and the error names the token. One service’s credential never silently overwrites another’s.
Listen address for the HTTP API. The same listener serves /health, /readyz, /metrics, and /version. The shipped example builds it from SERVER_PORT, which it sets to 4006.
—
Yes
SWAGGER_ENABLED
Serves the API reference at /swagger/docs and the OpenAPI 3.1 contract at /swagger/openapi.json and /swagger/openapi.yaml.
false
No
PLUGIN_AUTH_ENABLED
Turns on Access Manager authentication. Multi-tenant mode requires it.
false
No
PLUGIN_AUTH_ADDRESS
Access Manager address.
—
With auth
SCHEMA_CACHE_TTL_SECONDS
TTL for a cached datasource schema. An empty or invalid value falls back to 5 minutes.
300
No
REDIS_HOST
Valkey or Redis host. It backs the schema cache and the rate limiter on the connection test.
—
No
REDIS_PORT
Redis port.
—
No
REDIS_PASSWORD
Redis password.
—
No
REDIS_DB
Redis database index.
0
No
REDIS_TLS
Enables TLS to Redis.
false
No
REDIS_CA_CERT
Base64-encoded PEM CA used when REDIS_TLS=true.
—
No
MAX_PAGINATION_LIMIT
Ceiling on the limit query parameter of every list operation.
100
No
MAX_PAGINATION_MONTH_DATE_RANGE
Ceiling on the creation-date window of every list operation, in months.
1
No
The router refuses to build on two security mismatches. It reports tenant middleware requires effective authentication when multi-tenant mode runs with authentication off. It reports auth middleware is enabled but its address is empty when PLUGIN_AUTH_ENABLED=true and PLUGIN_AUTH_ADDRESS is blank. In both cases the Manager does not start.
Port for the Worker’s health micro-server. It serves /health, /readyz, /readyz/tenant/:id, and /metrics. The Worker has no other HTTP server.
4007
No
RABBITMQ_NUMBERS_OF_WORKERS
How many jobs the Worker processes in parallel.
5
No
MONGO_MAX_POOL_SIZE
Mongo client pool size for the metadata store. The shipped example sets 1000.
100
No
STREAMING_ENABLED
Must be true. Job terminal events are a product contract, so the Worker refuses to start without streaming.
false
Yes
STREAMING_BROKERS
Bootstrap broker list for the streaming layer, as comma-separated host:port pairs. The shipped example sets localhost:9092.
—
Yes
STREAMING_CLOUDEVENTS_SOURCE
CloudEvents source stamped on emitted Worker events. The shipped example sets //lerian.fetcher/worker.
—
Yes
STREAMING_CLIENT_ID
Streaming client identifier.
—
No
STREAMING_COMPRESSION
Compression codec: snappy, lz4, zstd, gzip, or none.
lz4
No
STREAMING_REQUIRED_ACKS
Producer acknowledgement level: all, leader, or none.
all
No
RABBITMQ_JOB_EVENTS_EXCHANGE
Exchange that carries job.completed and job.failed. The shipped example uses fetcher.job.events.
—
Yes
MULTI_TENANT_RECONCILE_INTERVAL_SEC
Interval at which the Worker reconciles per-tenant consumers against the active tenants.
60
No
ENGINE_MAX_RESULT_BYTES
Overrides the engine’s serialized-result ceiling, as a byte count. Zero or negative keeps the default.
268435456
No
CRYPTO_ENCRYPT_SECRET_KEY_PLUGIN_CRM
Encryption key for the CRM compatibility extraction path.
—
No
CRYPTO_HASH_SECRET_KEY_PLUGIN_CRM
Hash key for the CRM compatibility extraction path.
—
No
The Worker fails closed on event configuration.STREAMING_ENABLED unset or false aborts startup with STREAMING_ENABLED=true is required for mandatory job event notifications. A blank RABBITMQ_JOB_EVENTS_EXCHANGE, an empty STREAMING_BROKERS, and an empty STREAMING_CLOUDEVENTS_SOURCE each abort startup the same way, as does a STREAMING_COMPRESSION or STREAMING_REQUIRED_ACKS outside its accepted set. The Worker never falls back to a silent no-op emitter.
Compatibility flag for a rolling upgrade. It also accepts body-only signatures on consumed messages. A body-only signature does not bind the message to its tenant, job, exchange, or routing key. It ships off. Turn it on only while a fleet rolls forward, then turn it off.
Internal datasources are the databases of other Lerian products. Fetcher resolves them without an API-registered connection. You declare each one with a DATASOURCE_{NAME}_* group, where {NAME} is a prefix you choose.The accepted _CONFIG_NAME values are a fixed registry: midaz_onboarding, midaz_transaction, and plugin_crm. Fetcher skips any other name and logs a warning that names both the rejected value and the accepted set. For your own databases, register a connection through the API instead.
A bad value skips the datasource, loudly. An invalid _TYPE, an invalid _SSLMODE, or a missing _HOST or _DATABASE makes Fetcher skip that datasource and log a warning that names the config name and the offending value. Fetcher never downgrades a TLS setting to reach a database.
