One master key, four derived keys
APP_ENC_KEY is the only key you supply. Generate it with make generate-master-key, which produces a base64-encoded 32-byte value. Set the same value on the Manager and on the Worker.
Fetcher never uses that key directly. It expands it with HKDF-SHA256 (RFC 5869) into four independent keys, one per purpose.
Separation is the point. A consumer that holds the external key can check a result signature. It cannot decrypt a stored credential, and it cannot forge a message between the two services.
Key version and rotation
APP_ENC_KEY_VERSION labels the key that is in force. Every connection record stores the version that encrypted its password, so an operator can tell which key a record belongs to. Increment the version when you change the master key.
Fetcher does not retain prior credential keys. After changing the master key and version, provide each known password again to update and re-encrypt the connection under the current version, or deliberately delete and recreate the connection. Do not expect automatic key migration.
The external HMAC key changes with the master key. Derive the new key and give it to every consumer that checks signatures. Earlier results verify against the earlier key.
Generate the external key with make derive-key KEY="<your-base64-master-key>". The tool also reads APP_ENC_KEY from the environment or the key from standard input, and it prints a 64-character hex key.
Credentials at rest
A datasource password never reaches MongoDB in the clear. The Manager encrypts it with AES-256-GCM under the derived credential key, then stores the ciphertext and the key version. In single-tenant mode, fixed internal datasources load from
DATASOURCE_{NAME}_* environment variables. In multi-tenant mode, Fetcher resolves them per tenant through Tenant Manager. In both modes they are internal, in-memory connections with an empty key version and no MongoDB connection record.
Results at rest
The Worker protects a stored result in two steps:
- It signs the plaintext JSON with HMAC-SHA256 under the derived external key, and records the algorithm and the signature with the result.
- It encrypts the payload with AES-GCM under the derived storage key, with a fresh 12-byte random nonce, and stores the result base64-encoded.
scripts/crypto/derive-key/verification-guide.md.
Direct mode returns the rows inline without encryption. The engine reports them as plaintext and attaches a SHA-256 digest over the exact bytes.
Signed messages between the services
Every RabbitMQ message the Manager publishes to the Worker carries an HMAC-SHA256 signature under the derived internal key. The signature covers the timestamp, signature version, tenant identifier, exchange, routing key, and message body. Context binding prevents a signed message from being retargeted to another tenant or route. Freshness separately rejects messages older than five minutes and messages more than 30 seconds in the future. Context binding alone does not prevent replay of the same message in the same context during that window. By default, the Worker requires this canonical tenant- and route-bound envelope.
RABBITMQ_ALLOW_LEGACY_BODY_SIGNATURE_FALLBACK=true is a temporary migration setting that also accepts a legacy body-only signature so already-queued messages can drain. The publisher strips caller-supplied security headers before it signs, and the signer compares signatures in constant time.
Datasource host validation
A tenant that registers its own connection could point it at your internal network. With
MULTI_TENANT_ENABLED=true, Fetcher checks the host before it connects.
Validation runs at two layers:
- At request parse. Fetcher rejects an IP literal in a blocked range, with no DNS lookup.
- In the datasource factory. Fetcher checks the hostname against a blocklist that covers
localhost, cloud metadata names, and the.local,.internal, and.cluster.localsuffixes. It then resolves the hostname and checks every address it gets back.
lib-commons SSRF dependency. Internal datasources are deliberately exempt: in single-tenant mode they come from the operator environment, and in multi-tenant mode Tenant Manager resolves them.
Tenant isolation
The Engine scopes every operation by tenant identifier, and that identifier is the sole Engine isolation boundary. A malformed tenant identifier fails before Fetcher touches any resource. Fetcher also enforces product ownership at the host layer: an external connection belongs to one product and a job’s
metadata.source must match it; internal datasources are the exception.
In multi-tenant mode, Fetcher resolves metadata resources from the tenant context supplied by the tenant middleware. Worker consumers use their authoritative tenant context. Do not attribute tenant resolution specifically to JWT claims unless the middleware owner documents that contract. Access fails closed: a request with no resolved tenant database returns an error instead of reading a shared database.
Authentication and probe surfaces
PLUGIN_AUTH_ENABLED=true puts the Access Manager middleware in front of the API. Requests then carry a bearer token, and Fetcher authorizes each operation against a resource and an action.
/health, /readyz, /readyz/tenant/:id, /metrics, and /version mount before that middleware, so Kubernetes and load-balancer probes stay unauthenticated.
Errors never leak connection material
Fetcher discards the raw driver error at the engine boundary and returns a fixed message in its place. A raw driver error can embed a DSN or a credential, so a caller sees
failed to connect to datasource instead of the string the driver produced.
Failures arrive classified into stable categories — validation, unauthorized, forbidden, limit exceeded, connect, timeout, and others — so a host maps them to its own status codes without parsing text.
Next steps
Configuration
Every environment variable, per component.
Deployment
Dependencies, storage retention, scaling, and startup checks.
Observability
Probes, drain behavior, metrics, and tracing.
Connections
Register, test, and use a connection.