> ## Documentation Index
> Fetch the complete documentation index at: https://docs.lerian.studio/llms.txt
> Use this file to discover all available pages before exploring further.
# Operating Streaming Hub
> Deploy Streaming Hub with Helm in BYOC, run out-of-band migrations, wire liveness and readiness probes, set essential config, and observe the hub via OTLP.
This page is for operators who run Streaming Hub in their own infrastructure (BYOC). It covers deployment, the configuration that matters, the health and shutdown contract, and how to observe the service.
## Deploying with Helm
***
Streaming Hub ships as a dedicated Helm chart, `streaming-hub-helm`, separate from any other Lerian product chart. The chart runs the hub in one of two shapes:
* **`all`**: a single deployment that runs every background worker. This is the default and the simplest to operate.
* **`split`**: separate **ingest** and **delivery** deployments that scale independently: ingest replicas share one Kafka consumer group, while delivery replicas work delivery jobs from Postgres.
The split shape is driven per process by `STREAMING_HUB_ROLE` (`all` | `ingest` | `delivery`). The role gates **which background workers run and which Kafka clients dial**. It does **not** gate which HTTP routes mount. Every role serves the full control-plane API and, crucially, the `/readyz` endpoint your orchestrator and metrics scrape depend on. There is one image and one binary. The role is a deployment input, not a build.
## Running database migrations
***
Streaming Hub uses a single hub-owned PostgreSQL database, and it **never migrates itself**. Schema migrations run **out of band**. A separate migration step (for example, an ArgoCD PreSync hook) applies the versioned migrations before the hub starts. At boot the hub only *verifies* that the schema version it expects is present. It never runs a migration as a side effect of starting.
The hub does provision its own weekly table partitions ahead of time as a routine background task. That counts as internal housekeeping, not a schema migration. It needs no operator action, as long as the partition cron keeps running.
## Health and graceful shutdown
***
Streaming Hub exposes two distinct probe endpoints. Wire each to the matching Kubernetes probe:
| Endpoint | Probe | Behavior |
| ---------- | --------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `/healthz` | Liveness | Returns `200` unconditionally once the process is serving, independent of Postgres, Kafka, or application-ingest consumer health. A failing liveness restarts the pod, so it must not depend on dependencies. |
| `/readyz` | Readiness | Folds the full probe set for the role. `Healthy` or `Degraded` → `200` (kept in rotation); `Down` → `503` (pulled from rotation). A `Degraded` replica keeps serving. |
`/readyz` distinguishes two failure classes. A **runtime-probe** failure (Postgres unreachable, consumer dead) takes the replica **Down** and out of rotation. A **degrader** (elevated latency, consumer lag, a thin partition buffer) clamps the replica to **Degraded** but keeps it serving. An impaired replica should not refuse traffic. The probe set is role-aware. A missing ingest consumer never makes a delivery-role pod unready.
On `SIGTERM` the hub drains gracefully. It flips `/readyz` to `NotReady` **first**, before it stops serving. It then waits a bounded pre-stop window, so the orchestrator can pull the pod from the service before connection teardown. The `/healthz` endpoint stays `200` throughout, so the orchestrator does not kill the pod mid-drain. It then tears down in dependency-safe order (HTTP, then consumer, then dispatcher, then the background apps, then Kafka clients, then the pool, then telemetry).
Set the deployment's `terminationGracePeriodSeconds` at or above the hub's derived drain ceiling for your `STREAMING_HUB_SHUTDOWN_TIMEOUT`, not a fixed magic number. At the default 30-second shutdown timeout the ceiling is about **80 seconds**. That total is the 5-second pre-stop window plus the shutdown timeout itself. It also includes a worst-case dispatcher drain leg of `min(timeout, 55s)` and a fixed teardown margin for the remaining components.
A grace period below the ceiling risks a `SIGKILL` of a still-draining replica. That is safe for correctness, but it forfeits the clean drain. The hub reclaims and redelivers in-flight jobs, and the consumer deduplicates them.
`/version` (build identity) and `/runtime` (a cheap Go-runtime snapshot) round out the unauthenticated operational surface for incident triage.
## Essential configuration
***
Streaming Hub reads its configuration from `STREAMING_HUB_*` environment variables, plus shared `MULTI_TENANT_*`, `PLUGIN_AUTH_*`, and `OTEL_*` variables and the unprefixed `ENV_NAME`. The full inventory, with every default, lives in the service's environment reference. The variables you set most often:
| Variable | Default | Purpose |
| -------------------------------------- | ---------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `STREAMING_HUB_ENV` | `local` | Deployment environment (`local` \| `staging` \| `production`). Drives the production security gate that rejects the dev bypass flags. |
| `STREAMING_HUB_ROLE` | `all` | Deployable slice: `all` \| `ingest` \| `delivery`. Gates workers and Kafka clients, never routes. |
| `STREAMING_HUB_HTTP_LISTEN_ADDR` | `:8080` | The single control-plane bind address. |
| `STREAMING_HUB_POSTGRES_DSN` | *(required)* | The hub-owned PostgreSQL DSN. Boot fails if empty. |
| `STREAMING_HUB_KAFKA_BROKERS` | *(empty)* | Bootstrap broker list for the internal stream. |
| `STREAMING_HUB_TENANT_ID` | `default` | The BYOC tenant id. See the warning below before changing it. |
| `STREAMING_HUB_MANIFEST_SOURCES` | *(empty)* | Comma-separated producer base URLs whose manifests populate the catalog in BYOC mode. Invalid URLs fail boot. This variable is forbidden in multi-tenant mode. |
| `STREAMING_HUB_MANIFEST_CLIENT_ID` | *(empty)* | Optional client id used to fetch BYOC producer manifests. Set it together with `STREAMING_HUB_MANIFEST_CLIENT_SECRET`; leaving both empty uses anonymous fetches. |
| `STREAMING_HUB_MANIFEST_CLIENT_SECRET` | *(empty)* | Optional client secret for manifest fetches. Setting only one manifest credential fails boot. |
| `STREAMING_HUB_KEK_SOURCE` | `env` | Key-encryption-key provider (`env` \| `secretsmanager`). |
| `STREAMING_HUB_KEK_REF` | *(empty)* | The **name** of the env var holding the KEK material — never the material itself. |
| `PLUGIN_AUTH_ADDRESS` | *(plugin-auth default)* | The plugin-auth decision-point base URL for control-plane authorization. |
| `PLUGIN_AUTH_ENABLED` | `true` | Master auth switch. `false` is a local bypass, **rejected in production**. |
| `STREAMING_HUB_AUTODISABLE_ENABLED` | `true` | Kill switch for auto-disabling broken destinations. |
| `STREAMING_HUB_SHUTDOWN_TIMEOUT` | `30s` | Graceful-drain window. The pre-stop window plus `min(value, 55s)` must stay strictly below the 60-second dispatcher lease, or boot fails closed. |
| `MULTI_TENANT_ENABLED` | `false` | `false` is BYOC single-tenant; `true` uses Tenant Manager for the active-tenant roster while each ingest-capable replica runs one application Kafka consumer with `STREAMING_HUB_KAFKA_*`. |
| `MULTI_TENANT_URL` | *(required in multi-tenant mode)* | Base URL of the Tenant Manager service used for the initial roster and periodic refresh. |
| `MULTI_TENANT_SERVICE_API_KEY` | *(required in multi-tenant mode)* | Service credential used to authenticate Tenant Manager requests. |
| `MULTI_TENANT_REDIS_HOST` | *(empty; optional in multi-tenant mode)* | Optional Redis endpoint for near-real-time tenant lifecycle updates. When unset, the listener does not start. The periodic Tenant Manager refresh still converges the roster; on a delivery-only pod, the roster stays fixed until restart. Configure the matching password, database, and TLS settings when your Redis deployment requires them. |
| `ENV_NAME` | *(required in multi-tenant mode)* | Environment segment in the `tenants/{ENV_NAME}/...` Secrets Manager path for per-tenant M2M producer-manifest credentials. |
| `STREAMING_HUB_AWS_HUB_PRINCIPAL_ARN` | *(empty)* | The hub's public IAM principal embedded in AWS setup artifacts. Required for AWS sinks. |
| `STREAMING_HUB_AWS_SETUP_TEMPLATE_URL` | *(empty)* | The public CloudFormation template URL for the AWS quick-create link. |
| `OTEL_EXPORTER_OTLP_ENDPOINT` | *(empty)* | The OTLP collector endpoint telemetry exports to (no `STREAMING_HUB_` prefix). |
Removing a URL from `STREAMING_HUB_MANIFEST_SOURCES` stops future refreshes from that producer but keeps its last-known-good catalog rows. Manifest refresh is control-plane discovery only. A producer outage does not block event ingest or delivery matching.
Secret **values** never belong in committed configuration. You reference the KEK by the *name* of the env var that the deploy layer injects it into (`STREAMING_HUB_KEK_REF`). The hub reads the material from that named variable and never logs it. Deployment-level Kafka SASL and TLS material, the Tenant Manager service API key, and Redis credentials must come from your secret store. Per-tenant M2M credentials for producer-manifest discovery remain in Secrets Manager.
**`STREAMING_HUB_TENANT_ID` sets the BYOC control-plane tenant.** Ingest accepts any non-empty `ce-tenantid`. The hub persists a different tenant, but that tenant cannot match subscriptions owned by this BYOC tenant. If you set a different value, subscriptions must use that tenant. Otherwise ingest succeeds but delivery remains empty. A non-default value emits one startup warning. Heed it.
## DLQ forensics
***
`GET /admin/dlq` is the operator forensics surface for dead-letter observations. It is **cross-tenant by design**. The lib-auth admin scope gates it, it carries **no** tenant shim, and it returns records across all tenants. It is therefore not part of the customer-facing `/v1` API.
The dead-letter observations it reads are **observability only**. They come from upstream producers' dead-letter topics, and the hub never re-delivers them. Use it to investigate why records failed upstream. It does not replay them.
## Topic reconciler
***
The topic reconciler is a **read-only** drift detector, enabled by default (`STREAMING_HUB_RECONCILER_ENABLED`). On each pass it compares the live broker topics, the event catalog, and the distinct subscription targets. It flags three kinds of drift:
* **ghost topics**: a followed topic with no catalog entry.
* **dead subscriptions**: a subscribed event type and major with no live catalog entry.
* **lag-versus-retention** breaches.
It **detects, never corrects**. It emits count-only gauges and structured logs, and writes no broker or database state. When disabled, it spawns no goroutine and dials no admin client, so the disabled path costs nothing. A disabled reconciler loses an operational alarm but never affects delivery.
## Observability
***
Streaming Hub exports its metrics, traces, and logs through **OTLP** (lib-observability), pointed at the collector in `OTEL_EXPORTER_OTLP_ENDPOINT`. That is where the real `streaming_hub_*` metrics live.
The `/metrics` endpoint is **almost empty by design**: it serves only the static `streaming_hub_build_info` gauge. Scrape the hub's real metrics from your OTLP collector, not from `/metrics`.
Tenant identity is never a metric label. It lives on span attributes and log fields, so the metric cardinality stays bounded no matter how many tenants a deployment serves.
## Next steps
***
The delivery internals behind the operational surfaces above.
The control-plane operations your tenants use.