Skip to main content
Fetcher answers three operational questions through HTTP: is the process alive, can it serve traffic right now, and which dependency is at fault. This page covers each surface and what it reports.

​
Endpoints

The Manager serves all of these on SERVER_ADDRESS. The Worker has no API server, so it runs a health micro-server on HEALTH_PORT, which defaults to 4007. All of them mount before authentication. Kubernetes and load-balancer probes need no token. /readyz/tenant/:id is mounted on both services; outside multi-tenant mode it returns HTTP 400 stating that multi-tenant mode is disabled.

​
/health and the startup self-probe

/health is not a static 200. At boot, Fetcher runs every dependency probe once, in parallel, then flips a process-wide flag on the result. Until that self-probe succeeds, /health returns 503.
The kubelet restarts a pod whose dependencies failed at boot. It does not send it traffic. The flag starts false, so a process that crashes mid-probe never reports healthy by accident. Point your liveness probe at /health.
Each dependency also emits its self-probe outcome as a metric. A repeated boot failure therefore shows on a dashboard, not only in the logs.

​
/readyz and the dependency probes

/readyz runs every registered probe on every request, in parallel, one goroutine per dependency. The handler holds no cache and no background state. A cached answer opens a window where Kubernetes keeps routing to a degraded pod. A healthy aggregate returns 200. Anything else returns 503. The response body reports each dependency by name, with its status, its latency, and its TLS posture.

​
What each service probes

In multi-tenant mode, the shared MongoDB and RabbitMQ entries report n/a with the reason multi-tenant: see /readyz/tenant/:id, and the per-tenant probes move to that endpoint.

​
Per-dependency timeouts

Each probe runs under a fixed deadline. The values are not configurable, so every Lerian service has the same readiness latency envelope and one dashboard threshold works across the fleet. A probe that ignores its deadline does not block the response. The handler substitutes a down result for it.

​
Circuit breaker state

Tenant resolution runs behind a circuit breaker. The MULTI_TENANT_CIRCUIT_BREAKER_THRESHOLD variable sets how many consecutive failures open it, and it defaults to 5. The MULTI_TENANT_CIRCUIT_BREAKER_TIMEOUT_SEC variable sets how long it stays open, and it defaults to 30 seconds. The global tenant_manager check confirms client wiring only. During tenant validation, an open Tenant Manager breaker returns 503. After validation, a tenant-scoped MongoDB or RabbitMQ check can report down, circuit breaker open, and breaker_state: open. That distinguishes a tripped breaker from an ordinary connection failure.

​
Drain on SIGTERM

On SIGTERM or SIGINT, both services enter a drain before they tear down connections.
  1. /readyz short-circuits to 503 for READYZ_DRAIN_DELAY_SEC seconds, which defaults to 12 and has a minimum of 1.
  2. Kubernetes removes the pod from Service endpoints while it still serves in-flight work.
  3. Only then do connections close.
The service skips the real probes during the drain. The response carries one synthetic dependency named draining with status down, and it emits metrics like any other.
Alerts keep rating through a rolling deploy. The synthetic draining dependency keeps the metric series alive, so a dashboard shows a drain rather than a gap. Set your termination grace period above the drain window.

​
Metrics

/metrics serves Prometheus exposition, including the Go runtime and process collectors. The histogram buckets run from 1 ms to 5,000 ms. Metric names, labels, and buckets are a platform contract, and dashboards across the fleet depend on them. The duration histogram records the wall-clock time the probe contributed to the handler, not the latency the probe reported for itself. That is the number that explains a slow /readyz.

​
Tracing

Set ENABLE_TELEMETRY=true and point OTEL_EXPORTER_OTLP_ENDPOINT at your collector. Fetcher then exports OpenTelemetry traces and metrics over OTLP. Set the OTEL resource attributes for your deployment: OTEL_RESOURCE_SERVICE_NAME, OTEL_RESOURCE_SERVICE_VERSION, OTEL_RESOURCE_DEPLOYMENT_ENVIRONMENT, and OTEL_LIBRARY_NAME. The shipped examples use fetcher for the Manager and fetcher-worker for the Worker; those values are configurable, not runtime defaults. The engine emits its own spans through a one-method port. A host that supplies no tracer gets a no-op, and behavior does not change.

​
What to alert on

  1. /readyz 503 outside a deploy window. One dependency name in the response body tells you which.
  2. A rising dead-letter queue. Each message there is a job the Worker could not process. See Deployment.
  3. selfprobe_result at 0 for any dependency. A pod restarted into a broken dependency.
  4. A tenant-scoped check with breaker_state: open. An open Tenant Manager breaker can also surface as a 503 during tenant validation; the global tenant_manager check itself reports client wiring, not breaker state.

​
Next steps

Deployment

Dependencies, queues, scaling, and startup checks.

Configuration

Every environment variable, per component.

Security

Keys, signing, encryption at rest, and host validation.

Extraction jobs

Job lifecycle, terminal states, and events.