Skip to main content
An event moves through Streaming Hub in three stages. The hub ingests it from the internal stream, matches it against your subscriptions, and dispatches it to each matched destination. This page walks through each stage and the delivery guarantees that fall out of it.

​
Ingesting events from the stream

Streaming Hub consumes the platform stream as CloudEvents 1.0 messages in binary content mode. The CloudEvents context attributes travel in Kafka record headers (each prefixed ce-). The event body is the record value. The hub reads routing and identity from the headers without deserializing the payload. The hub subscribes only to v3 application fact topics:
ce-source is one lowercase, dot-free segment. It uses letters, digits, underscores, or hyphens. The topic carries no resource, event, or schema-major suffix. The hub dispatches by CloudEvents headers. The follow set excludes command topics (.commands) and dead-letter topics (.dlq). Midaz, Tracer, Matcher, Lender, and Consignado application fact topics now match this follow-set contract. A second consumer reads the .dlq topics that producers write. That consumer serves observability only. The follow-set grammar excludes a trailing .dlq, so a dead-letter record never becomes a delivery to one of your subscriptions. Operators read those records through GET /admin/dlq. See DLQ forensics. Every record the hub accepts must carry this context set: For each record the hub returns one of three verdicts: The tenant admitted at ingest comes only from the validated ce-tenantid attribute the parser extracted. Ingest has no tenant-roster drop gate. Stored events match only subscriptions with the same tenant id.
Three related strings look alike but are distinct. Never conflate them:
  • ce-type is the source-qualified CloudEvents type: studio.lerian.<source>.<resource>.<event> (for example, studio.lerian.lender.loan_application.approved).
  • Kafka topic is one v3 fact topic per application: lerian.streaming.<source> (for example, lerian.streaming.lender). Resource, event, and schema major do not appear in the topic.
  • X-Lerian-Event-Type is the bare <resource>.<event> key on a delivered webhook (loan_application.approved). It intentionally omits the producer. Use a subscription origin pin when the same key can come from more than one application.
See the event streaming overview for the full wire contract.

​
Consume-once deduplication

The internal stream is at-least-once, so the same record can arrive more than once. On the way into the event inbox the hub deduplicates on ce-id. A duplicate id writes nothing and counts as a dedup drop. The hub persists records per Kafka partition in one transaction. It commits the stream position for a partition only after that partition’s write commits. A failure on one partition therefore never loses or double-commits events on another.

​
Subscription matching

After the hub stores an event, it resolves which of your subscriptions should receive it. Subscription matching is internal and catalog-free: it evaluates each event directly on (tenant, origin, event type, schema major). The origin dimension is optional on the subscription. Without it, the same event key from any producing application can match. An event no subscription wants produces zero deliveries. That is a correct outcome, not an error. Matching admits a subscription only when it is both enabled and in verification_state = active. These are two independent conditions, and matching requires both. See the subscription model.

​
Dispatching to your destinations

For every match, the hub creates a delivery job and hands it to a worker pool (eight workers by default). Workers claim due jobs atomically with a short lease and interleave across tenants, so no single tenant’s backlog starves the others. Each worker loads the stored payload, decrypts the destination’s signing secret or credential in memory, and makes exactly one delivery attempt to the sink. Delivery is at-least-once: the hub may deliver the same event more than once (through retries or redeliveries after a worker restart). Every delivery carries a stable X-Lerian-Event-Id (the ce-id) for you to deduplicate on, and a per-attempt X-Lerian-Delivery-Id that changes on every retry. Treat a repeated X-Lerian-Event-Id as a duplicate and acknowledge it without reprocessing. See Consuming events for the consumer side of this contract.

​
Retries and back-off

When an attempt fails, the hub schedules the next one on a fixed back-off curve with full jitter:
The curve spans roughly 28 hours across its eight steps. Each subscription can override it with its own schedule (up to 12 steps, each capped at 10 hours). A malformed override falls back to the default curve rather than failing delivery.

​
Dead-lettering

The hub dead-letters an event that exhausts its retries. The hub stops attempting it and records the terminal outcome. The hub reclaims a job that instead keeps crashing a worker mid-attempt and never records an outcome. The reclaim limit is five by default. The hub then dead-letters it as poison, so a single toxic job can never occupy a worker forever.

​
Circuit breaker

Each (tenant, destination) pair has a circuit breaker. When a destination fails repeatedly, its breaker opens. Further jobs for that destination are shed (rescheduled without an attempt). A single broken endpoint does not burn worker capacity or hammer a struggling target. The breaker recovers on its own once the destination starts succeeding again.

​
Auto-disabling a broken destination

The hub automatically disables a destination that stays broken for a long time. Auto-disable trips only when the open failure span is both:
  • sustained: it lasts at least the failure window (120 hours by default), and
  • spread: at least 12 hours separate the first and last failure in the span.
Both gates must hold, and a single successful delivery clears the span, so a brief blip never accumulates toward the verdict. When auto-disable trips it flips the subscription’s enabled flag to false. It never touches verification_state. Delivery stops (matching requires enabled), and the hub records why. You recover an auto-disabled subscription with POST /v1/subscriptions/:id/verify, which re-probes the destination and, on success, re-enables it in place. A kill switch (STREAMING_HUB_AUTODISABLE_ENABLED) governs auto-disable, so an operator can ship it dark. See recovering an auto-disabled subscription.

​
Ordering guarantees

The internal stream is partitioned by tenant. Events for one tenant normally ride a single partition, and the hub preserves their first-in, first-out order end to end. The hub attributes and deduplicates on the CloudEvents headers, not the Kafka record key. Because of that, a producer that spreads (salts) a hot tenant across several partitions changes only physical placement. The hub still attributes and dedups correctly. Ordering is the one trade-off. A salted tenant’s events span multiple partitions with no cross-partition order guarantee, so that tenant forfeits strict FIFO. The arrival sequence the hub assigns reflects the order events arrived at the hub, not the order they were produced. A tenant that is not salted keeps single-partition FIFO throughout.

​
Next steps

Managing subscriptions

The subscription model, onboarding flows, and secret rotation.

Consuming events

Verify signatures, deduplicate, and pull events.