Async transaction processing

Async transaction processing validates the call fast, then persists the write through RabbitMQ, for lower API latency and higher throughput.

Why this matters

---

A client sends a transaction. Midaz must do two things: validate it and persist the result. In synchronous mode, both steps run in the same request. The client waits for every write to reach the database before it gets a response.

This model is simple and predictable, but it has a ceiling. At high volume, database writes become the bottleneck. Each transaction holds a connection, waits for locks, and competes for I/O.

Async mode breaks that dependency. Midaz validates the transaction, returns the response at once, and persists the data in the background through RabbitMQ. The client gets faster responses. With async processing enabled, the Bulk Recorder batches inserts by default; set BULK_RECORDER_ENABLED=false to persist queued messages individually.

For broader scaling guidance, see Scalability strategies.

How it works

---

Synchronous mode (default)

Midaz validates the transaction and writes it directly to PostgreSQL in the same request cycle. The API sends its response only after every database operation completes.

Figure 1. Synchronous transaction flow — the client waits until the database write is confirmed.

Here’s the full flow, step by step:

1. The client sends a POST /transaction to the Midaz API.

2. The API validates the request — it runs request-shape validation, balance checks, and limit enforcement here.

3. The API writes to PostgreSQL — it persists the transaction and its operations in the same request cycle.

4. PostgreSQL confirms the write — it commits all records.

5. The API returns 201 Created to the client with the created transaction. The response leaves the server only after the database confirms everything. The response carries the transient CREATED status; Midaz promotes the transaction to APPROVED asynchronously after balance processing. Do not treat the 201 as final approval; wait for the status to reach APPROVED before you treat the transaction as settled.

Characteristics:

• Response time includes database write latency.
• Each transaction is an independent database operation.
• Simpler to reason about. The response shows exactly what Midaz persists.

Even in synchronous mode, Midaz updates balances atomically in Redis during the request. Redis is authoritative for balances. The write above persists the transaction and its operations, not the Postgres balance rows. The always-on balance-sync worker reconciles those rows (see Balance synchronization).

Asynchronous mode

Midaz validates the transaction the same way. Instead of a direct database write, Midaz publishes a message to RabbitMQ. A background consumer picks up the message and handles persistence separately.

Figure 2. Asynchronous transaction flow — the client receives a response as soon as the message is published, and persistence happens in the background.

Here’s the full flow, step by step:

1. The client sends a POST /transaction to the Midaz API.

2. The API validates the request — it runs request-shape validation, balance checks, and limit enforcement exactly as in synchronous mode.

3. The API publishes the transaction payload to RabbitMQ instead of a direct database write.

4. The API returns 201 Created to the client as soon as the queue accepts the message — the client does not wait for database persistence. The response carries the transient CREATED status; Midaz promotes the transaction to APPROVED asynchronously after balance processing. Do not treat the 201 as final approval; wait for the status to reach APPROVED before you treat the transaction as settled.

5. RabbitMQ delivers the message to a background consumer, decoupled from the API request.

6. The consumer writes to PostgreSQL — it persists the transaction and its operations from the queued message. The balance-sync worker coordinates balance updates and keeps balances consistent in both modes (see the Balance synchronization section).

Characteristics:

• Response time excludes database write latency — the client waits only for validation and queue publish.
• Midaz serializes messages with MessagePack for compact, efficient transport.
• Background consumers write to the database at their own pace, with retries; batch inserts require an enabled Bulk Recorder.

The validation step is identical in both modes. Balance checks, request-shape validation, and limit enforcement — all of that happens before the API responds, regardless of processing mode. The difference is only in when the data hits the database.

Built-in resilience

---

If RabbitMQ is unavailable when async mode tries to publish a message, Midaz attempts a direct database write. If that write fails, Midaz returns the database error.

This means:

• During a queue outage, Midaz attempts to write directly to the database.
• The client can receive an error if the fallback database write fails.
• Midaz logs the queue failure and, if the direct write also fails, the fallback-write failure so your operations team can investigate them.

During a queue outage, latency can increase because writes go straight to the database. Monitor your RabbitMQ health to keep async mode active.

Enabling async mode

---

Set one environment variable in the ledger application:

Environment variable

RABBITMQ_TRANSACTION_ASYNC=true

With false (the default), all transactions use synchronous processing and persist directly to PostgreSQL. The current Ledger bootstrap still initializes RabbitMQ and wires its consumer.

With true, the ledger publishes transaction payloads to the configured RabbitMQ exchange. A background consumer then handles persistence.

RabbitMQ configuration

---

Async mode uses the following RabbitMQ settings (all in the ledger .env):

Variable	Description	Default
RABBITMQ_TRANSACTION_ASYNC	Enable async processing.	false
RABBITMQ_HOST	RabbitMQ server hostname.	midaz-rabbitmq
RABBITMQ_PORT_HOST	AMQP protocol port.	3003
RABBITMQ_PORT_AMQP	Management API port.	3004
RABBITMQ_DEFAULT_USER	Producer credentials (user).	transaction
RABBITMQ_DEFAULT_PASS	Producer credentials (password).	—
RABBITMQ_CONSUMER_USER	Consumer credentials (user).	consumer
RABBITMQ_CONSUMER_PASS	Consumer credentials (password).	—
RABBITMQ_NUMBERS_OF_WORKERS	Number of consumer worker goroutines.	5
RABBITMQ_NUMBERS_OF_PREFETCH	Messages prefetched per worker.	10
RABBITMQ_TRANSACTION_BALANCE_OPERATION_EXCHANGE	Exchange name for transaction messages.	transaction.transaction_balance_operation.exchange
RABBITMQ_TRANSACTION_BALANCE_OPERATION_KEY	Routing key.	transaction.transaction_balance_operation.key
RABBITMQ_TRANSACTION_BALANCE_OPERATION_QUEUE	Queue name.	transaction.transaction_balance_operation.queue

The consumer uses separate credentials (RABBITMQ_CONSUMER_USER / RABBITMQ_CONSUMER_PASS) from the producer. This follows the principle of least privilege — the consumer only needs read access to the queue.

Balance synchronization

---

A dedicated balance-sync worker coordinates balance updates. It uses Redis as a coordination layer. This worker runs in both synchronous and asynchronous modes. It keeps balances consistent even when multiple consumers process messages at the same time.

Variable	Description	Default
BALANCE_SYNC_BATCH_SIZE	Number of balance updates to batch before flushing.	50
BALANCE_SYNC_FLUSH_TIMEOUT_MS	Maximum wait time (ms) before flushing an incomplete batch.	500
BALANCE_SYNC_POLL_INTERVAL_MS	How often (ms) the worker checks for pending updates.	50

The balance-sync worker runs automatically in both modes. You need no extra setup beyond an available Redis instance.

RabbitMQ circuit breaker

---

When you enable async mode, Midaz depends on RabbitMQ for transaction persistence. A built-in circuit breaker protects against broker outages. It monitors the health of the RabbitMQ connection and fails fast when the broker goes down. This prevents request pileups and cascading failures.

The circuit breaker is active on the single-tenant RabbitMQ path. Multi-tenant RabbitMQ uses tenant connection management instead. The circuit breaker follows the standard three-state model:

• Closed (normal): requests flow through to RabbitMQ. The breaker counts failures.
• Open (tripped): the breaker does not contact RabbitMQ. Midaz bypasses the broker and attempts a direct database write for each async transaction; if that write fails, Midaz returns the database error. A background health checker monitors the broker and attempts recovery.
• Half-open (probing): the breaker lets a limited number of requests through to test RabbitMQ recovery. If they succeed, the circuit closes. If they fail, it reopens.

The circuit opens when either condition is true:

• The number of consecutive failures reaches the threshold, OR
• The failure ratio exceeds the configured percentage within the counting window

Circuit breaker configuration

Variable	Description	Default
RABBITMQ_CIRCUIT_BREAKER_CONSECUTIVE_FAILURES	Consecutive failures before the circuit opens.	15
RABBITMQ_CIRCUIT_BREAKER_FAILURE_RATIO	Failure percentage (0–100) that triggers open state.	50
RABBITMQ_CIRCUIT_BREAKER_MIN_REQUESTS	Minimum requests before evaluating the failure ratio.	10
RABBITMQ_CIRCUIT_BREAKER_INTERVAL	Time window (seconds) for counting failures. Counters reset after each interval.	120
RABBITMQ_CIRCUIT_BREAKER_TIMEOUT	How long (seconds) the circuit stays open before transitioning to half-open.	30
RABBITMQ_CIRCUIT_BREAKER_MAX_REQUESTS	Requests allowed through in half-open state to probe recovery.	3
RABBITMQ_CIRCUIT_BREAKER_HEALTH_CHECK_INTERVAL	How often (seconds) the background health checker pings RabbitMQ.	30
RABBITMQ_CIRCUIT_BREAKER_HEALTH_CHECK_TIMEOUT	Timeout (seconds) for each health check ping.	10

When the circuit is open, Midaz attempts direct database writes for async transactions. If a direct write fails, Midaz returns the database error; the fallback does not guarantee transaction delivery during broker outages.

For most production deployments, the defaults work well. Tune CONSECUTIVE_FAILURES and TIMEOUT if your RabbitMQ cluster has known recovery patterns. For example, lower the timeout if your broker recovers within seconds. Increase consecutive failures if you see transient network blips.

How async mode connects to Bulk Recorder

---

Async mode and the Bulk Recorder are complementary features that work together:

1. Async mode decouples the API response from persistence — transactions go to RabbitMQ instead of directly to PostgreSQL.

2. Bulk Recorder optimizes how the consumer writes those messages to the database — it batches multiple messages into single bulk inserts.

Bulk Recorder is active with async mode unless you explicitly set BULK_RECORDER_ENABLED=false. Without async mode, transaction persistence uses direct database writes and there is no transaction queue to batch.

Configuration	Processing behavior
Async false	Direct database write per transaction (synchronous)
Async true, Bulk Recorder false	Queue-based, one message processed at a time
Async true, Bulk Recorder enabled (default)	Queue-based, messages batched for bulk inserts

When to use async mode

---

Use async mode when:

• You need lower API response times for transaction creation.
• Your workload involves high transaction volumes (hundreds+ per second).
• You run batch operations like mass payouts or settlements.
• You want to decouple your API tier from database performance.

Keep synchronous mode when:

• You need direct, request-bound transaction persistence.
• Transaction volume is low to moderate.
• You want a successful API response to mean that Midaz already persisted the data.
• You work in a development or testing environment where simplicity matters more than throughput.

You can switch between modes at any time. Change RABBITMQ_TRANSACTION_ASYNC and restart the ledger application. You need no data migration, because the transaction format is the same in both paths.