Bulk Recorder

Use the Bulk Recorder to batch RabbitMQ messages into bulk PostgreSQL inserts, cutting database round trips and raising throughput on high-volume workloads.

Why this matters

---

Every transaction in Midaz creates balance operations that Midaz must persist. In the default synchronous mode, Midaz writes them directly to the database in the request cycle. This is fine for moderate volumes. At scale — thousands of transactions per second — it becomes the bottleneck.

The Bulk Recorder accumulates messages and writes them in batches. It makes fewer round trips to PostgreSQL, lowers lock contention, and raises throughput. High-volume workloads gain the most: mass payouts, batch settlements, and real-time payment processing.

For broader strategies to scale Midaz, see Scalability strategies.

How it works

---

The Bulk Recorder sits between the RabbitMQ consumer and the database layer. It does not insert each message at once. Instead, it collects messages in a buffer and flushes them under two conditions:

1. Batch size reached — the buffer fills to the configured size.
2. Timeout elapsed — the configured flush timeout expires, even if the buffer is not full.

Whichever condition happens first triggers the flush. This gives large batches under load and low latency during quiet periods.

Figure 1. End-to-end flow of the Bulk Recorder between RabbitMQ and PostgreSQL.

Here is the full flow, step by step:

1. RabbitMQ delivers messages to the BulkCollector one at a time — Message 1, Message 2, up to Message N.

2. The BulkCollector holds them in memory instead of one write per message. It collects until the batch fills or the flush timeout expires.

3. The BulkCollector sends a chunked bulk INSERT to PostgreSQL. Midaz splits large batches into chunks that respect PostgreSQL’s parameter limits. Each chunk uses ON CONFLICT (id) DO NOTHING, so retries and duplicate deliveries stay safe.

4. PostgreSQL confirms the write and stores the data.

5. The BulkCollector acknowledges each message back to RabbitMQ after the write. It acknowledges them one at a time, not in a single bulk ack. This stops a shared channel from acking messages that other workers still process.

Enabling Bulk Recorder

---

Bulk mode requires async mode. The explicit Bulk Recorder setting is optional because it defaults to enabled:

Environment variables

RABBITMQ_TRANSACTION_ASYNC=true
BULK_RECORDER_ENABLED=true

Async mode is required. With RABBITMQ_TRANSACTION_ASYNC=true, the Bulk Recorder is enabled by default; set BULK_RECORDER_ENABLED=false to process queued messages individually. Without async mode, Midaz persists transactions directly instead of consuming queued messages.

BULK_RECORDER_ENABLED defaults to true when you do not set the environment variable. If you already run with RABBITMQ_TRANSACTION_ASYNC=true, bulk mode is probably active. To confirm, check your application logs for Bulk mode is ACTIVE at startup.

Configuration

---

Variable	Description	Default
BULK_RECORDER_ENABLED	Enable or disable bulk mode.	true
BULK_RECORDER_SIZE	Number of messages to accumulate before flushing. 0 = auto-calculated.	0 (auto)
BULK_RECORDER_FLUSH_TIMEOUT_MS	Maximum time (ms) to wait before flushing an incomplete batch.	100

Auto-calculated batch size

When BULK_RECORDER_SIZE is set to 0 (the default), Midaz derives the batch size:

batch size = RABBITMQ_NUMBERS_OF_WORKERS × RABBITMQ_NUMBERS_OF_PREFETCH

This aligns the collector’s capacity with the real message flow from RabbitMQ. It prevents partial flushes and memory pressure.

If you set BULK_RECORDER_SIZE by hand, keep it aligned with your prefetch settings. A size much larger than workers × prefetch rarely fills the collector. The collector then flushes mostly on the timeout.

Tuning for your workload

---

The two main levers are batch size and flush timeout. The right balance depends on your priority: latency or throughput.

Low latency (real-time processing)

Keep batches small and timeouts short. Midaz persists messages quickly, even when batches are not full.

Low-latency configuration

RABBITMQ_NUMBERS_OF_WORKERS=5
RABBITMQ_NUMBERS_OF_PREFETCH=10
BULK_RECORDER_SIZE=0          # auto: 5 × 10 = 50
BULK_RECORDER_FLUSH_TIMEOUT_MS=50

High throughput (batch operations)

Larger batches and longer timeouts maximize database efficiency. Use this for mass payouts, end-of-day settlements, or migration workloads.

High-throughput configuration

RABBITMQ_NUMBERS_OF_WORKERS=10
RABBITMQ_NUMBERS_OF_PREFETCH=20
BULK_RECORDER_SIZE=0          # auto: 10 × 20 = 200
BULK_RECORDER_FLUSH_TIMEOUT_MS=300

Start with the defaults and adjust from what you observe. To confirm your settings, watch for the Bulk mode configured for consumer log at startup.

Safety guarantees

---

The Bulk Recorder stays safe under every condition:

Idempotency

Every bulk insert uses ON CONFLICT (id) DO NOTHING. If a message arrives twice — from a retry, a redelivery, or a network fault — PostgreSQL drops the duplicate. You get no data corruption and no constraint violations.

Deadlock prevention

Before each bulk insert, Midaz sorts the record IDs. All concurrent writers then acquire locks in the same order. This removes the most common source of PostgreSQL deadlocks under high concurrency.

Internal chunking

Midaz splits large batches into chunks that fit within PostgreSQL’s 65,535-parameter limit per query:

Record type	Columns per row	Rows per chunk	Parameters per chunk
Transaction	18	1,000	18,000
Operation	31	1,000	31,000

Midaz handles this chunking for you. Each INSERT carries at most 1,000 rows. BULK_RECORDER_MAX_ROWS_PER_INSERT is not currently applied to the PostgreSQL chunk size.

When to use

---

Use Bulk Recorder when:

• You process high volumes of transactions (hundreds or thousands per second).
• Your workload includes batch operations like mass payouts, settlements, or data migrations.
• You already use async transaction processing (RABBITMQ_TRANSACTION_ASYNC=true).
• You want to reduce database load and connection pressure.

Keep it disabled when:

• Your volume is low enough that individual inserts are not a bottleneck.
• You need strict per-message ordering that batch processing would break.
• You debug transaction processing and want a simpler, message-by-message flow.