Skip to main content
Midaz runs in production at high volume. This guide gives you the deployment, high-availability, and observability patterns Lerian recommends. Follow them to keep downtime low and protect your data.

​
Best-fit setup

Start a production deployment with these choices:
  • Deploy across multiple availability zones.
  • Run at least 3 worker nodes with autoscaling.
  • Separate application and database workloads.
  • Use managed services such as RDS, ElastiCache, and MongoDB Atlas.
  • Apply Kubernetes patterns for resilience, security, and observability.
  • Automate backups and alerting from day one.

​
Infrastructure planning

​
Cluster architecture

Plan the cluster for resilience and performance:
  • Deploy across multiple availability zones.
  • Run at least 3 worker nodes for high availability.
  • Enable node autoscaling to absorb workload spikes.
  • Separate application and database workloads when possible.

​
Resource sizing

  • Match node sizes to the expected workload.
  • Give critical services enough resources first.
  • Apply resource quotas to prevent contention.
  • Monitor usage and tune the sizing over time.

​
Storage

  • Use SSD-backed storage for all database components.
  • Define a storage class for each cloud provider.
  • Provision volumes with headroom for growth.
  • Use replicated or durable storage for critical data.

​
Database architecture and high availability

Midaz uses CQRS (Command Query Responsibility Segregation) to separate reads from writes. This design lets you scale each path on its own.

​
PostgreSQL

  • Use a dedicated primary for writes and replicas for reads.
  • Enable synchronous replication for critical data.
  • Configure automatic failover with Patroni or AWS RDS.
  • Monitor replication lag and consistency.
  • Prefer managed services such as AWS RDS or GCP Cloud SQL.

​
Redis / Valkey

  • Deploy in cluster mode across multiple zones.
  • Enable automatic failover with native clustering, or a Sentinel topology via REDIS_MASTER_NAME. Verify your client configuration — a managed service (ElastiCache, Memorystore) is the safer default.
  • Use managed services such as AWS ElastiCache or GCP Memorystore.

​
MongoDB

  • Use replica sets with members across zones.
  • Monitor role transitions and lag.
  • Schedule regular backups.
  • Do not write to secondaries unless you intend to.
  • Use managed services such as MongoDB Atlas or AWS DocumentDB.

​
Messaging infrastructure

Midaz runs two messaging surfaces, both off by default:
  • RabbitMQ carries the internal async transaction balance-operation pipeline (RABBITMQ_TRANSACTION_BALANCE_OPERATION_*, enabled with RABBITMQ_TRANSACTION_ASYNC=true) plus the legacy outbound transaction, overdraft, and audit event exchanges. Use a managed RabbitMQ service such as AWS MQ or CloudAMQP in production.
  • RedPanda (via lib-streaming) is the forward-looking event backbone. Set STREAMING_ENABLED=true and STREAMING_BROKERS; events publish to lerian.streaming.<resource>.<event> topics. Transaction-lifecycle and overdraft events currently publish to both transports during the migration window.
CQRS read/write separation is served by PostgreSQL replicas (DB_*_REPLICA_* DSNs), not by broker consumers rebuilding read models.

​
High availability strategies

​
Service redundancy

  • Deploy multiple replicas for every service.
  • Use anti-affinity rules to spread services across zones.
  • Apply Pod Disruption Budgets to limit downtime during updates.

​
Load balancing

  • Use ingress controllers with health checks.
  • Avoid session affinity unless a service requires it.
  • Enable connection draining for smooth rollouts.

​
Security considerations

​
Network security

  • Apply Kubernetes network policies to control traffic.
  • Give each service account minimal permissions.
  • Secure external access with TLS.
  • Restrict admin interfaces with IP allowlists.

​
Secret management

  • Use Kubernetes Secrets for credentials and tokens.
  • Rotate secrets on a regular schedule.
  • Never hardcode secrets in containers or config files.
  • Use an external secret manager for a stronger posture.

​
Monitoring and observability

​
Metrics

  • Monitor the key application and infrastructure KPIs.
  • Set alert thresholds that lead to action.
  • Use dashboards for real-time visibility.

​
Logging

  • Centralize logs across all services.
  • Use a structured format for easier filtering.
  • Apply log retention and rotation policies.
  • Define log-based alerts for critical events.

​
Tracing

  • Enable distributed tracing across services.
  • Sample traces to balance performance and cost.
  • Correlate traces with logs and metrics for full visibility.

​
Alerting

  • Create clear, reliable alerts.
  • Tune thresholds to reduce noise.
  • Route each alert through the right channel.
  • Keep runbooks for recurring issues.

​
Backup strategy

  • Automate regular backups for critical systems.
  • Store backups in more than one location or region.
  • Test the restore procedure on a regular schedule.
  • Keep the backup documentation current and accessible.

​
Idempotency

Protect critical operations against duplicate processing in production:
  • Send an idempotency key on every transaction create request with the X-Idempotency header.
  • Use explicit, deterministic keys tied to your business IDs (order IDs, payment references), not auto-generated keys.
  • Read the X-Idempotency-Replayed response header to tell a new transaction from a cached replay.
  • Set the X-TTL header in seconds to match your retry window. The default is 300. Use a shorter value for synchronous flows and a longer one for asynchronous flows.
All Lerian products support idempotency through their own header conventions. For implementation details and a comparison across products, see Retries and idempotency.

​
Final notes

Align your infrastructure to the Midaz architecture and you gain:
  • Clean read/write separation with CQRS.
  • Compatibility with managed cloud services.
  • A clear path to observability, failover, and secure operations.
Review your setup on a regular schedule to keep this foundation solid as you grow.

​
What’s next?

Ready to scale, migrate, or harden your production environment?