> ## Documentation Index
> Fetch the complete documentation index at: https://docs.lerian.studio/llms.txt
> Use this file to discover all available pages before exploring further.

# Helm troubleshooting

> Diagnose and resolve common issues when deploying or operating Midaz on Kubernetes: pod failures, ingress errors, and dependency conflicts.

This guide helps you diagnose and resolve common issues when deploying or operating Midaz on Kubernetes with Helm.

## General diagnostic commands

***

Start with these commands to get a broad picture of your deployment state.

```bash theme={null}
# List all Helm releases in the midaz namespace
helm list -n midaz

# Check the status of a specific release
helm status midaz -n midaz

# List all pods and their current state
kubectl get pods -n midaz

# Get events for the namespace (useful for spotting recent failures)
kubectl get events -n midaz --sort-by='.lastTimestamp'

# Describe a specific pod (replace <pod-name> with the actual name)
kubectl describe pod <pod-name> -n midaz

# Tail logs for a pod
kubectl logs <pod-name> -n midaz --tail=100

# Follow logs in real time
kubectl logs <pod-name> -n midaz -f
```

***

## Pods stuck in Pending

***

**Symptom:** One or more pods remain in `Pending` state and never start.

**Diagnostic commands:**

```bash theme={null}
kubectl get pods -n midaz
kubectl describe pod <pod-name> -n midaz
kubectl get events -n midaz --sort-by='.lastTimestamp'
kubectl top nodes
```

**Common causes and solutions:**

* **Insufficient CPU or memory on nodes**. The scheduler cannot find a node that satisfies the pod's resource requests.

  Check the `Events` section of `kubectl describe pod`. Look for messages like `Insufficient cpu` or `Insufficient memory`. Either reduce `resources.requests` in your `values.yaml`, or add more nodes to the cluster.

* **PersistentVolumeClaim not bound**. A PVC required by a dependency (PostgreSQL, MongoDB, Valkey) is stuck in `Pending`.

  ```bash theme={null}
  kubectl get pvc -n midaz
  kubectl describe pvc <pvc-name> -n midaz
  ```

  Verify that a StorageClass is available and that it is the default. See [PVC stuck in Pending](#pvc-stuck-in-pending) below.

* **Node selector or affinity mismatch**. The pod requires a specific node label that no node in the cluster has.

  Check your `values.yaml` for `nodeSelector` or `affinity` settings, and verify that your nodes have the expected labels:

  ```bash theme={null}
  kubectl get nodes --show-labels
  ```

***

## ImagePullBackOff

***

**Symptom:** Pods show `ImagePullBackOff` or `ErrImagePull` status.

**Diagnostic commands:**

```bash theme={null}
kubectl describe pod <pod-name> -n midaz
kubectl get events -n midaz --sort-by='.lastTimestamp' | grep -i image
```

**Common causes and solutions:**

* **Wrong image tag**. The specified tag does not exist in the registry. Inspect the selected chart package and compare `ledger.image.tag` with its metadata and values. The [Midaz chart version page](/en/platform/deploy/helm-version-compatibility) records the audited default.

* **Private registry requires authentication**. The cluster cannot pull images without credentials.

  Create an image pull secret and reference it in your `values.yaml`:

  ```bash theme={null}
  kubectl create secret docker-registry regcred \
    --docker-server=<registry-url> \
    --docker-username=<username> \
    --docker-password=<password> \
    -n midaz
  ```

  ```yaml theme={null}
  ledger:
    imagePullSecrets:
      - name: regcred
  ```

* **Missing `imagePullSecrets`**. The secret exists but the component's config does not reference it. Set `imagePullSecrets` for all affected components.

***

## CrashLoopBackOff

***

**Symptom:** Pods start and immediately crash, restarting repeatedly.

**Diagnostic commands:**

```bash theme={null}
kubectl get pods -n midaz
kubectl logs <pod-name> -n midaz --previous
kubectl describe pod <pod-name> -n midaz
```

<Tip>
  Use `--previous` to see logs from the last crashed container instance, not the currently restarting one.
</Tip>

**Common causes and solutions:**

* **Bad or missing environment variables**. A required config key is absent or has an incorrect value. Check the logs for messages like `missing env var`, `invalid config`, or similar. Review the `configmap` section of your `values.yaml`.

* **Missing Kubernetes Secret**. The pod references a secret that does not exist.

  ```bash theme={null}
  kubectl get secrets -n midaz
  kubectl describe secret <secret-name> -n midaz
  ```

  If the secret does not exist, create it manually or re-run the Helm install.

* **Wrong database credentials**. The service cannot authenticate with PostgreSQL, MongoDB, or Redis.

  Check logs for `authentication failed` or `connection refused`. Verify the `secrets` section in your `values.yaml` and confirm the credentials match those you used when you provisioned the databases.

* **OOMKilled**. The container exceeded its memory limit, and the kernel killed it.

  ```bash theme={null}
  kubectl describe pod <pod-name> -n midaz | grep -A5 "Last State"
  ```

  Look for `OOMKilled` in the `Last State` section. Increase `resources.limits.memory` in your `values.yaml`. See [Pod eviction / OOMKilled](#pod-eviction--oomkilled) below.

***

## Helm install timeout

***

**Symptom:** `helm install` or `helm upgrade` fails with a timeout error before the release reaches `deployed` state.

**Diagnostic commands:**

```bash theme={null}
helm status midaz -n midaz
kubectl get pods -n midaz
kubectl describe pod <pod-name> -n midaz
kubectl get events -n midaz --sort-by='.lastTimestamp'
```

**Common causes and solutions:**

* **Slow image pulls**. Large images on a slow connection can exceed the default timeout. Increase the timeout:

  ```bash theme={null}
  helm install midaz oci://registry-1.docker.io/lerianstudio/midaz-helm \
    --version <version> \
    -n midaz \
    --create-namespace \
    --timeout 15m
  ```

* **Init containers failing**. An init container (e.g., the database bootstrap job) hangs or retries. Check init container logs:

  ```bash theme={null}
  kubectl logs <pod-name> -n midaz -c <init-container-name>
  ```

* **Readiness probes failing**. The pod runs but does not pass its readiness check, so Helm waits indefinitely. Describe the pod and look at the `Conditions` and `Events` sections. You may need to increase `initialDelaySeconds` in your readiness probe settings, or investigate why the service is not healthy on startup.

***

## Services not reachable

***

**Symptom:** Midaz APIs are unreachable from outside the cluster, or services cannot communicate internally.

**Diagnostic commands:**

```bash theme={null}
kubectl get ingress -n midaz
kubectl describe ingress <ingress-name> -n midaz
kubectl get svc -n midaz
kubectl get endpoints -n midaz
```

**Common causes and solutions:**

* **Ingress misconfiguration**. The Ingress resource exists but the controller does not pick it up. Verify that `ingress.className` matches the class of your installed ingress controller:

  ```bash theme={null}
  kubectl get ingressclass
  ```

  Also check that the ingress controller pod itself runs:

  ```bash theme={null}
  kubectl get pods -n ingress-nginx
  ```

* **DNS not pointing to the load balancer**. The hostname in your Ingress does not resolve to the controller's external IP. Get the external IP and compare with your DNS record:

  ```bash theme={null}
  kubectl get svc -n ingress-nginx
  ```

* **TLS misconfiguration**. A missing or expired TLS secret causes the ingress to fail silently. Verify the secret exists and has not expired:

  ```bash theme={null}
  kubectl get secret <tls-secret-name> -n midaz
  kubectl describe secret <tls-secret-name> -n midaz
  ```

  If using cert-manager, check the Certificate resource status:

  ```bash theme={null}
  kubectl get certificate -n midaz
  kubectl describe certificate <cert-name> -n midaz
  ```

***

## PVC stuck in Pending

***

**Symptom:** A PersistentVolumeClaim remains in `Pending` state and the dependent pod cannot start.

**Diagnostic commands:**

```bash theme={null}
kubectl get pvc -n midaz
kubectl describe pvc <pvc-name> -n midaz
kubectl get storageclass
```

**Common causes and solutions:**

* **No default StorageClass**. The cluster has no default StorageClass.

  ```bash theme={null}
  kubectl get storageclass
  ```

  If none shows `(default)`, either create a StorageClass or explicitly set one in your `values.yaml` for the affected dependency (e.g., `postgresql.primary.persistence.storageClass`).

* **Wrong access mode**. The StorageClass does not support the access mode requested by the PVC (e.g., `ReadWriteMany` on a storage driver that only supports `ReadWriteOnce`).

  Check the `Events` section of `kubectl describe pvc`. Adjust `accessModes` in your `values.yaml` to match what your StorageClass supports.

* **Volume binding mode is `WaitForFirstConsumer`**. Some StorageClasses use delayed binding. The PVC will stay `Pending` until a pod consuming it is scheduled. This is normal behavior. Wait for the pod to be scheduled.

***

## Pod eviction / OOMKilled

***

**Symptom:** Pods are repeatedly evicted or show `OOMKilled` in their last state.

**Diagnostic commands:**

```bash theme={null}
kubectl get pods -n midaz
kubectl describe pod <pod-name> -n midaz | grep -A10 "Last State"
kubectl top pods -n midaz
kubectl top nodes
```

**Common causes and solutions:**

* **Memory limits set too low**. The container's `resources.limits.memory` is below what the service actually needs under load.

  Review the current memory usage with `kubectl top pods`, then increase the limit in your `values.yaml`:

  The chart defaults are `requests: 256Mi / 1500m` and `limits: 512Mi / 2000m`. Your override **replaces** them, so treat the defaults as the sizing baseline. Set the memory limit above the default when the container is OOMKilled:

  ```yaml theme={null}
  ledger:
    resources:
      requests:
        memory: "512Mi"
        cpu: "1500m"
      limits:
        memory: "1Gi"
        cpu: "2000m"
  ```

* **Node under memory pressure**. The node itself is under pressure and the kubelet evicts lower-priority pods. Check node conditions:

  ```bash theme={null}
  kubectl describe node <node-name> | grep -A5 Conditions
  ```

  Consider adding nodes or enabling cluster autoscaler. You can also set `PriorityClass` on Midaz pods to protect them from eviction.

***

## RabbitMQ definitions not loaded

***

**Symptom:** Midaz services start but transactions fail, queues do not exist, or messages stay unprocessed. Logs may show AMQP connection errors or missing exchanges/queues.

**Diagnostic commands:**

```bash theme={null}
kubectl get pods -n midaz | grep rabbit
kubectl logs <rabbitmq-pod-name> -n midaz --tail=100
# Check if the bootstrap job ran
kubectl get jobs -n midaz
kubectl logs job/midaz-bootstrap-rabbitmq -n midaz
```

<Note>
  The bootstrap Job names are `<release>-bootstrap-postgres`, `<release>-bootstrap-mongodb`, and `<release>-bootstrap-rabbitmq`. They set `ttlSecondsAfterFinished: 300`, so they delete themselves five minutes after finishing. Collect their logs promptly or the command returns `NotFound`.
</Note>

**Common causes and solutions:**

* **External RabbitMQ missing `load_definitions.json`**. When using an external RabbitMQ instance, the required queues, exchanges, and bindings are not present.

  Enable the bootstrap job in your `values.yaml`:

  ```yaml theme={null}
  global:
    externalRabbitmqDefinitions:
      enabled: true
      connection:
        protocol: "http"
        host: "your-rabbitmq-host"
        port: "15672"
        portAmqp: "5672"
  ```

  Or apply the definitions manually:

  ```bash theme={null}
  curl -u {user}:{pass} -X POST -H "Content-Type: application/json" \
    -d @load_definitions.json \
    http://{host}:{port}/api/definitions
  ```

  The `load_definitions.json` file is at `charts/midaz/files/rabbitmq/load_definitions.json` in the [Helm repository](https://github.com/LerianStudio/helm).

* **Bootstrap job failed silently**. The job ran but encountered an error (wrong credentials, network timeout, wrong port).

  ```bash theme={null}
  kubectl logs job/midaz-bootstrap-rabbitmq -n midaz
  ```

  Verify the `rabbitmqAdminLogin` credentials and that the management port (default `15672`) is reachable from within the cluster.

***

## Related resources

* [Deploy Midaz using Helm](/en/platform/deploy/midaz/midaz-installation): Initial installation guide
* [Upgrading Midaz and plugins via Helm](/en/platform/deploy/midaz/midaz-upgrade-guide): Upgrade procedures and rollback
* [Upgrading Helm](/en/platform/deploy/midaz/midaz-upgrading-overview): Breaking changes and migration paths between major versions
* [Midaz chart version compatibility](/en/platform/deploy/helm-version-compatibility): Current chart and application metadata
* [Helm repository](https://github.com/LerianStudio/helm): Source code and release notes