Why Your Kubernetes Pods Keep Restarting After Deployment – A Step‑by‑Step Debug Guide

Introduction

Kubernetes has become the de‑facto platform for running containerized workloads at scale. Yet, one of the most common pain points you’ll encounter is pods that keep restarting after a new deployment. A pod stuck in a CrashLoopBackOff or ImagePullBackOff state can bring down services, increase latency, and waste cloud resources.

In this article, we’ll walk through the typical reasons behind pod restarts, show you how to systematically troubleshoot the issue, and provide concrete code snippets to resolve it. By the end, you’ll have a repeatable process you can apply to any cluster—whether you’re on Amazon EKS, Azure AKS, Google GKE, or a self‑managed K8s installation.

---

1. Understand the Restart Cycle

Kubernetes restarts a pod for two main reasons:

Reason	What Happens	Typical Symptoms
Container failure (exit code ≠ 0)	Kubelet kills the container and starts a new one.	CrashLoopBackOff, logs show stack traces.
Node‑level events (eviction, OOM, node reboot)	The pod is rescheduled on another node.	Pending → Running repeatedly, OOMKilled.

The restart policy (Always, OnFailure, Never) is set to Always for most production workloads, so the platform will keep trying to bring the pod back up. Our job is to find out why the container fails.

---

2. Gather Baseline Information

Start with the high‑level view:

# List pods in the namespace and look for the restart count
kubectl get pods -n <namespace> -o wide

You’ll see something like:

NAME                     READY   STATUS             RESTARTS   AGE
api‑frontend‑7c9f5b9c9   0/1     CrashLoopBackOff   12         5m

Take note of the restart count and status. Next, describe the pod to get events and resource details:

kubectl describe pod api‑frontend‑7c9f5b9c9 -n <namespace>

Key sections to inspect:

• Events – look for FailedMount, BackOff, OOMKilled.
• Containers – check Limits/Requests and State.
• Conditions – Ready, PodScheduled.

---

3. Common Root Causes & Fixes

3.1 Application Crashes (Exit Code ≠ 0)

Symptoms: Logs show an exception, stack trace, or process exited with status 1.

Fix:

1. Pull the container logs.

   kubectl logs api‑frontend‑7c9f5b9c9 -n <namespace> --previous
   

2. Identify the error (e.g., missing env var, DB connection failure).
3. Update the Deployment manifest.

   apiVersion: apps/v1
   kind: Deployment
   metadata:
     name: api-frontend
   spec:
     replicas: 3
     template:
       spec:
         containers:
         - name: api
           image: myrepo/api:1.2.3
           env:
           - name: DATABASE_URL
             valueFrom:
               secretKeyRef:
                 name: db‑creds
                 key: url
   

4. Apply the change:

   kubectl apply -f deployment.yaml
   

3.2 Liveness/Readiness Probe Failures

Symptoms: Readiness probe failed or Liveness probe failed in kubectl describe events.

Fix:

• Verify the endpoint works inside the container.

  kubectl exec -it api‑frontend‑7c9f5b9c9 -n <namespace> -- curl -s http://localhost:8080/health
  

• Adjust the probe configuration to give the app more time to start.

  livenessProbe:
    httpGet:
      path: /healthz
      port: 8080
    initialDelaySeconds: 30   # increased from 10
    periodSeconds: 10
    failureThreshold: 5
  readinessProbe:
    httpGet:
      path: /ready
      port: 8080
    initialDelaySeconds: 15
    periodSeconds: 5
  

• Re‑apply the manifest.

3.3 Resource Limits & OOMKilled

Symptoms: Event OOMKilled or Memory cgroup out of memory.

Fix:

1. Check current limits.

   kubectl get pod api‑frontend‑7c9f5b9c9 -n <namespace> -o jsonpath='{.spec.containers[0].resources}'
   

2. Increase limits or request more memory.

   resources:
     requests:
       memory: "256Mi"
       cpu: "250m"
     limits:
       memory: "512Mi"
       cpu: "500m"
   

3. If the pod is part of a HorizontalPodAutoscaler (HPA), ensure the HPA thresholds match the new limits.

3.4 Image Pull Errors

Symptoms: ImagePullBackOff or ErrImagePull.

Fix:

• Verify the image name and tag.
• Ensure the node can reach the registry (test with curl from a debug pod).
• If using a private registry, confirm the imagePullSecrets are correct.

  imagePullSecrets:
  - name: reg‑cred
  

• Re‑create the secret if it expired:

  kubectl create secret docker-registry reg‑cred \
    --docker-server=registry.example.com \
    --docker-username=user \
    --docker-password=pass \
    [email protected] -n <namespace>
  

3.5 Node‑Level Issues (Disk Pressure, Network)

Symptoms: Pod evicted, NodeHasDiskPressure, or NetworkUnavailable.

Fix:

• Check node status:

  kubectl get nodes -o wide
  kubectl describe node <node-name>
  

• If disk pressure, clean up unused images:

  # On the node (via SSH or a privileged DaemonSet)
  docker system prune -af
  

• For network problems, verify CNI plugin health (e.g., Calico, Cilium) and restart the plugin daemonset if needed.

---

4. A Structured Debug Workflow

1. Identify the symptom – CrashLoopBackOff, ImagePullBackOff, OOMKilled.
2. Collect logs – kubectl logs (add --previous for the last container instance).
3. Inspect events – kubectl describe pod.
4. Check resource definitions – kubectl get pod -o yaml.
5. Validate external dependencies – DB, APIs, secrets.
6. Apply a targeted fix – update manifest, secret, or node configuration.
7. Monitor – watch the pod with kubectl rollout status deployment/<name> and confirm the restart count stops increasing.

---

5. Automating the Detection (Optional)

For large clusters, manually chasing every restart is impractical. You can set up a Prometheus alert that fires when a pod restarts more than three times in five minutes:

# prometheus.rules.yml
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: pod-restart-alerts
  namespace: monitoring
spec:
  groups:
  - name: kubernetes-pods
    rules:
    - alert: HighPodRestartRate
      expr: increase(kube_pod_container_status_restarts_total[5m]) > 3
      for: 2m
      labels:
        severity: warning
      annotations:
        summary: "Pod {{ $labels.pod }} is restarting frequently"
        description: "Container {{ $labels.container }} in pod {{ $labels.pod }} has restarted {{ $value }} times in the last 5 minutes."

Integrate this rule into your Prometheus stack, and route alerts to Slack, PagerDuty, or Microsoft Teams. Early detection prevents a single flaky pod from escalating into a service outage.

---

6. Preventive Best Practices

Practice	Why It Helps
Define explicit resource requests/limits	Avoids OOMKilled and ensures the scheduler places pods on suitable nodes.
Use health checks with sensible delays	Prevents premature restarts while the app boots.
Version‑pin container images	Reduces surprise crashes from upstream changes.
Store secrets in a dedicated system (e.g., Vault, AWS Secrets Manager)	Guarantees the pod receives required env vars at runtime.
Run a canary deployment	Detects issues before they affect the whole replica set.

---

7. Conclusion

A pod that keeps restarting is a symptom, not a problem itself. By following a methodical troubleshooting workflow, you can quickly pinpoint whether the root cause lies in application code, Kubernetes configuration, resource constraints, or the underlying node.

Implementing the fixes outlined above—and automating detection with Prometheus alerts—will dramatically reduce mean‑time‑to‑recovery (MTTR) for your cloud‑native services. Remember, the goal is not just to get the pod running again, but to understand why it failed so you can prevent recurrence.

If you’re still stuck after these steps, the One MegaByte cloud‑ops team is ready to dive deeper with you. Reach out for a personalized health‑check of your Kubernetes environment.

---

Keywords: Kubernetes pod restart, CrashLoopBackOff, troubleshooting, liveness probe, OOMKilled, image pull error, cloud infrastructure, DevOps, monitoring