⌚ About 2 min read
Identify why a container repeatedly starts and exits without hiding the cause with manual restarts.
Before you start
Work on a copy or a controlled test when the change can affect production. Keep timestamps, screenshots and the previous configuration so the result can be compared.
Step by step
- Inspect Pod state, restart count, and recent events.
- Read current container logs and previous-instance logs.
- Check command, args, environment variables, secrets, volumes, and network dependencies.
- Review liveness/startup probes and memory/CPU limits.
- Fix the cause in the Deployment or chart and observe a new rollout.
Validation
Repeat the original test after the change and confirm that the expected service works without creating a new regression. Document the final state.
Kubernetes / Prometheus: workload state and observability must stay correlated
Technical checkpoints
- CrashLoopBackOff is a restart backoff: the actual cause is in exit code, events, --previous logs, probes or resource limits.
- Prometheus scrapes a target; UP=1 proves scraping, not that every metric is semantically correct.
- High label cardinality can dramatically increase memory and storage without increasing metric value.
Kubernetes
For CrashLoop, read previous-instance logs before they disappear; for Prometheus, verify Target then a simple query.
kubectl describe pod <pod>
kubectl logs <pod> --previous
kubectl get pod <pod> -o yamlTopic-specific pitfalls
- Deleting a looping Pod does not fix the Deployment/StatefulSet configuration.
- Adding a label containing user/request IDs often creates explosive cardinality.
How to validate
- The Pod remains Ready without new restarts and probes reflect actual health.
- The Prometheus target is UP and the expected query returns stable series with controlled cardinality.