Recover a Docker service stuck in a restart loop

Objective

Diagnose a Docker Engine container that repeatedly exits on a Linux host, preserve its data and evidence, then correct the cause before restoring automatic restarts. Compose and orchestrated services require their own deployment controls.

Prerequisites

Step-by-step procedure

Establish the baseline and scope

Record the affected container, image, incident time and recent deployment changes. Save its current state and restart policy before stopping it. Compare with a known working deployment where available.

Capture state and deployment configuration

Identify the exact container and the deployment source before changing it. Capture state, image reference, restart count, restart policy and mounted-data locations; compare them with the intended Compose or deployment configuration. Environment inspection may reveal credentials, so restrict saved output and redact values before sharing it. Keep persistent volumes while investigating.

Read the first relevant failure

Use timestamps and a bounded log window around the first observed failure. Follow the earliest application error rather than only the final restart message. If the application writes to a mounted file or another logging destination, collect the relevant output there as well; empty docker logs output does not prove that startup was healthy.

Interpret the exit state

Inspect ExitCode, Error, StartedAt and FinishedAt with the application logs and deployment events. Exit code 137 indicates a SIGKILL result and is not proof of an out-of-memory event on its own; correlate it with OOMKilled and host evidence. Check the entrypoint, command, permissions and required configuration when the application exits immediately.

Correlate memory and storage evidence

Check OOMKilled, host kernel messages and the configured memory limit at the incident time. Also verify free space for application data and logs. A memory limit increase should follow an assessment of workload and host capacity; it may only postpone a leak. An empty or inaccessible kernel log leaves the memory cause unconfirmed.

Test the required dependency path

Check the configured DNS name, destination port, certificate and application credentials for dependencies such as a database or API. A successful host-side DNS or TCP check does not establish that the container’s network and application authentication work. Repeat the relevant test from the intended network context where possible without modifying persistent data.

Pause the loop deliberately

Record the existing policy and arrange the allowed interruption before stopping the affected container. A temporary no-restart policy can help observe one attempt; it does not fix the application fault. For Compose or an orchestrator, use its deployment source and lifecycle controls rather than relying on a manual change that the next deployment will replace.

Apply the fix and validate one restart

Correct the demonstrated cause in the versioned deployment configuration, image or secret source. Start only the intended workload and observe its first attempt and logs. Once the application test succeeds, restore the approved restart policy and monitor status and RestartCount over the agreed observation period. A running process alone does not establish that the service works.

Validate the complete service

Complete the validation checklist below and retain the observed results. Record unresolved checks instead of treating a command success as a complete service recovery.

Command examples and their effects

These examples target Docker Engine on Linux, not a Swarm service. Replace the quoted CONTAINER_NAME, DEPENDENCY_NAME and PORT_NUMBER tokens with the intended values. Run observations first; policy updates, stop and start change the service. Preserve the previous policy and use the deployment source for Compose-managed workloads. Do not share raw environment exports containing secrets.

List container status

docker ps -a --no-trunc

Observe the selected container state

docker inspect 'CONTAINER_NAME' --format '{{json .State}}'

Record the current restart policy

docker inspect 'CONTAINER_NAME' --format '{{json .HostConfig.RestartPolicy}}'

Read timestamped recent logs

docker logs --timestamps --tail 200 'CONTAINER_NAME'

Read the incident log window

docker logs --since 30m 'CONTAINER_NAME'

Service-changing action: temporarily disable automatic restarts

docker update --restart=no 'CONTAINER_NAME'

Service-changing action: stop the affected container

docker stop 'CONTAINER_NAME'

Observe OOM, exit code and error fields

docker inspect 'CONTAINER_NAME' --format 'OOM={{.State.OOMKilled}} Exit={{.State.ExitCode}} Error={{.State.Error}}'

Read relevant Linux kernel messages where permitted

dmesg -T | grep -i -E 'oom|killed process' | tail -30

Check filesystem capacity

df -h

Observe persistent mounts

docker inspect 'CONTAINER_NAME' --format '{{json .Mounts}}'

Sensitive output: inspect environment locally and redact secrets

docker inspect 'CONTAINER_NAME' --format '{{json .Config.Env}}'

Check dependency name resolution from the host

getent hosts 'DEPENDENCY_NAME'

Check dependency TCP reachability from the host

nc -vz 'DEPENDENCY_NAME' 'PORT_NUMBER'

Service-changing action: start one attempt after the targeted fix

docker start 'CONTAINER_NAME'

Follow output; use Ctrl+C to stop following logs

docker logs -f 'CONTAINER_NAME'

Service-changing example only: use unless-stopped only if it is the approved policy

docker update --restart=unless-stopped 'CONTAINER_NAME'

Observe restart count and status

docker inspect 'CONTAINER_NAME' --format 'Restarts={{.RestartCount}} Status={{.State.Status}}'

Validation

The procedure is validated when:

Rollback

Troubleshooting / common errors

Official references

Keep the selected scope, evidence and outcome with the procedure record. Use the official references for the installed product and version.