How to read and verify Prometheus metrics

⌚ About 2 min read

Verify that a target exposes consistent metrics and that Prometheus collects them successfully.

Before you start

Work on a copy or a controlled test when the change can affect production. Keep timestamps, screenshots and the previous configuration so the result can be compared.

Step by step

Open the target /metrics endpoint from the Prometheus network path.
Check metric names, labels, values, and unexpected high cardinality.
Inspect the target in Status > Targets and its latest scrape.
Run a simple PromQL query for the expected metric.
Compare scrape duration, errors, and timestamps before adjusting alerts.

Validation

Repeat the original test after the change and confirm that the expected service works without creating a new regression. Document the final state.

Technical deep dive

Kubernetes / Prometheus: workload state and observability must stay correlated

Technical checkpoints

CrashLoopBackOff is a restart backoff: the actual cause is in exit code, events, --previous logs, probes or resource limits.
Prometheus scrapes a target; UP=1 proves scraping, not that every metric is semantically correct.
High label cardinality can dramatically increase memory and storage without increasing metric value.

Kubernetes

For CrashLoop, read previous-instance logs before they disappear; for Prometheus, verify Target then a simple query.

kubectl describe pod <pod>
kubectl logs <pod> --previous
kubectl get pod <pod> -o yaml

Topic-specific pitfalls

Deleting a looping Pod does not fix the Deployment/StatefulSet configuration.
Adding a label containing user/request IDs often creates explosive cardinality.

How to validate

The Pod remains Ready without new restarts and probes reflect actual health.
The Prometheus target is UP and the expected query returns stable series with controlled cardinality.