1. Establish the incident boundary.
Start with the observed user impact, rather than immediately treating a red dashboard as a root cause. Record the affected service, start time in UTC, namespace, cluster and any recent changes. Distinguish a symptom from a hypothesis: a restarting pod is an observation; a bad application release is a possibility that still needs evidence.
- Confirm the intended Kubernetes context and cluster before collecting information.
- Use the least-privileged, read-only access allowed by your organisation. Do not add permissions just for convenience.
- Write down the evaluation window, expected service behaviour, and what would count as recovered.
- Never copy passwords, tokens, private customer records or unredacted logs into a public issue or shared chat.
The commands below inspect resources; they do not intentionally modify workloads. Read access can still expose sensitive metadata and log entries. Apply your organisation's RBAC, retention and redaction rules.
2. Capture workload status and cluster events.
Keep the snapshot small and reproducible. Determine which deployments and pods are affected, and whether the failure is isolated to one namespace, availability zone or recent rollout. Events are time-sensitive, so record the capture time with the findings.
kubectl config current-context
kubectl get deployments,statefulsets,daemonsets,pods -n NAMESPACE -o wide
kubectl get events -n NAMESPACE --sort-by=.metadata.creationTimestamp
Look for unscheduled pods, repeated restarts, failed readiness or liveness checks, missing endpoints, image-pull failures and warning events. Events do not form a complete historical record; they may expire or be unavailable.
3. Inspect the affected pod and previous container logs.
Use describe to inspect state transitions, container termination reasons, resource requests and probe failures. Compare current logs with the previous container instance when it restarted. A --previous request may legitimately fail if no previous instance is retained.
kubectl describe pod POD_NAME -n NAMESPACE
kubectl logs POD_NAME -n NAMESPACE --timestamps --tail=200
kubectl logs POD_NAME -n NAMESPACE --previous --timestamps --tail=200
For a multi-container pod, specify -c CONTAINER_NAME. Keep a record of the exact container, time range and whether log lines were redacted. Do not assume an OOMKilled reason always means a memory leak: resource limits, spikes and node conditions are competing explanations.
4. Compare observations with recent deployments.
Find the deployment revision and controller state, then compare the timing of rollout events with the first observed failures. A temporal match does not by itself prove the rollout caused the problem.
kubectl describe deployment DEPLOYMENT_NAME -n NAMESPACE
kubectl rollout history deployment/DEPLOYMENT_NAME -n NAMESPACE
kubectl get pods -n NAMESPACE --show-labels
Check whether a failed rollout, changed image, probe configuration or resource request aligns with the incident. Do not run rollback, delete, scale, patch or apply commands during this read-only stage. Consequential changes require a separately reviewed plan, authorised execution and rollback criteria.
5. Trace Service routing and readiness evidence.
If pods are running but clients still fail, investigate the route from Service to ready endpoints. Compare Service selectors with pod labels; look at EndpointSlices, port mappings and the relevant ingress controller evidence.
kubectl get services,ingresses,endpointslices -n NAMESPACE
kubectl describe service SERVICE_NAME -n NAMESPACE
kubectl get pods -n NAMESPACE --show-labels
Missing or non-ready endpoints may indicate a selector mismatch, failed readiness check, no matching pods, or insufficient target capacity. Verify DNS, load balancer and network policy hypotheses with separately authorised evidence where applicable.
6. Inspect node pressure when scheduling or health is implicated.
Node investigations often require additional cluster-scoped read permissions. If you are authorised, compare the affected pod's assigned node with node conditions, capacity and event history.
kubectl get nodes -o wide
kubectl describe node NODE_NAME
Check conditions such as MemoryPressure, DiskPressure, or NotReady, and consider whether failures cluster on particular nodes. A healthy-looking pod does not prove end-to-end service health.
7. Use signals to narrow the next question.
The following table is a hypothesis guide, not a diagnosis. Multiple causes can produce the same observed Kubernetes status.
| Observation | Investigate first | Evidence to preserve |
|---|---|---|
Pending | Scheduling constraints, resource requests, taints, storage binding | Pod events and scheduler messages |
CrashLoopBackOff | Startup failure, configuration, probes, dependency availability | Termination reason and previous logs |
OOMKilled | Memory use, limits, workload changes and node pressure | Container status, limits and memory metrics |
ImagePullBackOff | Image reference, registry access and pull credentials | Pod events and image identifiers (never secret values) |
| No ready endpoints | Service selectors, ready pods, ports and endpoint health | Service spec, EndpointSlices and pod labels |
Node NotReady | Node conditions, kubelet/network status, affected workloads | Node describe output and timing |
8. Produce an evidence record before approving a fix.
A useful incident record separates facts from interpretation. It can be reviewed by an engineer who did not run the original commands.
- Context: incident ID, owner, cluster, namespace, UTC window, observed user impact.
- Evidence: command or approved telemetry source, capture time, redaction note, relevant events and logs.
- Hypotheses: likely causes ranked by evidence and missing data, with explicit uncertainty.
- Proposed intervention: exact intended change, approval roles, risks, backout criteria and maintenance constraints.
- Independent verification: post-change readiness, service checks, customer-facing behaviour and whether the original symptom has stopped.
Never treat an AI suggestion or a correlated event as authorisation to change production. Define the allowed action, secure the right approval, verify independently, and preserve the audit trail.
For teams that cannot grant cluster access to an external evaluator, a redacted evidence pack or a guided screen-share can support an initial discussion. It may be insufficient for root-cause confirmation; document those limits rather than inferring a result.
Practical questions
Can this checklist be run without production write permissions?
Yes, the listed inspection commands are intended to read resources. Actual access depends on RBAC and can still reveal sensitive information. Follow your security team's approval and data-handling requirements.
Does a healthy Pod mean the incident is resolved?
No. Confirm that the Service has healthy endpoints and that the real user workflow behaves as expected. Recovery evidence should match the incident's impact, not only the pod's status.
Where does SteadyState fit?
SteadyState is a DevHeal Labs infrastructure-operations product with a guided demonstration of evidence-led investigation, policy-controlled workflows and recovery verification. Live adapters and supported actions must be checked for each environment. This guide is generic Kubernetes triage guidance; it does not claim automated production remediation is available for every cluster.
Official documentation and scope
These independent references explain the Kubernetes commands and failure states used above. This page is an educational checklist from DevHeal Labs, not official Kubernetes documentation.
- Kubernetes: Debug Pods
- Kubernetes: Troubleshooting Applications
- Kubernetes: kubectl debug reference (a separate, potentially state-changing or invasive debugging capability, not part of the read-only checklist)
DevHeal Labs / Cloud reliability
Want to examine an incident with a defined evidence and approval boundary?
Start with a free introductory conversation, a guided product evaluation, or a scoped reliability assessment. We'll discuss actual access, supported integrations, success measures and the next step before any production connection.