K8s & Containers9 min read•September 2026•Kernelry Staff

Kubernetes & Container Orchestration Production Scenarios

Kubernetes is the centerpiece of modern cloud-native infrastructure. In production triage and systems engineering, senior teams test whether you actually understand the reconciliation loop, etcd consistency, scheduling constraints, and real failure modes like CrashLoopBackOff or OOMKills rather than just memorizing kubectl flags.

Architecture Blueprint

Kubernetes Pod Lifecycle & CrashLoopBackOff Flow

Step-by-step trace of how kubectl apply transitions through the API Server, Scheduler, and Kubelet, and how failed probes trigger exponential backoff.

Architecture Blueprint: Pod Lifecycle & CrashLoop DiagnosticsControl Plane • Kubelet

The Core Concept

Kubernetes is a declarative container management system. You define what you want (for example, "3 replicas of my web container with 512MB RAM"), and a continuous reconciliation loop works to keep actual cluster state matching that target - scheduling pods, replacing failed nodes, and routing traffic across virtual networks.

Key Rules of Thumb for Production Systems

•CrashLoopBackOff is a pod status, not an exit code. Look at Last State and exit codes (137 = OOM, 1 = app exception).

•Service routing depends 100% on label selector matches with pod metadata labels.

•Pods only receive traffic if Readiness Probes pass. Failing liveness restarts the container; failing readiness drops traffic.

•Always inspect kubectl describe pod events before digging through application logs.

Why Production Systems Demand This Depth

When a Kubernetes deployment fails, you need a systematic approach. Real-world troubleshooting requires knowing how to inspect pod events, read container exit codes, check resource limits against node capacity, and verify Service selectors against Pod labels.

Common Diagnostic Pitfalls

•Confusing Pod status (CrashLoopBackOff) with container exit codes (137 OOM vs 1 application error).
•Assuming NetworkPolicies reject kubectl commands (NetworkPolicies filter data-plane traffic, not API operations).
•Modifying immutable fields in a Deployment spec instead of initiating a clean rollout.
•Forgetting that Service traffic routing relies strictly on label selector matches.

Best Practices for Scenario Drills

•Always start your answer with how you isolate the failure layer: API server, scheduler, kubelet, container runtime, or application.
•Explicitly cite kubectl describe pod to inspect the Events timeline before jumping into logs.
•Mention how resource limits and requests influence the Linux OOM score assigned to containers.

Worked Production Scenarios

Study real-world outage patterns, diagnostic trees, and clarifying questions.

Debugging a Production CrashLoopBackOff

Difficulty: mid

Scenario Prompt:

“A mission-critical deployment rollout was triggered 10 minutes ago. Several pods in the production namespace are stuck in CrashLoopBackOff. The old pods are still taking traffic, but the rollout cannot make progress. Walk me through how you diagnose and fix this.”

Substantive Technical Concepts to Hit:

Container Exit Codes

Check exit code 137 (SIGKILL / Out of Memory) vs exit code 1 (application crash or missing environment variable/config).

Pod Events & Termination Message

Inspect kubectl describe pod to review the Last State, Termination Reason, and the kubelet event stream.

Previous Container Logs

Run kubectl logs --previous to capture stderr output from the failed container instance before it restarted.

ConfigMap & Secret Mounts

Verify that required ConfigMaps, Secrets, or downward API volume projections exist and are properly mounted in the pod spec.

Senior Clarifying Questions to Investigate:

Are we seeing exit code 137 (OOMKilled) or an application startup failure with exit code 1?
Is the issue localized to specific worker nodes or spanning the entire cluster fleet?
Do we have an automated rollback strategy, or should we halt the deployment while investigating?

Service Routing & Endpoint Resolution Failures

Difficulty: senior

Scenario Prompt:

“An internal microservice returns HTTP 503 Service Unavailable when calling another internal service in the same cluster. DNS resolves the service name correctly, but requests fail to reach any pod. How do you isolate where traffic is dropping?”

Substantive Technical Concepts to Hit:

EndpointSlice Verification

Check kubectl get endpoints and endpointslices to confirm active healthy Pod IP addresses are mapped to the Service.

Label Selector Alignment

Ensure spec.selector on the Service strictly matches metadata.labels on the target deployment template.

Readiness Probe Status

Pods only enter Endpoint pools if their readiness probes succeed; a failing readiness probe removes the pod from service traffic without killing it.

CNI & NetworkPolicy Enforcement

Verify whether ingress or egress NetworkPolicies applied by the CNI plugin are silently dropping cross-namespace traffic.

Senior Clarifying Questions to Investigate:

Do the target pods pass their readiness probes and show as 1/1 Running in kubectl get pods?
Are there any NetworkPolicies in either namespace that restrict ingress on that port?
Are we traversing namespaces or staying within the same namespace boundaries?

Interactive Scenario Drills

That's where we come in.

Explaining architectural trade-offs under time constraints is very different from reading about them.

Kernelry helps you rehearse technical questions, evidence-led debugging, safe next steps, and the clear communication teams expect. The drills adapt to your answer, so you practice the reasoning rather than memorize a company-specific script. During early access, free accounts include ten daily scenarios across our mixed question pool, with no credit card required.

Start Today's Free Drill

Back to All Blog Posts