Kubernetes Resource Limits

Resource limits look trivial on a slide and then bite you at 3 AM when a container keeps restarting with exit code 137. This lab walks through five short tasks on Pod requests and limits — reading them, diagnosing a crash caused by them, and changing them on a live cluster.

Everything below assumes the usual alias:

alias k=kubectl

Task 1: Identify the CPU requirements on the rabbit Pod

A Pod called rabbit is running in the default namespace. The question asks for its CPU requirement — which means the request, not the limit.

kubectl describe dumps a lot of output, so pipe it through grep -A to get the container resources block and a few lines after it:

$ k describe po rabbit | grep -A 10 Limits
    Limits:
      cpu:  1
    Requests:
      cpu:        500m
    Environment:  
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-twxd8 (ro)
Conditions:
  Type                        Status
  PodReadyToStartContainers   True
  Initialized                 True

Answer: 500m (0.5 CPU).

Worth being precise about the two numbers here, because they do very different things:

CPU is a compressible resource. Exceeding the CPU limit makes your app slow, never dead. Memory is the opposite, which is exactly what Task 3 is about.

Task 2: Delete the rabbit Pod

k delete po rabbit

Nothing clever here. If it hangs, it’s a bare Pod with a finalizer or a long terminationGracePeriodSeconds — but in this lab it deletes immediately.

Task 3: Why does the elephant Pod keep crashing?

A second Pod named elephant is deployed in default and never reaches Running. Describe it and pull out the Last State block:

$ k describe po elephant | grep -A 10 'Last State'
    Last State:     Terminated
      Reason:       OOMKilled
      Exit Code:    137
      Started:      Mon, 17 Aug 2026 06:05:37 +0000
      Finished:     Mon, 17 Aug 2026 06:05:37 +0000
    Ready:          False
    Restart Count:  5
    Limits:
      memory:  10Mi
    Requests:
      memory:     5Mi

Answer: OOMKilled.

Two details in that output tell the whole story:

• Exit Code: 137 is 128 + 9, i.e. the process received SIGKILL.
• Started and Finished share the same second — the container died the instant it tried to allocate.

The kernel’s OOM killer did this, not Kubernetes. When a container crosses its memory limit, the cgroup kills it outright. There is no throttling, no grace period, no chance to flush anything.

Note also that Last State is the previous container, not the current one. State will usually show Waiting with reason CrashLoopBackOff — that’s the symptom. Last State holds the cause.

Task 4: Find the memory limit on the Pod

Same grep, different anchor:

$ k describe po elephant | grep -A 10 Limits
    Limits:
      memory:  10Mi
    Requests:
      memory:     5Mi
    Environment:  
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-thlkq (ro)
Conditions:
  Type                        Status
  PodReadyToStartContainers   True
  Initialized                 True

Answer: 10Mi.

The container runs stress with --vm-bytes 15M, so it deliberately asks for more than the 10Mi ceiling. It was never going to survive.

Task 5: Raise the limit to 20Mi

Pod specs are largely immutable — you cannot patch resources.limits on a running Pod. The workflow is: export, edit, delete, re-apply.

k get po elephant -o yaml > elephant.yaml

The only thing that changes is one line:

--- elephant.yaml.orig
+++ elephant.yaml
@@ -17,7 +17,7 @@
     name: mem-stress
     resources:
       limits:
-        memory: 10Mi
+        memory: 20Mi
       requests:
         memory: 5Mi

Here is the cleaned-up manifest, with the changed line highlighted:

apiVersion: v1
kind: Pod
metadata:
  name: elephant
  namespace: default
spec:
  containers:
  - args:
    - --vm
    - "1"
    - --vm-bytes
    - 15M
    - --vm-hang
    - "1"
    command:
    - stress
    image: polinux/stress
    imagePullPolicy: Always
    name: mem-stress
    resources:
      limits:
        memory: 20Mi
      requests:
        memory: 5Mi
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File

Then recreate and confirm:

k delete po elephant
k apply -f elephant.yaml
k get po elephant -w

The task says do not modify anything other than the required fields, which matters if you exported the live Pod: strip status, metadata.uid, resourceVersion and creationTimestamp before re-applying, or the API server will reject it. k get po elephant -o yaml --export is long gone, so either clean it by hand or use k replace --force -f elephant.yaml to do the delete-and-create in one shot.

The Pod now sits at Running with 15M of stress inside a 20Mi ceiling.

Key takeaways

• Requests schedule, limits constrain. The scheduler only reads requests; the kubelet and kernel enforce limits.
• CPU throttles, memory kills. Exceed a CPU limit and you get slow. Exceed a memory limit and you get OOMKilled with exit code 137.
• Last State holds the cause, State only shows the symptom (CrashLoopBackOff).
• Pod resource fields are immutable. Export, edit, replace --force.

The commands are short. Knowing which line of describe output actually answers the question is the part worth practising.