The Kubernetes Pod That Refused to Die

The last task asks you to delete a pod. kubectl delete pod says success. Then the pod comes back. A kubelet is reading a manifest from disk. Find that file, and you find the answer.

--- distro ---
PRETTY_NAME="Ubuntu 22.04.5 LTS"
VERSION_ID="22.04"
--- kernel / arch ---
Linux 6.8.0-107-generic x86_64
--- resources ---
cpus: 16
               total        used        free      shared  buff/cache   available
Mem:            61Gi        14Gi       1.3Gi       244Mi        46Gi        45Gi
Filesystem      Size  Used Avail Use% Mounted on
overlay         437G  176G  239G  43% /
--- privileges ---
user: root (uid 0), hostname: controlplane, passwordless sudo
--- container runtime ---
containerd containerd.io 1.7.22 7f7fdf5fed64eb6a7caf99b3e12efcf9d60e311c
crictl version v1.35.0
--- kubernetes ---
client/server gitVersion: v1.35.0
helm v3.20.0+gb2e4314
--- other tooling ---
git 2.34.1, python3 3.10.12, jq-1.6, curl 7.81.0

What makes a pod static

A static pod is created by the kubelet directly from a manifest file on the node’s filesystem. No scheduler, no controller, no API server involvement in its lifecycle. The kubelet polls a configured directory, and whatever valid pod manifests it finds there, it runs.

The API server does learn about them, but only as read-only mirror pods — a reflection the kubelet publishes so the pod shows up in kubectl get pods. You can look at a mirror pod. You can’t meaningfully change one. Every edit has to go back to the file.

That gives you the naming convention that solves the last task: a mirror pod is named <manifest pod name>-<node name>.

Counting the static pods

Two nodes:

$ k get no -o wide
NAME           STATUS   ROLES           AGE   VERSION   INTERNAL-IP     EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION      CONTAINER-RUNTIME
controlplane   Ready    control-plane   20m   v1.35.0   10.244.166.32           Ubuntu 22.04.5 LTS   6.8.0-107-generic   containerd://1.7.22
node01         Ready              19m   v1.35.0   10.244.96.7             Ubuntu 22.04.5 LTS   6.8.0-90-generic    containerd://1.7.22

And everything running on them:

$ k get po --all-namespaces
NAMESPACE      NAME                                   READY   STATUS    RESTARTS   AGE
kube-flannel   kube-flannel-ds-84t75                  1/1     Running   0          15m
kube-flannel   kube-flannel-ds-g94v9                  1/1     Running   0          15m
kube-system    coredns-6f6c7df987-fv844               1/1     Running   0          15m
kube-system    coredns-6f6c7df987-sc6jd               1/1     Running   0          15m
kube-system    etcd-controlplane                      1/1     Running   0          15m
kube-system    kube-apiserver-controlplane            1/1     Running   0          15m
kube-system    kube-controller-manager-controlplane   1/1     Running   0          15m
kube-system    kube-proxy-5wkqc                       1/1     Running   0          15m
kube-system    kube-proxy-phtm2                       1/1     Running   0          15m
kube-system    kube-scheduler-controlplane            1/1     Running   0          15m

Four: etcd, kube-apiserver, kube-controller-manager, kube-scheduler. The -controlplane suffix is the tell — those names are node names glued onto manifest names, and no ReplicaSet produces that pattern.

Everything else has a controller behind it. coredns-6f6c7df987-fv844 carries a ReplicaSet hash and a random suffix. The flannel and kube-proxy pods carry a single random suffix and appear exactly once per node, which is what DaemonSet pods look like. This is a chicken-and-egg thing worth internalising: the four control plane components have to be static, because the thing that would otherwise schedule them is one of them.

Deriving the manifest path instead of assuming it

Don’t memorize /etc/kubernetes/manifests; find out what the kubelet is actually configured to use.

Nearly everyone answers /etc/kubernetes/manifests from memory and is right. This lab is built to punish that, so I derived it instead. It’s the one place I went off-script, and it’s the reason the last task took two minutes rather than twenty.

The path lives in the kubelet’s config, and the kubelet tells you where its config is — it’s on its own command line:

$ ps -ef | grep kubelet
root        3776       1  0 09:48 ?        00:00:43 /usr/bin/kubelet --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf --config=/var/lib/kubelet/config.yaml
root       32670   11367  0 10:47 pts/2    00:00:00 grep --color=auto kubelet

--config=/var/lib/kubelet/config.yaml. The field you want in there is staticPodPath:

$ grep static /var/lib/kubelet/config.yaml
staticPodPath: /etc/kubernetes/manifests

Two commands, no assumptions, and it works on any node in any cluster regardless of how it was built. Confirm what’s actually running there:

$ k describe po kube-apiserver-controlplane -n kube-system | grep -A 5 Image
    Image:         registry.k8s.io/kube-apiserver:v1.35.0
    Image ID:      registry.k8s.io/kube-apiserver@sha256:32f98b308862e1cf98c900927d84630fb86a836a480f02752a779eb85c1489f3
    Port:          6443/TCP (probe-port)
    Host Port:     6443/TCP (probe-port)
    Command:

Creating a static pod

Generate the spec with kubectl run --dry-run=client, then move the file into the directory. There is no kubectl create step — dropping the file is the create step.

$ k run --restart=Never --image=busybox static-busybox --dry-run=client -o yaml --command -- sleep 1000 > static-busybox.yaml
$ mv static-busybox.yaml /etc/kubernetes/manifests/

apiVersion: v1
kind: Pod
metadata:
  labels:
    run: static-busybox
  name: static-busybox
spec:
  containers:
  - command:
    - sleep
    - "1000"
    image: busybox
    name: static-busybox
    resources: {}
  dnsPolicy: ClusterFirst
  restartPolicy: Never
status: {}

The highlighted lines are the two things the task actually asked for — sleep 1000 as the container command, and the busybox image. --restart=Never isn’t about restart semantics here; it’s what makes kubectl run emit a bare Pod rather than something with a controller attached.

Give the kubelet its sync interval — call it half a minute — and the mirror pod appears as static-busybox-controlplane:

$ k describe po static-busybox-controlplane | grep -A 5 Image
    Image:         busybox
    Image ID:      docker.io/library/busybox@sha256:dc2d74b28e4cf8984fa52af1f39bc7c3d9c73760b41a74d629f5d11b1ab28616
    Port:          
    Host Port:     
    Command:
      sleep
      1000

Changing the image

Same principle in reverse. Edit the file, save, wait.

--- /etc/kubernetes/manifests/static-busybox.yaml
+++ /etc/kubernetes/manifests/static-busybox.yaml
@@ -9,7 +9,7 @@
   - command:
     - sleep
     - "1000"
-    image: busybox
+    image: busybox:1.28.4
     name: static-busybox
     resources: {}

No apply, no rollout restart, no kubelet restart. Writing the file is the entire deploy. Twenty to thirty seconds later the digest has changed underneath you:

$ k describe po static-busybox-controlplane | grep -A 5 Image
    Image:         busybox:1.28.4
    Image ID:      docker.io/library/busybox@sha256:141c253bc4c3fd0a201d32dc1f493bcf3fff003b6df416dea4f41046e0f37d47
    Port:          
    Host Port:     
    Command:
      sleep
      1000

Note that restartPolicy: Never didn’t stop the kubelet replacing the pod. That policy governs container restarts within a pod, not the kubelet’s reconciliation of a changed manifest.

Finding and deleting a static pod on a worker node

The lab drops a pod called static-greenbox into the cluster and asks you to remove it.

$ k get po
NAME                          READY   STATUS    RESTARTS   AGE
static-busybox-controlplane   1/1     Running   0          2m16s
static-greenbox-node01        1/1     Running   0          51s

The suffix answers the first question for free: the manifest is on node01, not here. So SSH over and repeat the derivation — and this is where assuming /etc/kubernetes/manifests would have left you staring at an empty directory:

$ ssh node01
Welcome to Ubuntu 22.04.5 LTS (GNU/Linux 6.8.0-90-generic x86_64)

 * Documentation:  https://help.ubuntu.com
 * Management:     https://landscape.canonical.com
 * Support:        https://ubuntu.com/pro

This system has been minimized by removing packages and content that are
not required on a system that users do not log into.

To restore this content, you can run the 'unminimize' command.

$ ps -ef | grep kubelet
root       11815       1  0 11:14 ?        00:00:00 /usr/bin/kubelet --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf --config=/var/lib/kubelet/config.yaml

$ grep static /var/lib/kubelet/config.yaml
staticPodPath: /etc/just-to-mess-with-you

Named with feeling. The manifest is sitting in there, and its filename doesn’t match the pod name either:

$ ls /etc/just-to-mess-with-you/
greenbox.yaml
$ rm /etc/just-to-mess-with-you/greenbox.yaml
$ exit

Back on the control plane, the mirror pod disappears on the next kubelet sync:

$ k get po
NAME                          READY   STATUS    RESTARTS   AGE
static-busybox-controlplane   1/1     Running   0          5m21s

The KodeKloud lab hints you may be prompted for newRootP@ssw0rd on SSH. I wasn’t — key auth was already in place.

Gotchas

kubectl delete pod on a mirror pod succeeds and does nothing. There’s no error string to search for, which is exactly what makes it a trap. kubectl reports the pod deleted, the mirror pod vanishes, and then the kubelet’s next sync recreates it from the file that’s still on disk. If you’re deleting a pod and it keeps reappearing with no ReplicaSet or DaemonSet in sight, you’re looking at a static pod. Delete the manifest, not the pod.

staticPodPath is not guaranteed to be /etc/kubernetes/manifests. It’s a kubelet config field and a cluster operator can put it anywhere:

staticPodPath: /etc/just-to-mess-with-you

Derive it every time — two commands, and it works on nodes you didn’t build, in clusters that weren’t bootstrapped by kubeadm, and on the one node where somebody changed it three years ago and never wrote it down.

--command -- placement on kubectl run. Without the --command flag, arguments after -- are generated into args: rather than command:, which overrides the image’s CMD but leaves its ENTRYPOINT in place. For busybox that happens to work out; for an image with a real entrypoint it silently doesn’t do what you asked. If a task specifies “the command”, check that the generated YAML has command: before you move the file. I didn’t capture the failing case here — it’s a spec-shape difference, not an error message.

Filenames carry no meaning. greenbox.yaml produced a pod called static-greenbox. The kubelet reads metadata.name from inside the file; the filename is arbitrary. ls the directory, don’t grep for a filename matching the pod.

Replicate this locally

You don’t need a lab subscription for any of this.

kind gives you the closest match, including the two-node topology needed for the greenbox scenario. Create a cluster with a worker, then shell into a node with docker exec -it <cluster>-control-plane bash or docker exec -it <cluster>-worker bash — each node is a container, and /etc/kubernetes/manifests and /var/lib/kubelet/config.yaml are exactly where they’d be on a real node. To recreate the misdirection, edit staticPodPath in a node’s kubelet config, systemctl restart kubelet, and drop a manifest in the new directory.

minikube works too: minikube node add for a second node, then minikube ssh -n <node>. Slower to start, but you get systemd and a more VM-like feel.

Plain Docker won’t do it — there’s no kubelet, and the kubelet is the entire subject.

What I’d do differently

I’d have proved the mirror-pod relationship rather than asserting it. k get po static-busybox-controlplane -o jsonpath='{.metadata.ownerReferences}' shows the owning Node object, and k get po static-busybox-controlplane -o yaml | grep mirror surfaces the kubernetes.io/config.mirror annotation. I didn’t run either at the time, so they’re not in the output above — but they’re what turns “the naming convention suggests this is static” into a definitive check, and they’d have made a faster answer to task one than eyeballing suffixes.

I’d also have attempted kubectl delete pod static-greenbox-node01 deliberately, just to have the recreation captured.

What this doesn’t cover

Static pods can’t reference other API objects — no ConfigMaps, no Secrets, no ServiceAccounts. Everything comes from the file and the node’s filesystem, which is why kubeadm passes certificates into the control plane pods as hostPath mounts. This lab doesn’t touch that, nor kubelet standalone mode (running static pods with no control plane at all), nor how kubeadm uses static pod manifests during cluster upgrades — where a mistyped image tag in /etc/kubernetes/manifests/kube-apiserver.yaml takes your API server down and you get to debug it with crictl instead of kubectl.

Worth knowing that last one exists before you meet it.

• Static Pods — Kubernetes documentation
• Kubelet configuration (v1beta1) reference — the staticPodPath field and everything else in config.yaml
• Debugging Kubernetes nodes with crictl — for when the API server is the thing that’s broken
• Source lab on KodeKloud