Kubernetes DaemonSets

DaemonSet runs exactly one Pod per matching node, and it schedules onto new nodes automatically as they join the cluster. That’s why CNI plugins, log shippers and node exporters are almost always DaemonSets rather than Deployments.

Five short tasks below: three about reading an existing cluster, two about building a DaemonSet from scratch. Everything assumes the usual alias:

alias k=kubectl

Task 1: How many DaemonSets exist cluster-wide?

The trap is the default namespace scoping — you have to ask for all namespaces explicitly:

$ k get ds --all-namespaces
NAMESPACE      NAME              DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR            AGE
kube-flannel   kube-flannel-ds   1         1         1       1            1                              8m27s
kube-system    kube-proxy        1         1         1       1            1           kubernetes.io/os=linux   8m28s

Answer: 2.

Task 2: Which namespace holds kube-proxy?

Straight off the same output — kube-system. One get answers both of the first two tasks, which is the point of reaching for --all-namespaces first.

Task 3: On how many nodes are kube-proxy Pods scheduled?

describe gives you the node accounting directly:

$ k describe ds kube-proxy -n kube-system
Name:           kube-proxy
Namespace:      kube-system
Selector:       k8s-app=kube-proxy
Node-Selector:  kubernetes.io/os=linux
Labels:         k8s-app=kube-proxy
Annotations:    deprecated.daemonset.template.generation: 1
Desired Number of Nodes Scheduled: 1
Current Number of Nodes Scheduled: 1
Number of Nodes Scheduled with Up-to-date Pods: 1
Number of Nodes Scheduled with Available Pods: 1
Number of Nodes Misscheduled: 0
Pods Status:  1 Running / 0 Waiting / 0 Succeeded / 0 Failed

Answer: 1 — this is a single-node lab cluster, so the control plane node is the only target.

Note the Node-Selector: kubernetes.io/os=linux. If you ever see Desired lower than your node count, a node selector, taint, or affinity rule is filtering nodes out. Number of Nodes Misscheduled is the field to watch after you change a selector on a live DaemonSet — it counts Pods still running on nodes that no longer match.

Task 4: What image does kube-flannel-ds use?

This one has a catch. Flannel ships init containers, so a naive grep returns three images and it’s not obvious which belongs to the actual workload. Add -B to pull in the container names above each image:

$ k describe ds kube-flannel-ds -n kube-flannel | grep -A 5 -B 3 Image
  Service Account:  flannel
  Init Containers:
   install-cni-plugin:
    Image:      docker.io/flannel/flannel-cni-plugin:v1.2.0
    Port:       
    Host Port:  
    Command:
      cp
    Args:
--
    Mounts:
      /opt/cni/bin from cni-plugin (rw)
   install-cni:
    Image:      docker.io/flannel/flannel:v0.23.0
    Port:       
    Host Port:  
    Command:
      cp
    Args:
--
      /etc/kube-flannel/ from flannel-cfg (rw)
  Containers:
   kube-flannel:
    Image:      docker.io/flannel/flannel:v0.23.0
    Port:       
    Host Port:  
    Command:
      /opt/bin/flanneld
    Args:

Answer: docker.io/flannel/flannel:v0.23.0 — the image under Containers: → kube-flannel, running /opt/bin/flanneld.

The two init containers just cp files into place on the host and exit; only flanneld is the long-running process. If you’d rather skip the grep gymnastics, jsonpath scopes it exactly:

k get ds kube-flannel-ds -n kube-flannel \
  -o jsonpath='{.spec.template.spec.containers[*].image}'

Swap containers for initContainers if you want the other two.

Task 5: Deploy a FluentD DaemonSet

The spec:

• Name: elasticsearch
• Namespace: kube-system
• Image: registry.k8s.io/fluentd-elasticsearch:1.20

There is no kubectl create daemonset generator. The fastest reliable route is to generate a Deployment and convert it — change kind, drop replicas and strategy:

k create deploy elasticsearch -n kube-system \
  --image=registry.k8s.io/fluentd-elasticsearch:1.20 \
  --dry-run=client -o yaml > ds.yaml

Or copy the example from the DaemonSet docs and trim it down. Either way you end up here:

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: elasticsearch
  namespace: kube-system
  labels:
    app: fluentd-elasticsearch
spec:
  selector:
    matchLabels:
      app: fluentd-elasticsearch
  template:
    metadata:
      labels:
        app: fluentd-elasticsearch
    spec:
      containers:
      - name: elasticsearch
        image: registry.k8s.io/fluentd-elasticsearch:1.20

Apply and verify:

k apply -f ds.yaml
k get ds elasticsearch -n kube-system

The two things that break this manifest most often are both about labels: spec.selector.matchLabels is required on a DaemonSet, and it must match spec.template.metadata.labels exactly. Miss either and the API server rejects the object before anything gets scheduled. Note also that spec.selector is immutable after creation — get it wrong and you delete and recreate.

A real FluentD DaemonSet would additionally mount /var/log and /var/lib/docker/containers from the host and carry a toleration for control-plane taints, but the lab only asks for name, namespace and image.

Key takeaways

• k get ds -A first. Namespace-scoped resources hide from you by default.
• describe answers “how many nodes” through the Desired / Current / Misscheduled counters — no need to count Pods by hand.
• Add -B to your greps whenever init containers might be in play, or use jsonpath to scope the query precisely.
• Selector and template labels must match, and the selector can’t be changed later.

The commands are short. Knowing which line of describe output actually answers the question is the part worth practising.