beginner~4h

Workloads II: StatefulSet & DaemonSet

StatefulSet exists for workloads where Deployment's 'all replicas are interchangeable' assumption is false: databases, message brokers needing stable identity (predictable name, dedicated storage, fixed ordinal). DaemonSet solves a different problem: exactly one Pod on every eligible node for log collectors, monitoring agents, CNI/CSI components.

StatefulSet Pods get stable ordinal names like postgres-0, postgres-1 that persist across rescheduling.
Each StatefulSet Pod gets its own PVC from a volumeClaimTemplate. That PVC is NOT deleted when the Pod is deleted -- the replacement reattaches to the same PVC.
By default, Pods are created/scaled in strict ordinal order (OrderedReady). Switch to Parallel when ordering is not needed.
A DaemonSet controller watches nodes (not replica count) -- adding a node schedules a Pod there; removing a node removes it.
DaemonSet Pods need explicit tolerations to run on tainted nodes including control plane nodes.

For StatefulSet:

Create a headless Service (clusterIP: None) matching the StatefulSet's Pod labels first -- this provides the per-Pod stable DNS records the StatefulSet depends on.
Define the StatefulSet with volumeClaimTemplates so each replica gets its own dynamically-provisioned PVC, not a shared one.
Apply the StatefulSet; Kubernetes creates Pods in ordinal order (name-0, then name-1, ...), waiting for each to be Running and Ready before creating the next (unless podManagementPolicy: Parallel is set).
Each Pod gets a predictable DNS name: ...svc.cluster.local, used by peers for stable addressing (e.g. Postgres replication targets, Kafka broker discovery).
Scaling down removes the highest-ordinal Pod first, but its PVC survives -- scaling back up reattaches the same PVC to the same ordinal.

For DaemonSet:

Define the DaemonSet with a Pod template and, if it needs to run on control-plane/tainted nodes, matching tolerations.
Apply it -- the DaemonSet controller compares desired vs actual against every eligible node, not a target replica count.
Adding a new node to the cluster automatically schedules a Pod there; no manual scheduling step.
Use nodeSelector/affinity if only a subset of nodes should run the DaemonSet Pod (e.g. GPU-node-only agents).

StatefulSet: running a self-managed PostgreSQL or MySQL replica set where each replica needs a stable identity and its own persistent volume for WAL/data files.
StatefulSet: deploying a Kafka or Zookeeper cluster, where each broker's ordinal identity is baked into cluster configuration (broker.id, ensemble membership).
DaemonSet: running a log-shipping agent like Fluent Bit or Filebeat on every node to tail container logs.
DaemonSet: running a CNI plugin (Calico, Cilium) or a CSI node-driver, which must exist on literally every node to function.
DaemonSet: running a node-level monitoring agent (node-exporter) or a security/runtime agent (Falco) for uniform coverage.

Always front a StatefulSet with a headless Service -- without one, the stable per-Pod DNS records that make StatefulSet useful don't exist.
Size volumeClaimTemplates deliberately; growing them later usually needs a StorageClass with allowVolumeExpansion or manual PVC surgery, not a simple StatefulSet edit.
Use podManagementPolicy: Parallel for stateless-but-storage-heavy workloads that don't need ordered startup, to cut deployment time.
Give DaemonSet Pods tight resource requests/limits -- they run on every node including ones already busy with application workloads, so an oversized DaemonSet Pod steals capacity cluster-wide.
Add tolerations for control-plane taints on DaemonSets that must run everywhere (e.g. CNI, log shipping), otherwise they silently skip control-plane nodes.
Use updateStrategy: RollingUpdate with a sensible maxUnavailable for both StatefulSet and DaemonSet so upgrades don't take down the whole fleet in one step.

Deleting a StatefulSet's headless Service while Pods still run -- breaks peer-to-peer DNS resolution the application relies on (Kafka/Zookeeper members losing each other).
Assuming StatefulSet Pod deletion also deletes its PVC -- it doesn't; forgotten PVCs from decommissioned StatefulSets quietly consume storage and cost.
Running a stateful, ordered application as a plain Deployment because it 'already worked in testing' -- ReplicaSet gives no ordinal identity or stable storage per replica, so it silently breaks under real failover/restart scenarios.
Setting a DaemonSet's resource requests too high, so it doesn't fit on smaller nodes and leaves nodes without the agent it's supposed to run everywhere.
Forgetting tolerations on a DaemonSet that must run on tainted nodes (e.g. GPU nodes, control-plane) -- the Pod is silently never scheduled there.
Using OrderedReady (the default) for a StatefulSet with dozens of replicas where startup order doesn't actually matter -- adds unnecessary deployment latency.

StatefulSet Pod startup is serialized by default (OrderedReady) -- for large clusters where members don't strictly need sequential bootstrap, podManagementPolicy: Parallel cuts total rollout time significantly.
Per-Pod PVCs from volumeClaimTemplates mean storage I/O scales linearly with replica count but also means no shared-volume contention between replicas -- pick a StorageClass with performance characteristics matching what each stateful replica actually needs.
DaemonSet Pods compete for node resources with whatever application Pods land on the same node -- keep them lightweight so they don't distort bin-packing decisions the scheduler makes for everything else.
For DaemonSets doing log/metric collection, batch and buffer output rather than shipping on every event -- per-node fan-out to a central collector can become a bottleneck at cluster scale.

Back up StatefulSet PVCs independently of the StatefulSet object itself (via volume snapshots or an application-level backup job) -- deleting the StatefulSet won't delete PVCs, but a mistaken PVC deletion has no Kubernetes-level undo.
Monitor StatefulSet rollout status explicitly (kubectl rollout status statefulset/) in CI/CD -- a stuck ordinal (e.g. Pod 2 never becomes Ready) blocks all higher-ordinal Pods silently.
Run DaemonSets with a PriorityClass set high enough that they aren't evicted under node pressure before the workloads they support.
Test StatefulSet scale-down/scale-up cycles in staging before relying on them in production -- reattachment to the same PVC is expected behavior but easy to be surprised by in an incident.

Deploy a 3-replica StatefulSet running Postgres or a simple stateful app, with a headless Service, and confirm each Pod resolves via <pod>.<service>.default.svc.cluster.local.
Delete the middle-ordinal Pod (e.g. pod-1) and observe it comes back with the same name and reattaches to the same PVC.
Scale a StatefulSet down then back up and check kubectl get pvc before/after to confirm PVCs from removed ordinals aren't deleted.
Deploy a DaemonSet (e.g. a simple log-tailer) across a multi-node cluster and confirm exactly one Pod lands per node with kubectl get pods -o wide.

StatefulSet provides stable per-Pod identity, stable network addressing via headless Service, and persistent PVCs per ordinal.
PVCs from volumeClaimTemplates are NOT deleted on Pod deletion or scale-down by default.
DaemonSets reconcile toward one Pod per eligible node -- no replica count to manage.
DaemonSets need explicit tolerations for tainted nodes including control plane nodes.

Want a visual for this concept?

Generate a diagram tailored to “Workloads II: StatefulSet & DaemonSet” — the AI picks whichever visual (flowchart, comparison, sequence, etc.) best fits.

Practice quiz

Next Step

Continue to Workloads III: Job & CronJob →← Back to all Kubernetes chapters