beginner~4h

Kubernetes Architecture: Control Plane & Worker Nodes

Kubernetes is a declarative container orchestration platform. A cluster splits into control plane nodes (decision-making: API Server, etcd, Scheduler, Controller Manager) and worker nodes (execution:

Learning objectives

Explain the control plane / worker node split, and name what actually runs on each.
Explain why the control plane maintaining quorum (via etcd) matters for cluster resilience.
Trace what happens to running workloads if the entire control plane becomes temporarily unavailable.

📖 Story

Imagine a city with a city hall that makes all the planning decisions — zoning, permits, budgets — and a set of actual neighborhoods where people live and work, following whatever city hall has decided. A Kubernetes cluster is built the same way: the control plane is city hall, making all the decisions about what should run where; worker nodes are the neighborhoods, actually running the containers that make up your applications.

The control plane's job

The control plane doesn't run your application containers at all — its entire job is deciding and remembering what SHOULD be running, and continuously working to make that true (the control-loop pattern from the Docker category's Kubernetes Fundamentals chapter). It's made of several cooperating components — covered individually, in depth, in the next chapter — but conceptually, it's the cluster's brain: it stores the desired state, decides which node each Pod should run on, and detects when reality has drifted from that desired state.

Worker nodes' job

Each worker node is a machine (physical or virtual) that actually runs your containers, under instruction from the control plane. A node runs a kubelet (the local agent that takes instructions from the control plane and actually starts/stops containers via the container runtime, exactly as described in the Docker Kubernetes Fundamentals chapter) and kube-proxy (which handles the node's part of cluster networking, covered in the Networking chapters).

Why the control plane itself needs to be resilient

If city hall burns down, the neighborhoods don't instantly stop existing — people keep living their lives — but nobody can get a new permit approved, and no new zoning decisions can be made, until city hall is back. A Kubernetes cluster behaves similarly: if the ENTIRE control plane becomes unavailable, already-running Pods keep running (worker nodes don't need constant control-plane contact just to keep an already-scheduled Pod alive), but nothing NEW can happen — no new Pods scheduled, no failed Pods replaced, no scaling — until control plane access is restored. This is exactly why production clusters run the control plane itself with redundancy (multiple control plane nodes, an odd number for quorum, mirroring the distributed-systems consensus concepts from the Databases Mastery material), not as a single point of failure.

etcd — the control plane's single source of truth

Every piece of cluster state — every Deployment, every Pod's desired existence, every ConfigMap — is stored in etcd, a distributed, consistent key-value store that uses the Raft consensus protocol (the same class of consensus algorithm behind the leader-election concepts covered in the Databases Mastery Distributed Systems chapter) to keep multiple etcd replicas in agreement. Every other control plane component reads and writes cluster state exclusively through etcd (via the API server, covered next chapter) — etcd genuinely is the cluster's entire memory; if etcd's data is lost with no backup, the cluster has no way to know what it was even supposed to be running.

Why an odd number of control plane replicas

Exactly like the Raft-based leader election covered in the Databases Mastery Distributed Systems chapter, etcd (and the control plane built on top of it) needs a strict MAJORITY of its replicas to agree in order to make progress. An odd number of replicas (3 or 5, commonly) maximizes fault tolerance for a given replica count — a 3-node etcd cluster tolerates 1 failure while still having a majority (2 of 3); a 5-node cluster tolerates 2 failures (3 of 5) — this is the exact same quorum mathematics from that earlier material, applied directly to Kubernetes' own control plane.

What actually happens without control plane access

A worker node's kubelet, once it has received instructions for a Pod, doesn't need to keep re-confirming with the control plane just to keep that Pod's containers running — the container runtime keeps them alive independently. What genuinely stops working without control plane access: no new Pods can be scheduled, no failed Pods get automatically replaced (since that requires the ReplicaSet controller, which lives in the control plane, to notice and react), and kubectl commands querying or changing cluster state simply fail, since they all go through the API server.

An engineer runs a kubectl command (or any other API client) targeting the cluster.
The request reaches the control plane's API server, the cluster's single front door for all reads and writes to cluster state (detailed fully in the next chapter).
The API server validates the request and persists the resulting desired state into etcd.
Relevant controllers (watching for exactly this kind of change) notice the new desired state and react — for a new Pod, the scheduler decides which worker node should run it.
That decision is written back to etcd (again through the API server) as an update to the Pod's specification, now including its assigned node.
The specific worker node's kubelet, which is continuously watching for Pods assigned to its own node, notices this new assignment and instructs the local container runtime to actually start the container(s).
The kubelet continuously reports that Pod's actual status back to the API server (and therefore etcd), keeping the cluster's recorded state honest and up to date with reality.

A managed Kubernetes service (EKS, GKE, AKS) — the cloud provider runs and maintains the control plane's high availability for you, precisely because getting this right (etcd quorum, API server redundancy) is genuinely hard operational work that most teams would rather not own directly.
A single control plane node rebooting for a routine update on a cluster with only one control plane replica — briefly, no new deployments or scaling can happen, though already-running Pods continue serving traffic unaffected, illustrating the "city hall closed, neighborhoods keep functioning" behavior directly.
A 3-node etcd cluster losing one node to a hardware failure — the remaining 2 nodes still hold a majority, so the cluster continues operating normally, with no interruption to scheduling or API access at all.
A large enterprise running self-managed Kubernetes clusters with a dedicated platform team specifically responsible for control plane resilience, backup, and etcd health — a genuine, specialized operational discipline in its own right.
An engineer debugging why newly-pushed code "isn't deploying" and discovering the actual root cause is a control-plane-level issue (an unreachable API server), not anything wrong with their application or its container image at all.

Run production control planes with genuine redundancy (multiple replicas, an odd number, spread across independent failure domains) — never a single control plane node for anything beyond local learning or throwaway experimentation.
Prefer a managed Kubernetes offering for teams without a dedicated platform engineering function, specifically to offload control-plane HA and etcd operational burden onto a provider whose job that is.
Treat etcd backups as genuinely critical infrastructure backups, on the same level of seriousness as a production database backup — losing etcd's data with no backup means losing the cluster's entire memory of what it was running.
Understand the distinction between "control plane unavailable" (existing workloads keep running, but nothing new can happen) and "worker node unavailable" (that node's own workloads are affected) when diagnosing an incident — they have very different blast radii.
Size the control plane's own resources appropriately for cluster scale — a control plane under-provisioned for a large number of nodes/Pods can itself become a bottleneck for scheduling and API responsiveness.

⚠️ Why this keeps happening

A single-control-plane-node cluster works completely fine for local development and small-scale learning, so it's easy to underestimate just how central and singular a point of failure the control plane becomes once real production traffic and real availability expectations are involved.

Running production workloads on a cluster with only one control plane replica, discovering the operational risk only once that single node has a problem and everything new grinds to a halt.
Not backing up etcd at all, treating cluster configuration as something that could always be "just recreated from the YAML files" — which ignores runtime state (actual Pod status, dynamically-created resources) that isn't necessarily captured anywhere else.
Confusing a worker node failure with a control plane failure during incident diagnosis, applying the wrong mental model (and the wrong fix) to the actual problem.
Assuming a managed Kubernetes service's control plane needs the same manual HA configuration as a self-managed one — most managed offerings handle this specifically so customers don't have to, and duplicating that effort is wasted work.
Under-provisioning control plane resources for a growing cluster, leading to a slow, unresponsive API server that looks like an application problem but is actually a control-plane capacity issue.

Size etcd's underlying storage for fast disk I/O specifically — etcd is highly sensitive to disk write latency, since every state change requires a Raft-consensus-backed write before it's considered committed.
Monitor API server request latency and etcd's own internal metrics as first-class cluster health signals, not just application-level metrics — a degrading control plane shows up here first, often before it's visible anywhere else.
Keep the number of objects stored in etcd reasonable for your cluster's actual scale — an enormous number of Kubernetes objects (a common issue with excessive custom resources or overly-granular configuration) can degrade etcd and API server performance broadly.
For very large clusters, consider the specific scaling guidance from your Kubernetes distribution or managed provider around control plane sizing — this isn't a one-size-fits-all setting.
Separate etcd's own network traffic from general cluster traffic where practical, since etcd's Raft consensus is latency-sensitive to network delay between its replicas.

Run at least 3 control plane replicas in any production cluster, spread across independent failure domains (availability zones at minimum).
Establish and regularly test an etcd backup and restoration process, with the same rigor as any other critical production data backup — this directly extends the Docker category's disaster-recovery guidance to the cluster's own control plane.
Monitor control plane component health (API server, etcd, scheduler, controller manager — detailed next chapter) explicitly, distinct from application-level monitoring.
Document your cluster's control plane topology and failure-mode expectations clearly, so an on-call engineer diagnosing an incident can quickly distinguish a control-plane-level problem from a worker-node or application-level one.
Favor a managed control plane offering unless your organization has a specific, well-justified reason and genuine operational capacity to self-manage this level of infrastructure.

On a local multi-node cluster (or a managed cluster's node list), run kubectl get nodes and identify which nodes, if any, are marked as control plane nodes versus worker nodes.
For a 3-replica and a 5-replica etcd cluster, write out exactly how many simultaneous node failures each can tolerate while still maintaining a quorum majority.
Research (or, on a local test cluster, simulate) what happens to kubectl commands when the control plane is briefly unreachable, versus what happens to already-running Pods during that same window.
Explain in your own words why a single-control-plane-node cluster is an acceptable choice for local development but not for production, referencing the specific risk it introduces.

✓ Quick recap

The control plane decides and remembers what SHOULD be running; worker nodes actually run the containers, under the control plane's instruction.
etcd is the cluster's single source of truth for all state, kept consistent across its replicas via Raft consensus — the same quorum-majority mathematics as any other Raft-based distributed system.
An odd number of control plane/etcd replicas (3 or 5) maximizes fault tolerance for a given replica count.
If the control plane becomes unavailable, already-running Pods keep running, but nothing new can be scheduled, failed Pods won't be replaced, and kubectl stops working, until access is restored.
Production clusters need genuine control plane redundancy — a single control plane node is acceptable for local learning only, never for production.

Want a visual for this concept?

Generate a diagram tailored to “Kubernetes Architecture: Control Plane & Worker Nodes” — the AI picks whichever visual (flowchart, comparison, sequence, etc.) best fits.

Practice quiz

Next Step

Continue to Core Components: API Server, etcd, Scheduler, Controller Manager, Kubelet & Kube-Proxy →← Back to all Kubernetes chapters