advanced~3h

High Availability: Multi-Node Control Plane & Leader Election

A single control-plane node is a single point of failure for the entire cluster. This chapter is how Kubernetes runs the control plane itself across multiple nodes without multiple copies of the Scheduler or Controller Manager stepping on each other.

A production Kubernetes cluster runs MULTIPLE replicas of every control-plane component (API Server, Scheduler, Controller Manager) across multiple nodes, specifically so losing any ONE node doesn't take down the ability to schedule new Pods or reconcile cluster state.

The API Server is straightforward to run in multiple replicas — every instance is stateless and can independently serve requests, typically fronted by a load balancer. The Scheduler and Controller Manager are different: running two ACTIVE copies simultaneously would cause both to try scheduling the same pending Pod, or both trying to reconcile the same Deployment — Leader Election exists specifically to prevent that.

Multiple Scheduler and Controller Manager replicas all start up simultaneously, but only ONE of them becomes the active 'leader' at a time — the rest sit idle in standby, watching for the leader to fail.
Leader election works via a Lease object (a Kubernetes API resource) that the current leader must periodically renew — any replica able to acquire (or take over) that lease becomes the active leader; if the current leader stops renewing it (crashes, network partition), another replica's next renewal attempt succeeds and it becomes the new leader.
etcd itself (the cluster's datastore, covered in Core Components) runs its OWN separate leader-election and consensus protocol (Raft) across its own multi-node cluster — a majority (quorum) of etcd nodes must agree before any write is considered committed, which is why etcd clusters are deployed with an ODD number of nodes (3 or 5), so a quorum can always be unambiguously determined even if some nodes are unreachable.
The API Server does NOT use leader election — every replica is independently active simultaneously, since serving read/write requests doesn't have the same 'only one instance should be doing this at once' requirement that scheduling/reconciliation does.

Inspect the current Scheduler leader: kubectl get lease kube-scheduler -n kube-system -o yaml — the holderIdentity field shows which specific instance currently holds leadership.
Simulate a leader failure by deleting/restarting the current leader's pod (in a test cluster, not production) and watch kubectl get lease kube-scheduler -n kube-system -w to observe a NEW holderIdentity take over within the configured lease-renewal window.
Check etcd's own cluster health and leadership: etcdctl endpoint status --cluster shows which etcd node is the current Raft leader among the etcd cluster's own members.
Verify API Server replica count and that all are actively serving (not leader-elected) via kubectl get pods -n kube-system -l component=kube-apiserver and confirming multiple Running replicas.

A financial-services cluster runs a 3-node control plane specifically so that ANY single node's hardware failure, OS patch reboot, or availability-zone outage doesn't take the cluster's ability to schedule or reconcile workloads down with it — the leader election mechanism ensures a standby Scheduler/Controller Manager instance takes over within seconds.

An etcd cluster deployed with 3 nodes tolerates ONE node failure while still maintaining quorum (2 of 3 remaining) — deploying with an EVEN number like 4 nodes doesn't actually improve fault tolerance over 3, since a network partition splitting 2-and-2 has NO majority side at all, making an odd count strictly better for the same or lower node count.

A managed Kubernetes offering (EKS, GKE, AKS) handles all of this control-plane HA and leader election transparently — cluster operators of a managed service never directly observe leases or etcd quorum, but the exact same mechanism this chapter describes is running underneath.

Run control-plane components across at least 3 nodes, and ideally across multiple availability zones, so a single zone outage doesn't take the whole control plane down.
Size etcd clusters with an ODD number of nodes (3 or 5, not 4 or 6) — an even count adds cost without adding real fault tolerance, for the reason covered in Real-World Examples above.
Monitor lease-renewal latency and leader-election-transition frequency as real signals — frequent, unexpected leadership changes usually indicate network instability or resource pressure on control-plane nodes, not a benign background event.
For self-managed clusters, keep etcd on dedicated, fast storage (low-latency SSDs) — etcd's Raft consensus is latency-sensitive, and slow disk I/O on ANY etcd node can trigger unnecessary leader elections across the whole etcd cluster.

Assuming 'multiple replicas' alone means true high availability

— running 3 Scheduler replicas provides NO benefit if all 3 happen to be scheduled onto the SAME physical node or availability zone; HA requires actual node/zone-level distribution, not just replica count.

Deploying etcd with an even number of nodes, assuming more is always safer

— as covered above, an even-numbered etcd cluster can lose quorum-determining ability in an even network split, making it no more (and arguably less predictably) fault-tolerant than the next-lower odd number.

Ignoring frequent leader-election transitions as 'just how Kubernetes works'

— election churn is usually a genuine symptom of network flakiness or control-plane node resource starvation, and dismissing it as normal background noise means missing an early warning sign before a real outage.

Treating a single-control-plane-node cluster as production-ready 'for now'

— a common shortcut in early-stage deployments that quietly becomes a real single point of failure once real users depend on the cluster, often not revisited until the exact node it's running on fails.

A stable, non-flapping leader avoids the (small but real) disruption of in-flight scheduling/reconciliation work being abandoned mid-operation when leadership changes — tuning lease-duration and renewal-deadline settings appropriately for your infrastructure's actual network reliability reduces unnecessary churn.
etcd's Raft consensus latency directly gates how fast the WHOLE cluster's API writes can be acknowledged — co-locating etcd with fast local storage (not network-attached storage with variable latency) is one of the highest-leverage control-plane performance decisions in a self-managed cluster.

Run control-plane components across a minimum of 3 nodes spanning at least 2 (ideally 3) availability zones for any production cluster.
Alert specifically on etcd quorum health and leader-election frequency as dedicated, first-class cluster-health signals — not just generic 'is the API Server responding' checks, which can look healthy right up until a quorum-loss event.
Practice a control-plane node failure in a non-production cluster on a regular cadence (a 'game day' exercise) so a real production control-plane incident isn't the first time anyone has observed leader election actually happening.
For self-managed clusters, keep a documented, tested etcd backup/restore procedure (covered alongside Disaster Recovery in the Production chapter) — leader election handles a node FAILING, but doesn't help if etcd's data itself becomes corrupted and needs restoring from backup.

Watch kubectl get lease kube-scheduler -n kube-system -w in one terminal while restarting the current leader's Scheduler pod in another (on a test/kind cluster), observing the holderIdentity change to a standby replica.
Query etcdctl endpoint status --cluster against a multi-node etcd cluster (or a kind/minikube cluster with etcd exposed) to see each member's role and confirm exactly one reports itself as the Raft leader.
Calculate quorum requirements for 3-node vs. 4-node vs. 5-node etcd clusters by hand (majority = floor(n/2)+1) to internalize concretely why odd-numbered clusters are the standard recommendation.

A production control plane runs multiple replicas of every component across multiple nodes/zones so no single failure takes cluster management down — the API Server handles this by simply being stateless and multi-active, while the Scheduler and Controller Manager use Lease-based leader election to guarantee exactly one active instance at a time. etcd runs its own separate Raft-based leader election and consensus across an odd-numbered node count specifically so quorum can always be unambiguously determined, even during a network partition — the same underlying pattern (multiple replicas, one elected leader, automatic failover) appearing at two different layers of the same cluster.

Want a visual for this concept?

Generate a diagram tailored to “High Availability: Multi-Node Control Plane & Leader Election” — the AI picks whichever visual (flowchart, comparison, sequence, etc.) best fits.

Practice quiz

Next Step

Practice interview questions on this topic →← Back to all Kubernetes chapters