Production: HA, DR, Namespace Strategy & Cluster Hardening
Production Kubernetes requires deliberate design across four areas: High Availability (spreading control plane and workloads across failure domains), Disaster Recovery (etcd backup, cluster restore, a
Production Kubernetes requires deliberate design across four areas: High Availability (spreading control plane and workloads across failure domains), Disaster Recovery (etcd backup, cluster restore, application data backup), namespace and multi-tenancy strategy (namespace-per-team or namespace-per-environment with RBAC and quotas), and cluster hardening (CIS Benchmark, admission controllers, audit logging, runtime security).
-
HA control plane: 3 or 5 control plane nodes spread across AZs behind a load balancer. etcd members also spread across AZs. Single-AZ control plane is a production anti-pattern.
-
Worker node HA: workloads scheduled with Pod topology spread constraints or anti-affinity across zones. HPA + Cluster Autoscaler providing elastic node capacity.
-
etcd disaster recovery: automated hourly snapshots to object storage (S3/GCS). Restoration tested quarterly. RPO determined by snapshot frequency.
-
Namespace strategy: namespace-per-team provides isolation, independent quotas, and per-team RBAC without cluster-admin blast radius. Namespace-per-environment (dev/staging/prod) in the same cluster is simpler but provides less isolation.
-
Cluster hardening: enable audit logging (API Server --audit-policy-file), enforce Pod Security Standards (at least Baseline cluster-wide), restrict container privilege (no privileged containers in production), network segmentation via NetworkPolicy, and secrets encryption at rest (EncryptionConfiguration).
-
Design control plane HA: 3+ nodes across AZs, etcd co-located or dedicated, load balancer in front.
-
Design workload HA: topology spread constraints, PodDisruptionBudgets, multiple replicas per service.
-
Implement etcd backup CronJob writing snapshots to object storage; test restore quarterly.
-
Define namespace strategy and implement per-namespace ResourceQuotas and LimitRanges.
-
Apply Pod Security Standards (Baseline at minimum) cluster-wide; Restricted for production namespaces.
-
Enable API Server audit logging to capture all cluster activity for security and compliance.
-
Implement secrets encryption at rest via EncryptionConfiguration.
-
Platform team designing a cluster to survive an AZ outage with zero planned downtime.
-
SRE team defining and testing a DR runbook with RTO < 2 hours.
-
Multi-team organization using namespace-per-team isolation with ResourceQuotas.
-
Security team implementing CIS Kubernetes Benchmark hardening for a regulated workload.
-
Never run a production cluster with a single control plane node -- one failure causes total cluster management loss.
-
Define and test DR runbooks before you need them -- first cluster restore should not be an emergency.
-
Set ResourceQuotas on every production namespace to prevent one team from consuming all cluster resources.
-
Enable PodDisruptionBudgets for all multi-replica services -- unprotected services go to zero during node drain.
-
Treating the cluster itself as the unit of DR -- also need application-level backup for stateful data.
-
Not testing etcd restore -- backup that has never been tested is not a backup.
-
Over-restricting ResourceQuotas without LimitRanges -- Pods without resource requests cannot be scheduled when quotas are enforced.
-
Running privileged containers in production -- they effectively bypass container isolation.
-
Spread workloads across AZs using topology spread constraints -- cross-AZ traffic adds latency and cost.
-
Use node pools sized for your workload profiles rather than one-size-fits-all nodes.
-
Document and practice cluster upgrade runbooks -- not just application upgrade runbooks.
-
Implement cost attribution at the namespace level to make resource consumption visible to teams.
-
Conduct quarterly disaster recovery drills -- including full cluster restore from etcd backup.
-
Take an etcd snapshot and practice restoring it to a test cluster.
-
Create a ResourceQuota and LimitRange for a test namespace; attempt to exceed the quota.
-
Deploy a 3-replica Deployment with a PodDisruptionBudget of minAvailable: 2; run kubectl drain and observe the protection.
-
Production requires deliberate HA design: control plane across AZs, workloads topology-spread, HPA + CA for elasticity.
-
etcd backup is necessary but not sufficient -- tested restore is what matters.
-
Namespace-per-team with ResourceQuotas and LimitRanges provides isolation and fair-share resource governance.
-
Pod Security Standards, audit logging, and secrets encryption at rest are baseline production hardening requirements.
-
PodDisruptionBudgets are essential for zero-downtime node maintenance.
Want a visual for this concept?
Generate a diagram tailored to “Production: HA, DR, Namespace Strategy & Cluster Hardening” — the AI picks whichever visual (flowchart, comparison, sequence, etc.) best fits.
Sign in to generate a visual →