advanced~4h

Production Best Practices: Logging, Monitoring, Scaling, HA & Disaster Recovery

This chapter synthesizes everything from Chapters 1-12 into a single production-readiness lens. It does not introduce a new Docker feature; instead it answers the question every architect and senior engineer is actually asked in interviews and in real incident reviews: 'how do you run containers reliably in production, at scale, and recover when something breaks?' The five pillars covered are centralized logging, comprehensive monitoring (building on Chapter 8), horizontal/vertical scaling strategy, high availability (HA) design, and disaster recovery (DR) planning — each with concrete Docker/Compose/Kubernetes-level implementation guidance, not just theory.

Production readiness is not a single feature you turn on; it is the cumulative effect of many small correct decisions compounding. Logging must be centralized because container filesystems are ephemeral — once a container is removed, anything not shipped off-box is gone forever (Chapter 8). Monitoring must answer both 'is it up' (liveness) and 'is it healthy' (readiness, latency, saturation) because a process that is technically running but unable to serve traffic correctly is a worse failure mode than a crashed one — it fails silently. Scaling must be stateless-first: any container that holds session state, in-memory caches treated as a source of truth, or local writes that aren't synced elsewhere cannot be safely scaled horizontally or replaced during a rolling update. HA is achieved by removing every single point of failure — not just the application container, but the network, the load balancer, the database, and the orchestrator's own control plane. DR is the deliberate, tested, written-down answer to 'what do we do when HA fails anyway' — because every HA design has a blast radius beyond which it stops protecting you (an entire cloud region going down, a corrupted database backup propagating before anyone notices).

Ship every container's stdout/stderr to a centralized, off-host log store (Loki, ELK/EFK, or a cloud-native service like CloudWatch Logs) — never rely on docker logs against a host as your only access path.
Instrument the application for both metrics (Prometheus-style counters/gauges/histograms, Chapter 8) and structured logs (JSON, with a correlation/trace ID) so the two can be cross-referenced during an incident.
Define separate liveness and readiness probes for every service; liveness restarts a stuck process, readiness removes a not-yet-ready or overloaded instance from load balancer rotation without killing it.
Design every stateful dependency (database, cache, queue) with its own HA story — a replica set, a managed service with built-in failover, or a clustered mode — since application-layer HA is worthless if the database is a single point of failure underneath it.
Run at minimum 2 (ideally 3+) replicas of every stateless service across separate hosts/availability zones, fronted by a load balancer that performs active health checks and automatically removes unhealthy targets.
Automate backups for every stateful data store on a schedule that matches your Recovery Point Objective (RPO), and store backups off the primary infrastructure (different region/account where possible).
Write a disaster recovery runbook and periodically execute an actual restore drill — an untested backup is an assumption, not a guarantee.
Define and document Recovery Time Objective (RTO) and RPO numbers explicitly with the business, since they drive every downstream architecture decision (active-active vs active-passive, backup frequency, etc.).

An e-commerce platform defining RTO < 15 minutes and RPO < 1 minute for its order database, driving the choice of synchronous multi-AZ replication over cheaper asynchronous cross-region replication.
A SaaS company running blue-green deployments behind a load balancer so a bad release can be rolled back by an instant traffic switch rather than a slow container-by-container rollback.
A fintech service requiring immutable, encrypted, geographically separated backups specifically to satisfy both disaster recovery and regulatory compliance requirements simultaneously.
A media streaming service running active-active across two regions so an entire regional cloud outage causes degraded capacity rather than full downtime.

Treat 'it's running' and 'it's healthy' as two different questions answered by two different probes — never conflate liveness and readiness.
Centralize logs and metrics off-host from day one in production, not as an afterthought once an incident makes the lack of visibility painful.
Always run more than one replica of any service that has an SLA, across more than one underlying host/zone.
Automate backups; never rely on a human remembering to run a manual backup command.
Test restores on a real schedule (quarterly at minimum) — a backup that has never been restored is unverified.
Use rolling or blue-green deployments with automated rollback-on-failure as the default deployment strategy, not a one-shot 'stop everything, start everything new' approach.
Document RTO/RPO numbers explicitly per service rather than leaving disaster-recovery expectations implicit or assumed.
Practice incident response (game days / chaos drills) periodically so the runbook is validated under realistic conditions, not just on paper.

Running a single replica of a 'critical' service because 'it's never gone down' — until the one host it runs on does.
Treating container restart policies (restart: on-failure) as a substitute for actual high availability — restarting a crashed single instance still causes downtime during the restart window.
Storing backups on the same host or same cloud account/region as the primary data, defeating the purpose of a backup during a region-wide or account-wide incident.
Alerting on infrastructure symptoms (CPU, memory) without alerting on user-facing SLO breaches (error rate, latency) — the former can be noisy and miss real problems, the latter is what actually matters to users.
No documented runbook — tribal knowledge in one engineer's head is not a disaster recovery plan and fails exactly when that engineer is unavailable.
Confusing 'we have a backup' with 'we have disaster recovery' — DR also requires a tested, time-bound process to actually restore service, not just the existence of a backup file.

Right-size replica counts and resource requests/limits based on actual observed load (from Chapter 8's metrics) rather than guesswork, to avoid both under-provisioning (latency under load) and over-provisioning (wasted cost).
Use horizontal autoscaling (Kubernetes HPA, or scheduled scaling for predictable traffic patterns) so capacity tracks demand instead of being fixed at a peak-load estimate year-round.
Keep backup jobs from competing with production traffic for I/O by running them during low-traffic windows or against a read replica rather than the primary.
Use log sampling or tiered retention (hot/warm/cold storage) for high-volume services so centralized logging cost and query performance stay manageable as the system scales.

Formalize an incident severity matrix and on-call escalation policy before the first real incident, not during it.
Run regular chaos/game-day exercises (e.g., deliberately killing a replica, simulating a database failover) to validate that HA mechanisms work as designed rather than just as documented.
Maintain a single source of truth runbook (versioned alongside code, not in a wiki that drifts out of date) for each service's restart, rollback, and disaster-recovery procedures.
Set and track SLOs with error budgets, and tie deployment velocity/risk tolerance to remaining error budget rather than treating reliability work and feature work as entirely separate priorities.
Review and update RTO/RPO targets periodically as the business and its tolerance for downtime/data-loss change over time.

Add both a liveness and a readiness endpoint to a sample Spring Boot service (using Actuator's health groups) and demonstrate the orchestrator-level difference in behavior when each fails independently.
Write and test a backup script for a containerized database, then practice an actual restore from that backup into a fresh container to verify the backup is usable.
Deploy a small stack to Docker Swarm (docker stack deploy, not plain docker compose up — deploy.update_config/failure_action: rollback are Swarm-mode keys and are ignored by a non-Swarm docker compose up) with order: start-first and failure_action: rollback, then intentionally deploy a broken image version and observe the automatic rollback.
Draft a one-page disaster recovery runbook for a service of your choice, including explicit RTO/RPO targets, backup location, and step-by-step restore instructions.
Design (on paper) an HA architecture for a three-tier application, explicitly identifying and eliminating every single point of failure layer by layer (app, load balancer, database, DNS).

Production readiness is the compounding effect of centralized logging, SLO-based monitoring, stateless-first scaling, single-point-of-failure elimination (HA), and a tested disaster recovery plan — not any one feature.
Liveness and readiness are distinct concerns: liveness restarts a stuck process, readiness removes a not-yet-able-to-serve instance from rotation without killing it.
A restart policy on a single instance is not high availability; HA requires multiple replicas across independent failure domains behind a health-checked load balancer.
Backups without tested, documented, off-site-stored restore procedures are not disaster recovery.
RTO and RPO should be explicit, business-driven numbers per service, since they directly determine replication strategy, backup frequency, and architecture cost.
Game days and restore drills convert assumed resilience into verified resilience, and should be run periodically, not only after an actual incident exposes a gap.

Want a visual for this concept?

Generate a diagram tailored to “Production Best Practices: Logging, Monitoring, Scaling, HA & Disaster Recovery” — the AI picks whichever visual (flowchart, comparison, sequence, etc.) best fits.

Practice quiz

Next Step

Continue to Docker Registry & Image Distribution: Tagging, Versioning, Push & Pull →← Back to all Docker chapters