intermediateMonitoring: Logs, Metrics, Prometheus & Grafana

A production host ran out of disk space and the cause turned out to be container logs

Investigation shows /var/lib/docker/containers contains several multi-gigabyte -json.log files for a chatty service that's been running for months. Explain the root cause and the fix, including whether existing data needs cleanup. Expected reasoning: root cause is the default json-file driver with no rotation configured, so logs accumulated unbounded. Immediate fix: truncate or rotate the existing oversized log files (carefully, ideally restarting the container after setting rotation rather than truncating a file Docker has open); long-term fix: set max-size/max-file globally in daemon.json (o

This is a Pro chapter

Sign in, then upgrade to Pro or Power to unlock this and the full DevOps Mastery library.

A production host ran out of disk space and the cause turned out to be container logs

Next Step

Continue to Prometheus shows a service's memory metric climbing steadily for days before it gets OOM-killed← Back to all Docker questions