intermediate~4h

Monitoring: Logs, Metrics, Prometheus & Grafana

Observability for containerized applications rests on three pillars: logs (discrete event records), metrics (numeric time-series data — request counts, latencies, resource usage), and traces (request flow across services, covered conceptually here, detailed tooling out of scope). This chapter focuses on logs and metrics, the two most commonly tested in Docker interviews.

Docker captures whatever a container writes to stdout/stderr via a configurable logging driver (json-file by default). This is why production apps should log to stdout/stderr rather than to a file inside the container — Docker's logging pipeline only sees stdout/stderr by default.

Prometheus is a pull-based metrics system: it scrapes a /metrics HTTP endpoint on each target at regular intervals and stores the resulting time series. Grafana is the visualization layer on top, querying Prometheus (or other data sources) to render dashboards and power alerting.

Each container's stdout/stderr is captured by the configured Docker logging driver (json-file, local, journald, syslog, fluentd, awslogs, gcplogs, etc.) and written wherever that driver sends it — docker logs only works with drivers that support log retrieval (json-file, local, journald).
The default json-file driver writes each log line as a JSON object to a file under /var/lib/docker/containers//-json.log on the host — without log rotation configured, this file grows unbounded and can exhaust host disk space.
Spring Boot Actuator exposes a /actuator/prometheus endpoint (via micrometer-registry-prometheus) that formats JVM, HTTP, and custom application metrics in the Prometheus text exposition format.
Prometheus's scrape loop polls each configured target's /metrics endpoint on an interval (commonly 15s), parses the exposition-format text, and appends new samples to its on-disk time-series database (TSDB), tagged with labels (job, instance, and any custom labels).
Grafana connects to Prometheus as a data source and executes PromQL queries to populate dashboard panels; it can also evaluate alerting rules and fire notifications (Slack, email, PagerDuty) based on query thresholds.
Container-level resource metrics (CPU/memory/network per container) are exposed by cAdvisor (often bundled into kubelet in Kubernetes, or run standalone alongside Docker), which reads cgroup statistics directly from the kernel and exposes them in Prometheus format.

Configure the Spring Boot app with micrometer-registry-prometheus on the classpath and enable management.endpoints.web.exposure.include=health,prometheus so /actuator/prometheus is reachable.
Add a scrape_configs: entry in prometheus.yml pointing at the app's host:port and metrics path.
Run Prometheus (commonly as a container itself) with that config mounted in; it begins scraping on its configured interval and storing results in its TSDB.
Run Grafana, add Prometheus as a data source (URL: http://prometheus:9090 if on the same Compose network), and import or build a dashboard using PromQL queries.
Configure Docker's logging driver options (max-size, max-file) in daemon.json or per-container to enable automatic log rotation and prevent unbounded disk growth.
Optionally ship logs to a centralized aggregator (ELK/EFK stack, Loki+Grafana, or a managed service) so logs from many containers/hosts are searchable in one place rather than scattered per-host.

Spring Boot microservices exposing JVM metrics (heap usage, GC pause time, thread pool saturation) and custom business metrics (orders processed, queue depth) via Micrometer, visualized in a shared Grafana dashboard across the whole platform.
Alerting on container-level resource exhaustion (CPU throttling, OOM kill risk) using cAdvisor metrics and Prometheus alerting rules, catching problems before they cause an outage.
Centralizing container logs (via Fluentd/Fluent Bit sidecars or Docker logging drivers shipping directly to a log aggregator) so debugging a production incident doesn't require SSHing into individual hosts.
Capacity planning by tracking historical resource usage trends in Grafana/Prometheus over weeks or months to right-size container resource requests/limits ahead of traffic growth.

Always log to stdout/stderr from inside the application, never to a file inside the container's own filesystem — Docker's logging pipeline (and most log aggregators) expect this convention.
Configure log rotation (max-size/max-file) globally in daemon.json from day one — unbounded json-file logs are a frequent, entirely preventable cause of disk-full incidents.
Expose structured (JSON) logs from the application itself when possible, so downstream log aggregators can parse fields without fragile regex-based extraction.
Track the 'four golden signals' (latency, traffic, errors, saturation) at minimum for every service, not just raw CPU/memory — these directly map to user-facing impact.

Writing application logs to a file inside the container and then being surprised docker logs shows nothing — only stdout/stderr is captured by default.
Never configuring log rotation, leading to /var/lib/docker filling the host disk over weeks/months of accumulated json-file logs.
Treating docker stats output as authoritative production monitoring instead of a quick ad hoc check — it has no history/alerting and is not a substitute for Prometheus/Grafana.
Scraping metrics too infrequently (e.g., 5-minute intervals) for latency-sensitive services, missing short-lived spikes that matter for incident diagnosis.

Keep Prometheus scrape intervals reasonable (15-30s is typical) — overly frequent scraping increases load on both the target and Prometheus's storage without proportional diagnostic benefit for most services.
Use metric cardinality carefully — avoid unbounded label values (like raw user IDs) in custom metrics, which can explode Prometheus's storage and query performance.
Use the local or json-file log driver with rotation for typical workloads; only ship every line to a remote aggregator if you actually need centralized search — local-first logging with periodic shipping balances cost and visibility.

Centralize logs across all hosts/containers using a log aggregator (ELK/EFK, Grafana Loki, or a managed cloud logging service) so incident response doesn't require manually SSHing into individual hosts.
Define Prometheus alerting rules tied to user-facing SLOs (error rate, p99 latency) rather than only infrastructure metrics (CPU%), since infrastructure health doesn't always correlate directly with user experience.
Set explicit container resource limits and monitor against them — Prometheus/cAdvisor data showing a container consistently near its memory limit is an early warning before an actual OOM kill occurs.
Retain enough metrics history (Prometheus retention, or a remote-write long-term store like Thanos, Mimir, or Cortex) to support both incident postmortems and longer-term capacity planning.

Configure log rotation (max-size=10m, max-file=3) on a running container and verify with docker inspect.
Stand up the Compose stack (app + Prometheus + Grafana + cAdvisor) from this chapter's code example and build one Grafana panel showing JVM heap usage over time.
Use docker stats while load-testing a container to observe real-time CPU/memory changes, then compare against the same data visualized in Grafana.
Write a Prometheus alerting rule that fires when a service's error rate (5xx responses) exceeds 5% over a 5-minute window.

Docker captures only stdout/stderr by default — applications must log there, not to internal files, for docker logs and most aggregators to see anything.
Configure log rotation (max-size/max-file) from day one to prevent disk exhaustion.
Prometheus pulls metrics from metrics-style endpoints on a schedule; Spring Boot exposes this via Micrometer + Actuator.
Grafana visualizes and alerts on Prometheus data via PromQL; it is not a metrics store itself.
cAdvisor adds container-level resource metrics (CPU/memory/IO) from cgroups, complementing application-level metrics.

Want a visual for this concept?

Generate a diagram tailored to “Monitoring: Logs, Metrics, Prometheus & Grafana” — the AI picks whichever visual (flowchart, comparison, sequence, etc.) best fits.

Practice quiz

Next Step

Continue to Spring Boot Integration: Dockerizing, PostgreSQL, Redis, Kafka & Microservices →← Back to all Docker chapters