intermediate~4h

Observability: Prometheus, Grafana & AlertManager

Kubernetes observability is built from three tools: Prometheus collects metrics via pull-based scraping. Grafana reads from Prometheus to build dashboards. AlertManager receives alerts from Prometheus

Prometheus uses a pull (scrape) model -- it fetches metrics from HTTP /metrics endpoints on configured targets.
ServiceMonitor and PodMonitor CRDs (Prometheus Operator) enable dynamic target discovery by label selector.
Recording rules pre-compute expensive aggregations as new time-series for efficient reuse.
Alerting rules fire when PromQL expressions return results for a configured duration (for:) -- AlertManager handles routing.
AlertManager handles deduplication, grouping, inhibition, silencing, and routing to receivers.

Instrument or choose applications that expose metrics on an HTTP /metrics endpoint in Prometheus's text exposition format (many common libraries/exporters do this out of the box).
Deploy Prometheus (commonly via the Prometheus Operator) and define ServiceMonitor/PodMonitor CRDs with label selectors so Prometheus dynamically discovers new scrape targets as Pods/Services matching those labels appear.
Prometheus scrapes each target on its configured interval and stores the resulting time series in its local TSDB.
Write PromQL recording rules for any expensive, frequently-reused aggregation, so it's pre-computed rather than recalculated on every query/dashboard load.
Connect Grafana as a data source pointed at Prometheus, and build dashboards querying those metrics (raw or recording-rule-derived).
Write alerting rules in Prometheus (PromQL expression + for: duration) that fire into AlertManager, which handles deduplication, grouping, and routing to actual notification channels (Slack, PagerDuty, email).

Tracking Deployment rollout health via Prometheus metrics on Pod restart counts and readiness status, visualized on a Grafana dashboard the on-call team watches during a release.
An AlertManager rule paging on-call when error rate exceeds a threshold for 5 minutes, with routing that escalates to a secondary responder if unacknowledged.
Using ServiceMonitor CRDs so a newly-deployed microservice is automatically scraped the moment it's labeled correctly, with zero manual Prometheus config changes.
A capacity-planning dashboard built from recording rules aggregating per-node CPU/memory usage trends over weeks, too expensive to compute live on every dashboard load.
AlertManager's inhibition rules suppressing a flood of 'Pod not Ready' alerts during a known, deliberate node-drain maintenance window.

Expose metrics with meaningful labels (not high-cardinality ones like raw user IDs) -- cardinality explosions are the most common way Prometheus deployments run into memory/performance trouble.
Use ServiceMonitor/PodMonitor CRDs instead of hand-editing Prometheus's static scrape config -- keeps target discovery declarative and versioned like everything else.
Pre-compute expensive, frequently-dashboarded aggregations as recording rules rather than running the raw heavy PromQL query on every dashboard refresh.
Set alerting rule for: durations deliberately (not zero) to avoid paging on transient blips that self-resolve within seconds.
Use AlertManager's grouping and inhibition rules to prevent alert storms -- a single root-cause event shouldn't generate one page per affected Pod.
Set retention and storage sizing for Prometheus's TSDB based on real query/dashboard needs -- very long local retention on high-cardinality metrics can balloon disk usage fast.

Exposing metrics with unbounded-cardinality labels (user ID, request ID, raw URL path) -- this is the single most common cause of a Prometheus instance running out of memory.
Writing alerting rules with no for: duration, causing pages on momentary blips that resolve before anyone could plausibly act on them.
Treating Grafana as the source of truth instead of Prometheus -- Grafana only visualizes; if the underlying PromQL query or recording rule is wrong, the dashboard confidently shows wrong numbers.
Not setting up AlertManager routing/grouping at all, leading to alert fatigue that trains on-call engineers to ignore pages.
Forgetting that Prometheus's pull model means a target that's down looks identical to 'no alert configured' unless you specifically alert on scrape failures (up == 0).
Running a single, unreplicated Prometheus instance for production monitoring -- it's a single point of failure for the very system meant to tell you when something's failing.

High-cardinality labels are the dominant performance/memory cost driver in Prometheus -- audit label cardinality before it becomes an incident.
Recording rules trade write-time compute for read-time speed -- use them for any query hit frequently on dashboards or by alerting rules, not for one-off ad hoc queries.
Tune scrape_interval per target based on actual need -- scraping everything at a very tight interval multiplies storage and CPU cost without proportional observability benefit for slow-changing metrics.
For long-term retention beyond what local Prometheus storage should hold, use remote-write to a long-term store (e.g. Thanos, Mimir, Cortex) rather than growing local TSDB retention indefinitely.

Run Prometheus with adequate persistent storage and consider a long-term-storage/HA solution (Thanos, Mimir, or similar) for production, rather than a single ephemeral instance.
Define alerting rules around genuine user-impacting symptoms (error rate, latency, saturation) rather than only low-level causes -- pages that map directly to user impact are more actionable.
Route AlertManager alerts by real severity and team ownership, with escalation policies for unacknowledged critical alerts, not a single flat notification channel.
Regularly review and prune both dashboards and alerting rules -- stale, noisy, or duplicate alerts erode trust in the monitoring stack faster than missing coverage does.

Deploy the Prometheus Operator, a sample app exposing /metrics, and a ServiceMonitor selecting it -- confirm it appears as a scrape target in the Prometheus UI.
Write a recording rule aggregating a raw metric, and confirm it's queryable as its own new metric name.
Connect Grafana to Prometheus as a data source and build a simple dashboard panel from a PromQL query.
Write an alerting rule with a for: 1m duration on a test condition you can trigger manually, and confirm it reaches AlertManager and fires a notification.

Prometheus uses pull-based scraping -- metrics fetched from /metrics endpoints.
ServiceMonitors and PodMonitors enable dynamic scrape target discovery via label selectors.
Recording rules pre-compute expensive aggregations; alerting rules fire when conditions persist for 'for:' duration.
AlertManager handles deduplication, grouping, inhibition, silencing, and routing.
High label cardinality is the primary scalability risk for Prometheus.

Want a visual for this concept?

Generate a diagram tailored to “Observability: Prometheus, Grafana & AlertManager” — the AI picks whichever visual (flowchart, comparison, sequence, etc.) best fits.

Practice quiz

Next Step

Continue to Packaging: Helm 3 & Kustomize →← Back to all Kubernetes chapters