advanced~2h

Observability — Metrics & Tracing

Token usage, evaluator pass rates, tool-call latency — none of it is visible unless you deliberately make it so. This module wires Spring AI into the same Micrometer/Actuator stack you'd use for any Spring Boot service.

Learning objectives

Beginner: Explain why AI-feature latency/cost/quality can't be assumed constant the way a typical CRUD endpoint's can.
Intermediate: Enable Spring Boot Actuator metrics for an AI-calling endpoint and view them via Prometheus/Grafana.
Advanced: Trace a full RAG request (embedding + retrieval + generation) as one span via OTLP/Jaeger to pinpoint where latency or cost is actually going.

◆ The problem

Standard HTTP metrics (request count, latency, error rate) tell you your endpoint responded — they say nothing about token cost, which model answered, how many tool calls it took, or whether the answer passed evaluation. AI-specific operational concerns need AI-specific instrumentation.

<dependency>
  <groupId>org.springframework.boot</groupId>
  <artifactId>spring-boot-starter-actuator</artifactId>
</dependency>
<dependency>
  <groupId>io.micrometer</groupId>
  <artifactId>micrometer-registry-prometheus</artifactId>
</dependency>

Spring AI auto-instruments ChatClient calls with Micrometer metrics out of the box once Actuator + a metrics registry are on the classpath — no manual timer/counter code needed for the baseline metrics below.

Metric	Reports
spring.ai.chat.client	Call count and latency, tagged by model/provider.
gen_ai.client.token.usage	Prompt/completion token counts — the ground truth for cost tracking.
spring.ai.vector.store	Vector store operation latency (add/similaritySearch).

💻 Code example

<dependency>
  <groupId>org.springframework.boot</groupId>
  <artifactId>spring-boot-starter-actuator</artifactId>
</dependency>
<dependency>
  <groupId>io.micrometer</groupId>
  <artifactId>micrometer-registry-prometheus</artifactId>
</dependency>

management:
  endpoints:
    web:
      exposure:
        include: prometheus,health,metrics
  metrics:
    tags:
      application: my-ai-service

With /actuator/prometheus exposed, a Prometheus server scrapes it on an interval, and Grafana queries Prometheus to build dashboards — the exact same pipeline used to monitor any other Spring Boot service, just now including token usage and model-tagged latency alongside your normal HTTP metrics.

◆ Under the hood — dashboarding what actually matters for AI cost

A Grafana panel summing gen_ai.client.token.usage over time, grouped by model tag, turns "our AI bill is high" from a monthly provider-invoice surprise into a queryable, real-time, per-feature (if you tag calls appropriately) cost signal — the same operational maturity teams expect for compute/storage cost, applied to token spend.

💻 Code example

management:
  endpoints:
    web:
      exposure:
        include: prometheus,health,metrics
  metrics:
    tags:
      application: my-ai-service

Metrics tell you that something is slow in aggregate; distributed tracing tells you which specific request was slow and where the time went — critical for AI calls specifically because a single user request can fan out into multiple model calls, tool invocations, and a vector store query (Module 15's workflows make this fan-out explicit), and you need to see the whole waterfall, not just a single aggregate latency number.

management:
  tracing:
    sampling:
      probability: 1.0
  otlp:
    tracing:
      endpoint: http://localhost:4318/v1/traces

▲ Pitfall

sampling.probability: 1.0 (trace every request) is reasonable for development but can add meaningful overhead and storage cost at real production volume — tune sampling down (and consider always sampling requests that fail evaluation or error out) rather than tracing 100% of traffic indefinitely once you're past initial rollout.

✓ Quick recap

What does Spring AI auto-instrument once Actuator + a metrics registry are present? ChatClient call count/latency and token usage, without manual timer/counter code. Why does tracing matter more for AI calls than for a typical single-hop REST endpoint? A single user request can fan out into multiple model calls, tool invocations, and vector store queries — tracing shows the whole waterfall, not just an aggregate latency number.

💻 Code example

management:
  tracing:
    sampling:
      probability: 1.0
  otlp:
    tracing:
      endpoint: http://localhost:4318/v1/traces

Want a visual for this concept?

Generate a diagram tailored to “Observability — Metrics & Tracing” — the AI picks whichever visual (flowchart, comparison, sequence, etc.) best fits.

Practice quiz

Next Step

Continue to Multi-Provider, Multi-Model & Secrets Management →← Back to all Spring AI chapters