ChatOptions, Response Types & Streaming
Same prompt, wildly different behavior depending on a handful of numeric knobs. This module is those knobs, plus the three shapes a response can come back in.
Learning objectives
- Beginner: Explain what the temperature parameter controls and why it's set near 0 for factual/extraction tasks.
- Intermediate: Configure ChatOptions (temperature, max tokens) per request rather than relying on provider defaults.
- Advanced: Implement a streaming chat response end to end and explain why it improves perceived latency without changing total generation time.
ChatOptions is Spring AI's portable options object — a common set of properties (model, temperature, max tokens) that map onto every provider's own parameter names underneath, plus an escape hatch for provider-specific options that don't generalize.
String creative = chatClient.prompt() .user("Write a tagline for a coffee shop") .options(ChatOptions.builder().temperature(0.9).maxTokens(60).build()) .call() .content();
💻 Code example
String creative = chatClient.prompt() .user("Write a tagline for a coffee shop") .options(ChatOptions.builder().temperature(0.9).maxTokens(60).build()) .call() .content();
Temperature controls how the model samples its next token from the probability distribution it computes internally. At every generation step, the model doesn't produce one "answer" — it produces a probability for every token in its vocabulary being next; temperature reshapes how sharply that distribution is followed.
| Temperature | Effect | Good for |
|---|---|---|
| 0.0 – 0.3 | Strongly favors the single most probable token — near-deterministic, focused | Factual Q&A, code generation, structured extraction |
| 0.5 – 0.8 | Balanced — some variety while staying coherent | General chat, summarization |
| 0.9 – 1.0 | Flattens the distribution — more surprising, less predictable token choices | Creative writing, brainstorming |
▲ Pitfall
A high temperature on a task requiring strict correctness (e.g. generating JSON for structured output, or a factual lookup) increases the odds of a plausible-but-wrong token being selected at exactly the wrong moment — for anything you'll parse or trust as fact, default low and only raise it deliberately.
maxTokens caps how many tokens the model is allowed to generate in its response — it does not extend the context window, it truncates output. Combined with §01.3's tokenization behavior, this is why a response can end abruptly mid-sentence: generation hit the cap, not a natural stopping point.
◆ Under the hood — metadata & usage
Every ChatResponse carries a Usage object reporting prompt tokens, completion tokens, and total tokens actually consumed by that call — this is the ground truth for cost and context-budget accounting, and should be what you log/monitor (see Module 17) rather than estimating token counts client-side.
ChatResponse response = chatClient.prompt().user("...").call().chatResponse(); Usage usage = response.getMetadata().getUsage(); log.info("prompt={} completion={} total={}", usage.getPromptTokens(), usage.getCompletionTokens(), usage.getTotalTokens());
💻 Code example
ChatResponse response = chatClient.prompt().user("...").call().chatResponse(); Usage usage = response.getMetadata().getUsage(); log.info("prompt={} completion={} total={}", usage.getPromptTokens(), usage.getCompletionTokens(), usage.getTotalTokens());
| Call | Returns | Use when |
|---|---|---|
| .call().content() | String | You just want the text. |
| .call().chatResponse() | ChatResponse | You need metadata — usage, finish reason, model name. |
| .call().entity(Type.class) | A typed Java object | Structured output (Module 05). |
| .stream().content() | Flux | You want to render the response token-by-token as it's generated. |
◆ The problem
A long response can take several seconds to fully generate. Waiting for the entire thing before showing anything to the user feels noticeably slower than a UI that renders tokens as they arrive — even though the total time to completion is identical either way.
@GetMapping(value = "/chat/stream", produces = MediaType.TEXT_EVENT_STREAM_VALUE) public Flux<String> streamChat(@RequestParam String question) { return chatClient.prompt() .user(question) .stream() .content(); // Flux<String> — one element per generated chunk }
◆ Under the hood
Streaming works by consuming the provider's own Server-Sent Events stream and re-emitting it as a reactive Flux — Spring AI doesn't wait for the full response and chop it up; it forwards chunks as the provider itself sends them. This means total latency to completion is the same as non-streaming, but time-to- first-visible-token drops dramatically, which is what actually improves perceived responsiveness.
✓ Quick recap
Does maxTokens extend how much context the model can read? No — it caps how much the model is allowed to generate; it truncates output, not input capacity. Why is a low temperature preferred for structured output tasks? It favors the single most probable token, reducing the odds of a plausible-but-incorrect token breaking a format you intend to parse. Does streaming reduce the total time until the full response is done generating? No — it reduces time to the first visible token, improving perceived speed, while total generation time is unchanged.
💻 Code example
@GetMapping(value = "/chat/stream", produces = MediaType.TEXT_EVENT_STREAM_VALUE) public Flux<String> streamChat(@RequestParam String question) { return chatClient.prompt() .user(question) .stream() .content(); // Flux<String> — one element per generated chunk }
Want a visual for this concept?
Generate a diagram tailored to “ChatOptions, Response Types & Streaming” — the AI picks whichever visual (flowchart, comparison, sequence, etc.) best fits.
Sign in to generate a visual →