advanced~2h

Multimodal — Transcription, Text-to-Speech & Image Generation

Everything so far assumed text in, text out. Spring AI's provider-agnostic pattern extends to audio and images with the same shape of API.

Learning objectives

Beginner: Use Spring AI's TranscriptionModel to convert an audio file to text.
Intermediate: Explain why image generation typically routes to a separate model/API rather than the same chat model producing images inline.
Advanced: Combine transcription, chat, and image generation in one application using Spring AI's consistent provider-agnostic API shape.

TranscriptionModel converts spoken audio into text — the audio equivalent of an embedding model's job, mapping one modality into a form downstream text-based logic (including a normal ChatClient call) can consume directly.

@PostMapping("/transcribe")
public String transcribe(@RequestParam MultipartFile audio) {
    AudioTranscriptionPrompt prompt = new AudioTranscriptionPrompt(new ClassPathResource... );
    return transcriptionModel.call(
        new AudioTranscriptionPrompt(audio.getResource())).getResult().getOutput();
}

A common real-world composition: transcribe a customer call, then feed the transcript straight into a ChatClient call for summarization or sentiment classification — exactly the Chain workflow pattern from Module 15, just with a transcription step as the first link instead of another chat call.

💻 Code example

@PostMapping("/transcribe")
public String transcribe(@RequestParam MultipartFile audio) {
    AudioTranscriptionPrompt prompt = new AudioTranscriptionPrompt(new ClassPathResource... );
    return transcriptionModel.call(
        new AudioTranscriptionPrompt(audio.getResource())).getResult().getOutput();
}

@GetMapping(value = "/announce", produces = "audio/mpeg")
public byte[] announce(@RequestParam String text) {
    TextToSpeechPrompt prompt = new TextToSpeechPrompt(text,
        OpenAiAudioSpeechOptions.builder().voice("alloy").speed(1.0).build());
    return textToSpeechModel.call(prompt).getResult().getOutput();
}

▲ Version note

Older Spring AI builds (and some tutorials still circulating) use SpeechPrompt/SpeechModel/SpeechResponse — these classes were removed and replaced by TextToSpeechPrompt/TextToSpeechModel/TextToSpeechResponse. If you see the old names in an example, treat it as stale.

A real production use case: a flight/gate announcement system generating spoken audio dynamically from structured flight data (rather than a fixed set of pre-recorded phrases), or streaming synthesized audio incrementally rather than waiting for the entire clip to render before playback starts.

💻 Code example

@GetMapping(value = "/announce", produces = "audio/mpeg")
public byte[] announce(@RequestParam String text) {
    TextToSpeechPrompt prompt = new TextToSpeechPrompt(text,
        OpenAiAudioSpeechOptions.builder().voice("alloy").speed(1.0).build());
    return textToSpeechModel.call(prompt).getResult().getOutput();
}

ImageResponse response = imageModel.call(
    new ImagePrompt("A minimalist logo for a Java coffee-themed backend startup",
        OpenAiImageOptions.builder().width(1024).height(1024).build()));

String imageUrl = response.getResult().getOutput().getUrl();

◆ Under the hood — same shape, different modality

Notice the pattern repeats exactly across Transcription, TTS, and Image Generation: a *Model interface, a *Prompt input type, provider-specific *Options, and a.call() returning a typed result. Spring AI deliberately keeps the same request/response shape across every modality — once you know the pattern from ChatModel, every other model type is a variation on it, not a new API to learn from scratch.

✓ Quick recap

What common structural pattern do ChatModel, TranscriptionModel, TextToSpeechModel, and ImageModel all share? A *Model interface, a *Prompt input, provider-specific *Options, and a.call() returning a typed result — the same shape across every modality.

💻 Code example

ImageResponse response = imageModel.call(
    new ImagePrompt("A minimalist logo for a Java coffee-themed backend startup",
        OpenAiImageOptions.builder().width(1024).height(1024).build()));

String imageUrl = response.getResult().getOutput().getUrl();

Want a visual for this concept?

Generate a diagram tailored to “Multimodal — Transcription, Text-to-Speech & Image Generation” — the AI picks whichever visual (flowchart, comparison, sequence, etc.) best fits.

Practice quiz

Next Step

Continue to Capstone — Building a Real-World AI Agent →← Back to all Spring AI chapters