advanced~2h

Multi-Provider, Multi-Model & Secrets Management

Module 02 showed provider-swapping in principle. This module is running several providers side-by-side in one production application, choosing between them at runtime, and keeping every credential out of source control.

Learning objectives

Beginner: List at least three trade-offs (cost, latency, quality) that differ across LLM providers.
Intermediate: Implement runtime model selection so different requests can be routed to different providers/models.
Advanced: Design a secrets-management approach that keeps every provider API key out of source control in a multi-provider production deployment.

Provider	Spring AI starter	Typical reason to choose it
OpenAI	spring-ai-starter-model-openai	Broad capability, mature ecosystem, native structured output support.
Azure OpenAI	spring-ai-starter-model-azure-openai	OpenAI models inside an existing Azure tenant — compliance, VNet integration, enterprise billing.
AWS Bedrock	spring-ai-starter-model-bedrock-converse	Multiple model families behind one AWS-native API, inside your existing AWS security boundary.
Google Vertex AI (Gemini)	spring-ai-starter-model-vertex-ai-gemini	Gemini models inside GCP, large native context windows.
Mistral AI	spring-ai-starter-model-mistral-ai	Strong open-weight-lineage models, often favorable cost/performance for certain workloads.
Ollama	spring-ai-starter-model-ollama	Fully local, no external API dependency (Module 02 §3).

◆ The problem

Module 02 §6 wired up multiple ChatClient beans, chosen at compile time by which one your code injects. A real system often needs to pick a model per-request — e.g. a cheaper model for simple queries, a stronger one when a complexity threshold is crossed, or a specific model a paying customer has selected in settings.

@Service
public class ModelRouter {

    private final Map<String, ChatClient> clientsByName;

    public ModelRouter(List<NamedChatClient> clients) {
        this.clientsByName = clients.stream()
            .collect(Collectors.toMap(NamedChatClient::name, NamedChatClient::client));
    }

    public String chat(String modelName, String question) {
        ChatClient client = clientsByName.getOrDefault(modelName, clientsByName.get("default"));
        return client.prompt().user(question).call().content();
    }
}

▲ Pitfall

Different providers/models don't behave identically for the same prompt — a prompt tuned against GPT-4o-mini can produce meaningfully different (sometimes worse) results against a Mistral or local Ollama model without re-tuning. Runtime model selection means your test suite needs coverage across every model actually reachable in production, not just the one you developed against.

💻 Code example

@Service
public class ModelRouter {

    private final Map<String, ChatClient> clientsByName;

    public ModelRouter(List<NamedChatClient> clients) {
        this.clientsByName = clients.stream()
            .collect(Collectors.toMap(NamedChatClient::name, NamedChatClient::client));
    }

    public String chat(String modelName, String question) {
        ChatClient client = clientsByName.getOrDefault(modelName, clientsByName.get("default"));
        return client.prompt().user(question).call().content();
    }
}

◆ The problem

Module 02 resolved the API key from an environment variable — fine for local development, but environment variables on a production host are still visible to anything with process/shell access, and don't provide rotation, audit logging, or centralized revocation.

Production Spring AI deployments typically resolve credentials from a dedicated secrets manager at startup, not a plain environment variable baked into deployment config.

Secrets manager	Typical integration
Google Secret Manager	Spring Cloud GCP's secret manager config integration resolves sm://project/secret -style property values at startup.
AWS Secrets Manager	Spring Cloud AWS resolves secrets as property sources, fitting the same ${OPENAI_API_KEY} -style placeholder Module 02 already used.
Azure Key Vault	Spring Cloud Azure's Key Vault config integration works the same way, resolved automatically at application startup.

spring:
  cloud:
    aws:
      secretsmanager:
        region: us-east-1
spring.ai.openai.api-key: ${sm://prod/openai-api-key}

◆ Under the hood

The application code and even the ChatClient configuration don't change at all switching from a plain env var to a secrets manager — only the property source resolving spring.ai.openai.api-key changes, because Spring's property abstraction doesn't care where a value ultimately comes from. This is the same portability pattern that's run through every module of this site, applied to credentials instead of providers.

✓ Quick recap

What operational capability does a real secrets manager give you that a plain env var doesn't? Rotation, audit logging, and centralized revocation — not just "keeping the key out of source control." Why does runtime model selection increase your testing burden? A prompt tuned against one model/provider isn't guaranteed to perform identically on another reachable in production.

💻 Code example

spring:
  cloud:
    aws:
      secretsmanager:
        region: us-east-1
spring.ai.openai.api-key: ${sm://prod/openai-api-key}

Want a visual for this concept?

Generate a diagram tailored to “Multi-Provider, Multi-Model & Secrets Management” — the AI picks whichever visual (flowchart, comparison, sequence, etc.) best fits.

Practice quiz

Next Step

Continue to Multimodal — Transcription, Text-to-Speech & Image Generation →← Back to all Spring AI chapters