7 LLM Generation Parameters
The 7 levers that control every LLM generation — max tokens, temperature, top-k, top-p, frequency penalty, presence penalty, and stop sequences — plus bonus min-p sampling.
KV Parameters:
7 LLM Generation Parameters
Each lever controls a different aspect of generation. Temperature/top-k/top-p/min-p tune randomness; the rest shape length and repetition behavior.
| Controls | Low value | High value | |
|---|---|---|---|
| Max tokens | Output length limit | Cuts text off early (risk: truncation) | Wastes compute on unnecessarily long output |
| Temperature | Randomness / focus | Deterministic, focused, repetitive | Creative, diverse, sometimes incoherent |
| Top-k | Candidate pool size | Very restricted vocabulary choices | Wider, more varied vocabulary choices |
| Top-p (nucleus) | Cumulative probability cutoff | Only the most likely tokens considered | Long tail of unlikely tokens allowed in |
| Min-p | Minimum relative probability | Aggressively prunes low-probability tokens | Keeps more low-probability tokens available |
| Frequency penalty | Repetition tolerance | Repetition-friendly (e.g. poetry) | Repetition-discouraging (e.g. summarization) |
| Presence penalty | Topic novelty | Sticks to known patterns | Pushes toward novel topics |
| Stop sequences | Hard cutoff | — | Generation stops the instant the string appears (unlike the other 6, gradual, levers) |
▶📚 Prerequisites(1)
🎓 Learning objectives
- •Name and configure all 7 generation parameters plus bonus min-p sampling
- •Explain the difference between frequency penalty and presence penalty
- •Use stop sequences to enforce strict structured-output boundaries
- •Choose an appropriate max_tokens value to avoid truncation without wasting compute
What is it?
Every generation from an LLM is shaped by parameters under the hood, and knowing how to tune them is what separates sharp, controlled outputs from noisy or truncated ones. There are 7 levers that matter most: max tokens (a hard cap on response length), temperature (controls randomness), top-k (restricts sampling to the k most probable tokens), top-p / nucleus sampling (samples from the smallest set of tokens covering a cumulative probability mass), frequency penalty (discourages reusing already-frequent tokens), presence penalty (encourages introducing tokens not yet seen), and stop sequences (custom tokens that immediately halt generation) — plus a bonus 8th lever, min-p sampling, which dynamically tightens or loosens the sampling pool based on the model's confidence. Note: temperature, top-k, top-p, and min-p are covered in full mathematical depth in the companion 'Text Generation: Decoding & Sampling' topic — this topic gives them a brief treatment for completeness and focuses full depth on max tokens, frequency penalty, presence penalty, and stop sequences.
Why it exists
An LLM's raw next-token probability distribution, used unmodified, doesn't automatically produce the kind of output a given application needs — a chatbot answering factual questions needs different behavior than a brainstorming tool, and a structured-JSON-extraction pipeline needs strict boundaries a free-form chat response doesn't. These 7 parameters exist because 'just call the model' isn't enough — each parameter gives you a specific, composable lever to shape length, randomness, repetition, novelty, and termination behavior to match your specific use case, without needing a different model or a different prompt for each variation.
Problem it solves
Max tokens solves the problem of runaway or truncated generations — too low cuts off a response mid-thought, too high wastes compute on responses that should have ended already. Frequency penalty solves the 'the model keeps repeating the same phrase' problem common in summarization and long-form generation. Presence penalty solves the 'the model keeps circling back to the same few ideas' problem in exploratory or brainstorming use cases. Stop sequences solve the 'the model keeps generating extra commentary after the actual answer' problem, which is especially damaging for structured outputs like JSON where trailing text breaks a downstream parser.
Intuition
Think of generating text as pouring water (the model's raw probability distribution) through a series of adjustable filters before it reaches the glass (the final output). Max tokens is simply how big the glass is — pour stops when it's full, whether or not you're done pouring. Frequency penalty is like a filter that gets progressively harder to pass through for water that's already flowed a lot — discouraging repetition. Presence penalty is like a filter that specifically favors water that hasn't flowed at all yet — encouraging novelty. Stop sequences are like a sensor that instantly cuts off the tap the moment a specific marker appears in the glass, regardless of how full it is.
Analogy
Think of max_tokens like a strict word-count limit given to a writer: it protects against a submission going on forever, but if set too low, an important conclusion might get cut off unfinished. Frequency penalty is like an editor flagging 'you've used this word 6 times already, try a synonym,' discouraging redundancy. Presence penalty is like a brainstorming facilitator specifically nudging 'we haven't heard from this direction yet,' pushing toward covering new ground rather than circling the same few ideas repeatedly. Stop sequences are like telling a speaker 'the moment you say the word done, stop talking immediately, no matter what' — a hard, unambiguous boundary rather than a soft nudge.
Technical explanation
(1) Max tokens is a hard cap on how many tokens the model can generate in a single response — set too low and outputs get truncated mid-thought; set too high and you risk wasted compute on generations that should have naturally ended sooner.
(2-4) Temperature, top-k, and top-p (nucleus sampling), plus the bonus min-p — govern randomness, restrict sampling to the k most probable tokens, sample from the smallest token set covering a cumulative probability mass, and dynamically tighten/loosen the sampling pool based on model confidence, respectively; these are covered in full mathematical depth in the companion Text Generation: Decoding & Sampling topic.
(5) Frequency penalty reduces the likelihood of reusing tokens that have already appeared frequently in the generation so far — positive values discourage repetition (useful for summarization, where redundancy hurts quality), while negative values exaggerate it (occasionally useful for tasks like poetry where intentional repetition/refrain is desirable).
(6) Presence penalty encourages the model to bring in new tokens not yet seen in the text at all (as opposed to frequency penalty, which cares about how OFTEN a token has appeared) — higher values push for novelty and topic diversity, lower values let the model stick to familiar, already-established patterns; handy for exploratory generation where diversity of ideas is specifically valued.
(7) Stop sequences are a custom list of tokens that immediately halt generation the moment they appear — critical for structured outputs (e.g., generating JSON and stopping the instant a closing brace + specific marker appears), preventing spillover text that would otherwise need to be manually stripped, and letting you enforce strict response boundaries without heavy additional prompt engineering.
Architecture
These 7 parameters sit in the logit-processing pipeline between the model's raw output and the final selected token: raw logits → (frequency penalty and presence penalty adjustments, applied based on tokens already generated so far) → (temperature scaling, top-k/top-p/min-p filtering, covered elsewhere) → sample next token → append to output → check against stop sequences (halt if matched) → check against max_tokens (halt if reached) → repeat. Frequency and presence penalties are unique among the 7 in that they depend on the generation's own history so far (what's already been produced), unlike temperature/top-k/top-p/min-p which operate purely on the current step's distribution.
Workflow
- Set max_tokens based on your expected response length plus a safety margin — too tight risks truncation, especially for variable-length tasks; monitor for truncated responses in production and adjust.
- For factual/QA/deterministic tasks, keep temperature low; for creative/brainstorming tasks, raise it — this is the primary randomness lever, covered in depth elsewhere.
- If you notice repetitive phrases or redundant content in long-form or summarization outputs, add a positive frequency penalty.
- If you notice the model repeatedly circling back to the same few ideas in exploratory/brainstorming contexts, add a positive presence penalty to push toward novel territory.
- For any structured output format (JSON, XML, a specific delimiter-based format), define explicit stop sequences matching your format's natural termination point, rather than relying on max_tokens alone or post-hoc string trimming.
- Tune frequency/presence penalties empirically and separately — they solve related but distinct problems (how often vs. whether at all) and shouldn't be conflated or treated as interchangeable.
Example
from openai import OpenAI
client = OpenAI()
Structured JSON extraction: tight boundaries, no repetition/novelty pressure
extraction_response = client.chat.completions.create( model='gpt-4.1', messages=[{'role': 'user', 'content': 'Extract fields as JSON: ...'}], max_tokens=200, temperature=0.0, stop=['\n\n', '```'], # halt the moment structured output ends )
Open-ended brainstorming: encourage novelty, discourage repeated phrasing
brainstorm_response = client.chat.completions.create( model='gpt-4.1', messages=[{'role': 'user', 'content': 'Give me 10 startup name ideas'}], max_tokens=500, temperature=0.9, frequency_penalty=0.6, # discourage reusing the same words/roots presence_penalty=0.8, # push toward genuinely different naming directions )
Real-world usage
Summarization products (meeting-notes summarizers, article condensers) commonly apply a positive frequency penalty specifically to avoid the common LLM failure mode of restating the same point in slightly different words. Brainstorming and ideation tools (naming generators, creative-writing assistants) apply presence penalty to push the model away from its 2-3 most statistically likely, generic suggestions toward more varied territory. Structured-data-extraction pipelines (invoice parsers, JSON-API-response generators) rely heavily on stop sequences to guarantee the output terminates exactly at the closing structure marker, with no downstream regex-based text-cleanup needed. Chat applications with strict per-turn cost or latency budgets tune max_tokens carefully as a direct cost-control lever, since output token count is often the dominant cost component of an API call.
Trade-offs
Setting max_tokens too conservatively risks truncated, unusable responses; setting it too generously risks wasted cost on responses that should have ended sooner. Frequency and presence penalties, if set too aggressively, can push the model toward genuinely incoherent or off-topic output in pursuit of avoiding repetition/staying novel — these are levers to nudge behavior, not hard constraints, and extreme values degrade quality. Stop sequences are the most 'free' of the 7 — they add negligible cost or risk and simply enforce a boundary you already know you want, though choosing a stop sequence that could plausibly appear in a legitimate part of the intended output will cause premature truncation.
Visual explanation
A 7-lever control panel diagram, each lever labeled with its effect direction. Max tokens: a length-limit slider (too low → scissors icon cutting off text; too high → wasted-compute icon). Temperature/Top-k/Top-p/Min-p: grouped together as 'already covered — see Text Generation: Decoding & Sampling,' shown as a single dial labeled 'randomness/focus control.' Frequency penalty: a slider from negative (repetition-friendly, e.g. poetry) to positive (repetition-discouraging, e.g. summarization). Presence penalty: a slider from low (stick to known patterns) to high (push for novel topics). Stop sequences: a hard 'STOP' sign icon triggered the instant a specified string appears in the generated output, cutting the text stream immediately, contrasted with the soft, gradual effect of the other 6 levers.
Advantages
- —
Gives fine-grained, composable control over length, randomness, repetition, novelty, and termination without needing a different model per use case
- —
Frequency and presence penalties directly target two distinct, common LLM failure modes (repetition, idea-circling) with a simple numeric lever each
- —
Stop sequences let you enforce strict structured-output boundaries cheaply, without post-hoc text cleanup or additional prompt engineering
- —
All 7 parameters are supported natively by essentially every major LLM provider's API, making this knowledge broadly portable
Disadvantages
- —
Frequency and presence penalties are easy to confuse with each other despite solving distinct problems
- —
Overly aggressive penalty values can push output toward incoherence in pursuit of avoiding repetition or staying novel
- —
Max tokens set too low silently truncates responses, which can be a subtle production bug if not actively monitored
- —
Stop sequences that overlap with legitimate output content cause premature, incorrect truncation
Common mistakes
- —
Confusing frequency penalty (discourages tokens that have appeared often) with presence penalty (discourages tokens that have appeared at all), and tuning the wrong one for the observed problem
- —
Setting max_tokens far higher than needed 'just to be safe,' silently increasing cost and sometimes perceived latency without any quality benefit
- —
Not using stop sequences for structured-output tasks and instead relying on brittle post-hoc regex/string trimming to remove trailing commentary
- —
Applying strong frequency or presence penalties to short, factual responses where they provide no benefit and only risk introducing incoherence
- —
Not monitoring production traffic for truncated (max_tokens-limited) responses, missing a real but silent quality issue
📂 Subtopics
Temperature, Top-k & Top-p: What They Control and How to Tune Them
Temperature reshapes how sharply the model favors its top choice; top-k restricts sampling to a fixed number of candidates; top-p restricts sampling to the smallest set covering a probability mass. Three related but distinct dials for the same underlying randomness.
~20 min
Max Tokens, Stop Sequences & Frequency/Presence Penalties
Max tokens caps response length; stop sequences give you a hard exit point for structured output; frequency and presence penalties push the model away from repeating itself or toward exploring new ground.
~15 min
Sampling Strategies: Greedy, Multinomial, Beam Search & Contrastive Search
Even with a probability distribution over next tokens, you still need a strategy to actually pick one. The 4 main strategies — greedy, multinomial, beam search, contrastive search — trade off speed, coherence, and diversity very differently.
~20 min
Practical Tuning Guide: Which Parameters to Change for Which Use Case
A concrete parameter cheat sheet across 3 common use cases — creative writing, factual QA, and code generation — showing which of the 7 levers actually matter for each, and in which direction to tune them.
~15 min