Temperature and Sampling Settings, Explained

Large language models do not think in sentences; they generate text one token at a time. At every single step, the model calculates a probability distribution for what the next token should be, based on the prompt and the text generated so far. Sampling settings—like temperature, top-p, top-k, and penalties—are the API parameters that dictate how the model selects from that distribution. For AI builders, understanding these dials is essential. They determine whether your application produces consistent, structured data for a database, or varied and creative text for a user-facing chatbot.

Understanding these parameters allows you to move beyond default API settings and tailor model behavior to specific tasks. Pushing these dials to their extremes often degrades performance, but knowing how to balance them gives you precise control over the reliability and tone of your application.

### How LLMs Choose the Next Token

When an LLM processes a prompt, it assigns a raw score, known as a logit, to every possible token in its vocabulary. These scores are normalized into percentages. If you always instruct the model to pick the token with the highest percentage, you get a process called greedy decoding. Greedy decoding is highly deterministic, but it can lead to unnatural, repetitive text or looping behavior. To make models generate fluid language, we use sampling—essentially rolling a weighted die based on the calculated probabilities. The sampling parameters you set in your code dictate exactly how that die is loaded and which tokens are allowed to compete.

### Temperature: Flattening or Sharpening the Odds

Temperature scales the logits before they are converted into probabilities. A temperature of 1.0 leaves the model's native distribution unchanged. Lowering the temperature below 1.0 sharpens the distribution, making the most likely tokens even more probable and pushing the probabilities of unlikely tokens closer to zero. A low temperature, such as 0.0 to 0.2, makes the model highly deterministic, focused, and sometimes repetitive. Setting the temperature to 0.0 usually forces greedy decoding, which is useful for reproducing bugs or getting strict factual answers.

Conversely, raising the temperature above 1.0 flattens the distribution, giving lower-probability tokens a better chance of being chosen. This introduces variety, but if pushed too high, the model will produce incoherent text. As a builder, you should use low temperatures for tasks requiring strict accuracy: data extraction, JSON generation, or writing automated tests. Use higher temperatures, typically 0.7 to 0.9, for brainstorming, creative writing, or generating diverse synthetic training data.

### Top-p and Top-k: Truncating the Tail

Even with a high temperature, you rarely want the model to consider the entire vocabulary. While common words might have high probabilities, the model's distribution also contains a long tail of completely nonsensical tokens. Top-k and top-p are methods to cut off this tail, ensuring the model only rolls its die among the most plausible candidates.

Top-k limits the pool to a fixed number of the highest-scoring tokens, such as the top 40. The problem with top-k is that the ideal pool size changes depending on the context. Sometimes the next token is incredibly obvious, and the top 3 tokens capture all the probability. Other times, the next token is ambiguous, and the top 40 might include irrelevant noise. Top-p, also known as nucleus sampling, solves this by dynamically limiting the pool to the smallest set of tokens whose cumulative probability exceeds a threshold, such as 0.9. Top-p adapts to the model's confidence. When the model is certain, the nucleus is small; when it is uncertain, the nucleus is large. Most builders rely on top-p rather than top-k. If you set top-p to a very low number, like 0.1, the model will only ever consider the absolute most likely tokens, making the temperature setting largely irrelevant. A common, safe default for general tasks is a temperature of 0.7 and a top-p of 0.9.

### Frequency and Presence Penalties

While temperature and top-p alter the base probability distribution, frequency and presence penalties address token repetition directly. A frequency penalty reduces the probability of a token proportionally to how many times it has already appeared in the generated output. A presence penalty applies a flat reduction to any token that has already appeared at least once, regardless of how often.

These penalties are useful for long-form text generation, where a model might get stuck in a loop repeating a specific phrase. They encourage the model to introduce new topics or vocabulary. However, they can be actively harmful for code generation or structured data, where repetition is often correct and necessary. Programming languages rely on repeated keywords, brackets, and standard library calls. Applying presence penalties to a code-writing agent will often break the syntax. Start with these penalties at zero and only increase them if you notice looping behavior in conversational or narrative tasks.

### Choosing Settings for Your App

The right configuration depends entirely on your specific use case. If you are building a script to parse job postings from the Neural Goldmine feed into a structured JSON format, use temperature 0, top-p 1, and no penalties. You want strict adherence to the schema. If you are building a tool that drafts cover letters for remote AI jobs, a temperature of 0.7 and top-p of 0.9 will produce more natural, human-sounding language that does not repeat itself.

When evaluating your application, adjust parameters in isolation. If you change temperature and top-p simultaneously, it becomes difficult to determine which setting caused a change in behavior. Start with the API provider's defaults, run a small set of evaluation prompts, and adjust one parameter at a time. Keep in mind that extreme settings often degrade performance. A temperature of 0.0 can cause loops, while a temperature of 1.5 might produce gibberish. Moderation and careful testing are always more effective than pushing the dials to their extremes.