Temperature and Randomness: Why AI Sometimes Deviates

Reading time: approx. 5 min

In the previous moments, we have learned how we formulate effective prompts and how the AI processes information. Now we will explore an exciting aspect of AI models that affects how creative and varied their answers become: temperature and other related parameters for randomness. Understanding this helps you fine-tune the AI's behavior, from strictly fact-based answers to more imaginative texts.

1. What is temperature in AI contexts?

In AI models, temperature is a parameter that controls how "creative" or "random" the model chooses the next word (token) in its answer. The AI model calculates a probability for each possible next word.

Low temperature (often between 0.0 and 0.3): When the temperature is low, the model almost always chooses the words that have the highest probability of following. This leads to very conservative, direct, and predictable answers. The same prompt will usually give almost identical answers each time. It is like the AI always chooses the "safest" alternative.

High temperature (often between 0.7 and 1.0): At a higher temperature, the model allows a greater degree of randomness in its choice of next word. It can choose words that have a lower probability but are still reasonable. This results in more creative, varied, and sometimes unexpected answers. The same prompt can then give completely different results each time. It is like the AI gets "freer reins" to experiment.

2. How randomness affects the answers

The concept of "randomness" in AI is central to understanding temperature. When the AI generates text, it works roughly like this:

Probability distribution: After processing your prompt and the existing text, the AI creates a list of all possible next words and how likely it is that each word should come.

Word selection: Without temperature (or with temperature 0), the AI would always choose the word with the highest probability (this is called greedy sampling). With a higher temperature, the model "softens" this distribution, which makes even words with slightly lower probability have a chance to be chosen.

Other techniques: Besides temperature, there are other related techniques that affect the selection, such as Top-k (the model only chooses among the K most probable words) and Top-p (the model chooses among the most probable words whose combined probability reaches a certain percentage, for example 90%). These are used to further fine-tune the balance between creativity and relevance.

2.1 Deep dive: Top-K and Top-P

To get even more control over the AI's answers, you can use two additional techniques together with temperature:

Top-K sampling: Limits the AI to only choosing among the K most probable words for the next token. For example, if Top-K is set to 40, the AI will only consider the 40 most probable words as candidates for the next word. This prevents the AI from choosing very unlikely words that can lead to meaningless text.

Top-P sampling (Nucleus sampling): Instead of limiting to a fixed number of words, Top-P chooses the most probable words whose combined probability reaches a certain percentage. For example, if Top-P is set to 0.9 (90%), the AI will choose among the words that together constitute 90% of the probability for the next word. This is often more flexible than Top-K since the number of words can vary depending on how distributed the probability is.

Greedy decoding: When the temperature is set to 0, the AI uses so-called "greedy decoding," which means it always chooses the most probable word. This gives completely predictable results, the same prompt will always give exactly the same answer.

3. Concrete examples of temperature in practice

Let us see how the temperature can affect an answer in a school context:

Prompt: "Write a creative introduction to photosynthesis for students in grade 7."

With Temperature 0.0 (Low/Conservative):

Result: "Photosynthesis is a process where plants use sunlight, water, and carbon dioxide to make their own food. This is fundamental to life on Earth."

Analysis: Very straightforward, fact-based, and effective introduction. Good for basic facts and definitions.

With Temperature 0.8 (High/Creative):

Result: "Imagine a little green soldier, each plant is one of those, catching the sun's golden rays with its leaves like little catching nets. With each sip of water it drinks and each breath it takes of the air's carbon dioxide, it dances a magical dance that transforms everything into the energy it needs to grow and give us oxygen to breathe. That is photosynthesis, nature's own factory!"

Analysis: More descriptive, uses metaphors and unexpected formulations. Works well for inspiring, generating creative texts, or finding new angles.

4. Tips for teachers regarding temperature

Being able to adjust the temperature gives you a powerful tool for steering the AI's answers:

Balance precision and creativity:

Use low temperature (0.0-0.3) for tasks that require precision, fact-based answers, summary texts, proofreading, or generation of lists and tables. For example: "Summarize the most important points", "Generate a practice exercise".

Use high temperature (0.7-1.0) for creative exercises, idea generation, stories, poems, or when you want more variation in the answers. For example: "Write a short fairy tale", "Brainstorm ideas for a project", "Generate different scenarios".

Experiment: Test the same prompt with different temperature settings to see how the answers differ. This is the best way to get a feel for how the temperature affects the result and find the "right" tone for your task.

Document settings: If you find a temperature that works particularly well for a certain type of task or student group, note it. This can become part of your "prompt bank" and facilitate future work.

5. Troubleshooting: When AI gets stuck in repetitions

Sometimes AI models can get stuck in so-called "repetition loops" where they repeatedly write the same words or phrases. This happens most often when:

• The temperature is too low and Top-K or Top-P are incorrectly set
• The prompt is unclear and the AI does not know how it should end its answer
• The context window is full and the AI loses the thread

Solutions for repetition problems:

1. Increase the temperature somewhat (from 0.0 to 0.2-0.3) to introduce more variation
2. Adjust Top-K and Top-P to give the AI more alternatives to choose between
3. Reformulate the prompt with clearer instructions about desired length and structure
4. Restart the conversation if the repetition has gone on for a long time

6. Reflection exercise

To understand how the temperature works in practice:

Choose a prompt that asks the AI to be a bit creative, for example: "Write a short, imaginative description of what it looks like inside a cell."

Generate answers twice:

Once with temperature 0.0 (or as close to 0.0 as your AI service allows).

Once with temperature 0.9 (or as close to 0.9 as your AI service allows).

Compare the results: What differences do you notice? Which version is most "creative"? Which is most "precise"?

Discuss in the staff: When in your teaching can it be advantageous to steer the AI toward more creative answers (high temperature), and when is it crucial with high precision and strict facts (low temperature)? Which subjects can benefit most from adjusting the temperature?

Next moment: Hallucinations: Why AI sometimes "makes things up" - now that we have learned about the AI's creative abilities, it is important to understand one of its biggest challenges: that AI sometimes can generate incorrect information, so-called "hallucinations". We will learn how you can identify these errors and how you best handle them in the classroom.