Understanding temperature, top_p, top_k, logit_bias in LLM parameters
If you have been following around the LLM world and have experimented around, you must have come across terms like temperature, top_p, top_k and others. If you have not, here is a link to OpenAI playground where you can try them out and try to derive how they affect the variety, tone and even accuracy of the generated output from the model.
In this blog post, I go over how these parameters are able to control the model responses.
Background
Transformer-based LLMs predict the next token by considering the entire sequence of previous tokens, using an attention mechanism to weigh their relevance. The generation process repeats this for each new token until a designated stopping condition is met.
The model calculates logits for all possible next tokens based on the context of the input sequence. These logits represent the raw predictions of the model for each token. A softmax function is then applied to the logits, converting them into probabilities that sum to one. Based on these probabilities, the model samples the next token, which may be done randomly or in a more deterministic fashion depending on the sampling strategy used.
temperature
According to OpenAI API spec, this controls randomness. Lower value means more predictable output while higher allow for more creativity. The value is between 0 and 2.
As mentioned above the token logits are converted to probabilities using the softmax function, however that isn’t true exactly. Instead of directly calculating softmax, the logits are divided by the temperature value resulting in below equation:
Once the logits are divided by the temperature (and then softmax’ed), the distribution of probabilities becomes more even, which increases the chance of selecting a wider variety of tokens and makes the output more random.
As a side note, I like to remember this by the analogy that a hotheaded (read higher temperature) person can say anything, so can an LLM.
top_p
The top_p controls the model output by augmenting the vocabulary size as only those tokens are considered for which the cumulative probability is greater than the top_p value (the cumulative probability is calculated by sorting the tokens by their probability in a descending order and then adding them up). After the tokens are selected, we re-calculate their softmax with reduced vocab size.
close to 1 (e.g. 0.95): A large portion of the vocabulary is considered, making the output more random. This is because even tokens with relatively low probabilities are included in the nucleus from which we’re sampling.
close to 0 (e.g. 0.1): Only the very top tokens (those with the highest probabilities) are considered, making the output more deterministic.
top_k
This works in similar way as top_p by reducing the vocabulary size but instead of cumulative probability it works by reducing the vocabulary size to top k tokens (sorted in descending order by their probabilites). Similar to top_p, we re-calculate probabilities (using softmax on logits) on reduced vocabulary.
logit_bias
The open AI API definition covers this pretty well:
Modify the likelihood of specified tokens appearing in the completion.
Accepts a json object that maps tokens (specified by their token ID in the tokenizer) to an associated bias value from -100 to 100. Mathematically, the bias is added to the logits generated by the model prior to sampling. The exact effect will vary per model, but values between -1 and 1 should decrease or increase likelihood of selection; values like -100 or 100 should result in a ban or exclusive selection of the relevant token.
This one is simple to understand as well, the values supplied here are directly added to raw token logit values on each completion step.
Although this one is simple to understand, I couldn’t completely connect the dots till I came across a use-case of this in guidance-ai library. Guidance allows users to do selection based generation where users can provide valid options and the LLM would do a completion only within those options, something like this:
{
"armor": "{{#select 'armor'}}leather{{or}}chainmail{{or}}plate{{/select}}",
"weapon": "{{select 'weapon' options=valid_weapons}}"
}They achieve this internally by increasing the logit bias of the valid options provided in the generation template. Following is the code-snippet from guidance codebase.
Conclusion
Alright, that’s your crash course on LLM knobs and levers. Toy around with them — see how temperature, top_p, top_k, and logit bias can turn your text generator from a predictable bot into a creative genius.
Got questions, feedback or cool findings? Drop a comment below. I’m all ears.
Thanks for sticking around, catch you on the next one and happy LLM’ing!
