Why Temperature 0 Fails to Guarantee Deterministic LLM Outputs
Despite popular belief, setting a large language model's temperature to zero does not guarantee deterministic results. Floating-point arithmetic rounding and parallel hardware execution cause top tokens to occasionally flip, leading identical completions to diverge significantly after generating roughly one hundred tokens. This impacts reproducibility in scientific research and production software applications.
Source: Towards Data Science