Why does the same prompt give a different answer each time?

Because the model doesn’t produce one fixed output. At each step it samples the next token from a probability distribution, and above temperature zero that sampling step alone can make an identical prompt land on a different token and diverge from there. Temperature zero, greedy decoding, narrows this but doesn’t remove it. A study from Thinking Machines Lab sent the same prompt to Qwen3-235B 1,000 times at temperature zero and got 80 different completions, and the cause wasn’t sampling at all. GPUs batch concurrent requests for efficiency, and floating-point addition isn’t associative, so the order results get summed in shifts with server load, nudging the math by a rounding error that compounds token by token until the output diverges. Making the kernels behave the same regardless of batch size brought all 1,000 completions back into agreement. So the variation comes from two layers stacked on top of each other: the sampling you control with temperature, and a serving stack underneath it that you mostly don’t.