Experiments
Where no source has the number, I measure it.
Each experiment has its own page: the question, the setup, the findings, the full results and the code to rerun it.
-
More thinking costs more tokens, not always more accuracy.
Four open models on one RTX 5090, thinking off and on, a single answer and an agent. Thinking multiplied tokens per task up to 6.8 times, while accuracy moved by −20 to +14 points.
Code and data are on GitHub.
Every experiment has a repository with the test code and the raw results.