Experiment 1 · November 2026

More thinking costs more tokens, not always more accuracy.

I ran four open AI models on my own graphics card and counted the tokens each task used: thinking off and on, a single answer and an agent. The price per token stayed fixed. Only the tokens per task and the success rate changed.

4 open models
12 settings
4 tasks
2,540 answers scored against a key
1 RTX 5090
Run on 8 November 2026

Code and data on GitHub Watch the video

The short version

Tokens per task decide the bill, and they move a lot.

×6.8tokens with thinking onSmall model, hard maths, and 20 points less accuracy.
×11tokens as an agentBut only ×3.5 the cost, because input is cheap.
×128cost, frontier vs small modelSame school maths questions, at list prices (projection).
×14speed gap on one GPU287 against 20 tokens per second.

Measure tokens per task and the success rate before you pay for more thinking, a bigger model or an agent.

Finding 1 · Thinking

More thinking means more tokens, not always more accuracy.

The same fifty hard maths problems, with thinking off and on. Tokens per problem rose 1.3 to 6.8 times. Accuracy moved by −20 to +14 points.

The small model thought until it ran out of room on 10 of the 50 problems, and solved fewer than without thinking. On a hundred school maths problems, thinking bought one to three points for 1.3 to 6.5 times the tokens.

Turn thinking on only where it raises the success rate on your own test.

Thinking multiplied tokens per problem by up to 6.8.

Hard maths (MATH-500, level 5), 50 problems: mean tokens per problem · % correct

Thinking off (gpt-oss: effort low)
Thinking on (Qwen3.8-27B: xhigh, its default; gpt-oss: effort high)

Source: own measurement, Dr. Kalun Ho (2026) · chart redrawn in brand style

Finding 2 · Classifying

The model that used the most tokens was not the most accurate.

A hundred customer messages, each to be sorted into one of 77 intents. The best result came from gpt-oss-20b at low effort: 87 % for 570 tokens per message. Qwen3.8-27B at its default effort scored 79 % for 676 tokens. The small model with thinking on reached 75 % for 2,856.

Test each task on several models before you pick one.

The best setting used a fifth of the tokens of the most expensive one.

Banking77, 100 customer messages, 77 intents: % correct by mean tokens per message, every setting

Thinking off or low effort
Thinking on, medium, high or xhigh

Source: own measurement, Dr. Kalun Ho (2026)

Finding 3 · Agents

An agent sends its whole context again at every step.

As an agent with three tools, Qwen3.8-27B answered all twenty invoice questions correctly, with 3.8 calls and 5,613 tokens per question. A single answer to a school maths problem took 494 tokens: the agent used 11 times as many, on a different task.

Each call sends everything seen so far. 93 % of the tokens are input, which costs a fifth of output at list prices. So the cost rises about 3.5 times, not 11.

Use an agent only when the task needs to look something up.

Every call sends more: 712 tokens, then 2,025 by the fourth.

Invoice agent, Qwen3.8-27B, thinking off: input tokens sent per call (mean of the 12 four-call questions)

Source: own measurement, Dr. Kalun Ho (2026)

Finding 4 · Cost

Which model answers moves the bill more than thinking or agents.

Priced at OpenAI’s list prices, a million school maths questions cost €144 with the small model and €18,325 with the frontier model: 128 times more. Thinking at xhigh adds a quarter (€22,859). The agent, on its invoice questions, comes to €64,643.

The token counts come from the open models and the prices from a commercial price list, so this is a projection, not a bill.

Route each task to the smallest model that passes your test.

The frontier model costs 128 times the small one.

€ per million questions, measured tokens at OpenAI list prices (projection)

Source: own measurement, Dr. Kalun Ho (2026); OpenAI API pricing, 8 November 2026; ECB rate 1 € = 1.1165 $

Finding 5 · Speed

On your own hardware, time is the bill.

One request at a time on one RTX 5090: gpt-oss-20b wrote 287 tokens per second, Qwen3.5-4B 157, Gemma 4 31B 71 and Qwen3.8-27B 20. That is 14 times apart.

Thinking multiplies the wait. The small model took 19 seconds per hard problem with thinking off, and 162 seconds with it on.

On your own hardware, measure the time per task, not only the tokens.

The fastest model writes 14 times as fast as the slowest.

Single stream, one RTX 5090, vLLM: output tokens per second (mean of 3 runs)

Source: own measurement, Dr. Kalun Ho (2026)

How it was measured

Same items for every model, success counted against a key.

Four tasks with known answers, so nothing is judged by taste. Every token of every call is counted, including the context an agent sends again at each step.

Task	Data	Items	Stands for	Success
Classify	Banking77 test set (PolyAI, CC BY 4.0), random sample	100	routing a support ticket	exact intent, of 77
Calculate	GSM8K test set (OpenAI, MIT), random sample	100	a multi-step business calculation	final number matches
Hard maths	MATH-500 (HuggingFaceH4, MIT), level 5, integer answers	50	a hard reasoning problem	integer matches
Agent	Synthetic invoice ledger: 500 invoices, 12 suppliers, 4 quarters; tools to list suppliers, fetch invoices and calculate; up to 15 steps	20	an analyst question answered by looking things up	matches the key computed from the ledger

Model	Maker, release	Build on the card	Settings tested
Qwen3.5-4B	Alibaba, March 2026	BF16	thinking off, thinking on
Qwen3.8-27B	Alibaba, August 2026	NVFP4, 4-bit	thinking off; effort low, medium, xhigh (its default); as an agent: thinking off, effort low
Gemma 4 31B	Google, 2026	QAT W4A16, 4-bit	thinking off, thinking on
gpt-oss-20b	OpenAI, August 2025	MXFP4	effort low, effort high

Hardware. One NVIDIA RTX 5090 with 32 GB, my own, at home. No employer hardware and no employer data.
Software. vLLM 0.26.0, one model at a time; Qwen3.8-27B ran on a vLLM development build for its 4-bit kernels. Context up to 32,768 tokens, answers up to 28,000.
Sampling. The model makers’ recommended settings for each mode. Items sampled with seed 42, requests with seed 42, one run per item.
Counting. Tokens per item are prompt plus completion tokens, summed over every call. Reasoning tokens are part of the completion. An answer that hit the limit counts as wrong.

Full results

Every setting, every task.

Means per item. The same numbers are in the repository as CSV, one row per item and per call.

Hard maths: MATH-500 level 5, 50 problems

Model	Setting	Correct	Accuracy	Tokens per item	Median	Reasoning tokens	Truncated	Tokens per correct answer
Qwen3.5-4B	thinking off	50 of 50	100%	2,422	1,441	0	0%	2,422
Qwen3.5-4B	thinking on	40 of 50	80%	16,400	18,760	15,553	20%	20,501
Qwen3.8-27B	thinking off	47 of 50	94%	2,318	1,572	0	0%	2,465
Qwen3.8-27B	effort low	48 of 50	96%	2,374	1,324	1,703	0%	2,473
Qwen3.8-27B	effort medium (no instruction)	49 of 50	98%	1,969	1,298	1,352	0%	2,009
Qwen3.8-27B	effort xhigh (default)	49 of 50	98%	3,027	1,584	2,551	0%	3,089
Gemma 4 31B	thinking off	49 of 50	98%	897	841	0	0%	916
Gemma 4 31B	thinking on	50 of 50	100%	4,319	3,948	3,491	0%	4,319
gpt-oss-20b	effort low	42 of 50	84%	1,224	905	620	0%	1,457
gpt-oss-20b	effort high	49 of 50	98%	7,954	4,106	7,362	2%	8,117

Classify: Banking77, 100 customer messages, 77 intents

Model	Setting	Correct	Accuracy	Tokens per item	Median	Reasoning tokens	Truncated	Tokens per correct answer
Qwen3.5-4B	thinking off	70 of 100	70%	456	453	0	0%	651
Qwen3.5-4B	thinking on	75 of 100	75%	2,856	1,828	2,400	0%	3,808
Qwen3.8-27B	thinking off	71 of 100	71%	456	454	0	0%	642
Qwen3.8-27B	effort low	79 of 100	79%	632	622	146	0%	800
Qwen3.8-27B	effort medium (no instruction)	80 of 100	80%	664	648	209	0%	830
Qwen3.8-27B	effort xhigh (default)	79 of 100	79%	676	579	178	0%	855
Gemma 4 31B	thinking off	81 of 100	81%	603	601	0	0%	744
Gemma 4 31B	thinking on	85 of 100	85%	987	876	377	0%	1,161
gpt-oss-20b	effort low	87 of 100	87%	570	555	41	0%	656
gpt-oss-20b	effort high	84 of 100	84%	979	782	450	0%	1,165

Calculate: GSM8K, 100 school maths problems

Model	Setting	Correct	Accuracy	Tokens per item	Median	Reasoning tokens	Truncated	Tokens per correct answer
Qwen3.5-4B	thinking off	92 of 100	92%	406	364	0	0%	441
Qwen3.5-4B	thinking on	93 of 100	93%	2,641	1,326	2,318	2%	2,840
Qwen3.8-27B	thinking off	96 of 100	96%	494	425	0	0%	515
Qwen3.8-27B	effort low	96 of 100	96%	434	396	172	0%	452
Qwen3.8-27B	effort medium (no instruction)	96 of 100	96%	449	387	203	0%	468
Qwen3.8-27B	effort xhigh (default)	97 of 100	97%	627	319	419	0%	647
Gemma 4 31B	thinking off	94 of 100	94%	341	320	0	0%	363
Gemma 4 31B	thinking on	97 of 100	97%	1,401	802	1,032	0%	1,445
gpt-oss-20b	effort low	96 of 100	96%	254	234	83	0%	265
gpt-oss-20b	effort high	97 of 100	97%	995	618	814	0%	1,025

Agent: 20 invoice questions, 3 tools, up to 15 steps

Model	Setting	Correct	Accuracy	Tokens per item	Median	Reasoning tokens	Calls per item	Truncated	Tokens per correct answer
Qwen3.8-27B	agent, thinking off	20 of 20	100%	5,613	5,920	0	3.8	0%	5,613
Qwen3.8-27B	agent, effort low	20 of 20	100%	5,477	4,826	380	3.3	0%	5,477

Speed: one request at a time

Model	Time to first token	Tokens per second	Output tokens	Runs
gpt-oss-20b, effort low	0.05 s	286.6	649	3
Qwen3.5-4B, thinking off	0.07 s	157.1	607	3
Gemma 4 31B, thinking off	0.05 s	71.3	425	3
Qwen3.8-27B, thinking off	0.16 s	20.3	545	3

Decode speed after the first token, mean of 3 runs. Speed depends on the 4-bit format, the vLLM version and how many requests share the card.

Limits

A measurement of this setup, not of AI in general.

Open models only. Commercial models count and reason differently. The multipliers are the point, not the absolute token counts.
Benchmarks stand in for work. The tasks have answer keys, real business tasks rarely do. The agent task is synthetic.
Small samples. One run per item, 20 agent questions and 50 hard problems. A second seed would show the spread.
Effort is an instruction. For Qwen3.8-27B, “effort” is a line in the system prompt, not a token budget, and “medium” adds nothing.
Euros are a projection. Measured tokens at commercial list prices, which change.

Data and code

Check it, rerun it, cite it.

The test code, the answer keys and the raw results, one row per item and per call, are on GitHub, together with a technical report that adds confidence intervals and the full method. Questions and corrections are welcome there as issues.

kalunho17/yt-experiment-001 Technical report

How to cite

Kalun Ho (2026). Cheaper Tokens, Costlier Tasks: Measuring Tokens per Task Across Reasoning Settings, Model Size and Agentic Tool Use. Technical report. https://github.com/kalunho17/yt-experiment-001

Licences: code MIT; results, figures, report and the invoice ledger CC BY 4.0. The benchmarks keep their own licences: Banking77 (PolyAI, CC BY 4.0), GSM8K (OpenAI, MIT), MATH-500 (HuggingFaceH4, MIT).

The video explains what it means for your bill.

Subscribe on YouTube to see it. The handout has every chart and source.

Subscribe on YouTube