Adaptive Thinking Budget
One model switches between no thinking and deep thinking based on difficulty.
Problem
Many current models can answer directly or reason at length first, with the amount of thinking set per request. A fixed setting is wrong for most traffic: always-deep wastes time and tokens on easy requests, and always-shallow fails the hard ones.
- Thinking tokens improve hard answers but add latency and cost.
- Request difficulty varies widely, even within one product.
- Two separate models mean two things to evaluate, version and monitor.
- The model’s own sense of difficulty is useful but not fully reliable.
Solution
Use one model that supports a thinking or effort setting, and choose the setting per request. A cheap policy picks the level from signals available before the call: task type, input length, user tier, stakes and past failure rates for similar requests. Some models can also adjust their own effort within a cap. Set a hard upper budget per request, and raise the level on retry when a check fails.
Diagram
Same model, different setting. The policy picks the effort level before the call; a failed check can retry one level higher.
When to use
Use it when
- Your model supports adjustable thinking or effort levels
- Request difficulty varies a lot within the same product
- You want fast/slow behavior without running two models
- You can measure quality per effort level to tune the policy
Avoid it when
- Your model has no effort control
- All requests are similar; just pick one fixed level
- You need a different model's strengths, not just more thinking
Trade-offs
Gains
- Fast/slow behavior with one model to evaluate and version
- Spending tracks difficulty instead of averaging it
- Simple to retry at a higher level when a check fails
Costs
- Policy errors put hard requests on low effort
- Thinking budgets and pricing differ across model versions
- Harder to compare quality when effort varies per request
Worked example
An internal coding assistant serves about 3,000 requests a day across an engineering org.
- PolicyReads signals before each call: task type, files touched and whether tests exist.
- Low effort“Rename this variable across the file.” Answers in 2 seconds with no thinking.
- Medium effort“Add input validation to this endpoint.” Spends a few thousand thinking tokens and answers in 9 seconds.
- High effort“Why does this race condition only appear under load?” Gets the maximum budget and answers in 70 seconds with a reproduction plan.
- RetryA medium-effort change fails the test suite. The policy retries once at high effort, and the tests pass.
- Weekly reviewAbout 70% of requests run at low effort. Rerun failures cluster in database migrations, so the policy now starts those at high.
Failure modes
| Failure | Symptom | Mitigation |
|---|---|---|
| Hard request at low effort | Quick, confident, wrong answers on tricky tasks | Retry one level higher when a check fails; learn from reruns |
| Effort creep | Policy drifts toward high effort for everything | Track the effort mix as a cost metric with a target |
| Overthinking simple tasks | Long reasoning that changes nothing on trivial requests | Cap budgets per task type; compare outputs at two levels |
| Version drift | A model update changes what each level costs or does | Re-baseline the policy on every model version change |