Fast/Slow Escalation
System 1 by default; escalate to System 2 when signals say the task is hard.
Problem
Reasoning models are accurate but slow and expensive. Fast models are cheap and responsive but make confident mistakes on hard or unusual requests. Sending everything to the reasoning model wastes money and latency; sending everything to the fast model ships errors.
- Latency and cost push toward the fast model on every turn.
- Accuracy on hard or high-stakes cases pushes toward the reasoning model.
- Fast models are poor judges of their own confidence, so the switch cannot rely on self-report alone.
- Most real traffic is routine, so the savings from defaulting fast are large.
Solution
Run every request through a System 1 model first. An escalation gate checks a small set of signals: model confidence, novelty (distance from known cases), business stakes, and the result of a cheap rule or schema check. If any signal trips, the same request, plus System 1’s draft, goes to the System 2 model. Log every gate decision so the thresholds can be tuned against real outcomes.
Diagram
The gate sits between the models. Dashed path: escalation, typically 10–25% of traffic.
When to use
Use it when
- Traffic is mostly routine with a long tail of hard cases
- Latency matters, such as chat, voice or customer-facing flows
- You can define stakes in business terms, such as refund amount or account tier
- You have logs to calibrate thresholds
Avoid it when
- Every request is hard, so escalation would fire constantly
- A single wrong answer is unacceptable; use System 2 plus human review
- You have no way to measure whether escalations were correct
Trade-offs
Gains
- Large cost and latency savings on routine traffic
- Reasoning budget concentrated where it changes outcomes
- Gate logs give a clear view of what users find hard
Costs
- Threshold tuning is ongoing work
- Two models to version, evaluate and monitor
- Misses happen when System 1 is confidently wrong and no external signal fires
Worked example
A retail support agent handles order questions. Refunds over $200 need policy reasoning.
- User“Where’s my order #4412?”
- System 1Answers from the order lookup tool. Confidence 0.94, known intent, low stakes.
gate: pass. Reply in 0.6s. - User“It arrived broken. I want a $340 refund plus shipping on the replacement.”
- System 1Drafts an approval. Confidence 0.81, but the stakes rule trips (refund > $200).
gate: escalate (stakes) - System 2Reads the policy, notices the item was a final-sale clearance product, and drafts a replacement-only offer with an apology credit.
- Gate logRecords the escalation reason and both drafts. Weekly review shows a 14% escalation rate, and that stakes caught 3× more errors than confidence did.
Failure modes
| Failure | Symptom | Mitigation |
|---|---|---|
| System 1 confidently wrong | Confidence high, answer wrong, no other signal fires | Add rule checks and novelty signals; never gate on confidence alone |
| Escalation creep | Thresholds tightened after each incident until most traffic escalates | Track escalation rate as a budget metric with an owner |
| Context loss on handoff | System 2 gets the question but not the tool results | Pass the full trace plus System 1's draft |
| Silent drift | A model update changes confidence calibration | Re-calibrate thresholds on every model version change |