Evaluator–Optimizer

Also known as: generator–critic loop, actor–judge, iterative refinement

One agent generates, another grades against a rubric, and they loop until it passes.

TrackAgentic systems FamilyMulti-agent Maturity Established ChangesVerificationCoordination

Problem

A single generation pass often misses requirements that are easy to check but hard to satisfy on the first try: tone, coverage, format, factual grounding. Self-reflection by the same model tends to approve its own work.

Quality bars are explicit enough to write as a rubric.
Generating is harder than judging, so a separate judge adds real signal.
Every loop iteration adds latency and cost.
Loops can oscillate or never converge without a stopping rule.

Solution

Split the work into two roles. A generator produces a draft. An evaluator scores it against a written rubric and returns structured feedback for each failing criterion. The generator revises using that feedback. Stop when all gates pass or after a maximum number of iterations, then escalate whatever is still failing.

Diagram

Feedback is structured per rubric item, not a free-text opinion. The iteration cap prevents endless loops.

When to use

Use it when

Quality criteria can be written down as a rubric
A first draft is usually close but not quite right
Outputs go to customers or executives, where polish matters
You want an audit trail of why an output passed

Avoid it when

There is no clear rubric; the judge will just add noise
Latency budgets allow only one pass
The judge and generator share the same blind spots (same model, same prompt)

Trade-offs

Latency

Each iteration adds a full generate + judge cycle

Cost

Typically 2–3× a single pass

Complexity

Two prompts and a loop

Reliability

Strong when the rubric is concrete

Gains

Measurable quality lift on rubric criteria
Per-criterion scores double as eval data
Clear point to insert a human for edge cases

Costs

Rubric writing and maintenance
Judge can be gamed by surface fixes
Higher latency and token spend

Worked example

Scenario
An agent writes customer-call talk scripts for account managers. The rubric checks personalization, pain-point relevance, speakability and invented facts.

GeneratorDrafts a 6-line opening script for a renewal call with Acme.
EvaluatorScores: personalization 4/5, speakability 5/5, pain-point relevance 2/5. The invented-facts check flags line 3, a claimed product integration not in the account data. verdict: revise
GeneratorRewrites line 3 from real account notes and leads with the support-backlog pain point.
EvaluatorAll gates pass. Pain-point relevance is now 4/5. verdict: pass (iteration 2)
LogStores both drafts and scores. Over a month, the most common revision reason becomes the top prompt improvement for the generator.

Failure modes

Failure	Symptom	Mitigation
Rubber-stamp judge	Judge approves nearly everything	Use a different model or prompt for the judge and calibrate on labeled examples
Oscillation	Fixing one criterion breaks another	Feed back all failing items at once and keep passing items locked
Surface gaming	Generator learns phrases the judge likes	Rotate judge examples; spot-check with humans
Unbounded loops	No convergence on hard cases	Cap at 2–3 iterations, then escalate