Part 14 · Evaluation — how you know this works

Level: ⭐⭐⭐⭐ · Time: ~5 h · Requires: parts 3 (harness), 13 (fleet) Produces: a golden set, trajectory evaluation, a judge with external verification, a regression gate

---

0. A number that should change your priorities

An agent's score changes more with the harness than with the model. The same model under two scaffolds produces significantly different numbers. The harness decides how tools are exposed, how errors are shown and how many attempts the agent gets. The prompt and the planning loop can add or subtract points before the model even starts working. — Prefactor, Agent Benchmarks

Two consequences, both practical:

1. Other people's model benchmarks tell you nothing about how the model will behave in your system. They were measured under a different harness.
2. You have to measure your own system, because otherwise you do not know whether the change you just made helped or hurt.

⭐ This is the part without which all the previous ones are guesswork. Every "threshold", "limit" and "variant" from parts 4–13 rests on a measurement. Here you build the machine that performs it.

---

1. Four levels of evaluation

Most people measure only level 1 and are surprised that the agent is unreliable.

⭐ Level 3 is the one that separates a demo from a system. An agent that passes a task in 7 out of 10 runs does not work 70% of the time — it does not work, because you do not know which run you will get.

1.1 · pass^k — a measure worth knowing

pass@k  = did it succeed at least once in k attempts     ← a measure of CAPABILITY
pass^k  = did it succeed in ALL k attempts               ← ⭐ a measure of RELIABILITY

An agent with an 80% success rate per step has pass^5 = 0.8⁵ = 33%. That means that in two thirds of cases a series of five tasks will fail at least once.

⭐ Report pass^k, not pass@k. pass@k is a number for presentations. pass^k is the number you make decisions on.

---

2. The golden set

2.1 · Construction

A hundred questions is a good target. Fifty is enough to start. Twenty is too few to detect a 5-point change.

{
  "id": "Z041",
  "kategoria": "wyszukanie_z_cytatem",
  "pytanie": "What is the payment due date of invoice FV/017/03/2026?",
  "oczekiwane": {
    "zawiera": ["2026-04-14"],
    "nie_zawiera": ["I don't know", "I did not find"],
    "wymaga_cytatu": true,
    "zrodlo": "faktury/2026/03/f017.txt"
  },
  "trajektoria": {
    "max_krokow": 4,
    "wymagane_narzedzia": ["znajdz_dokumenty"],
    "zakazane_narzedzia": ["powloka"]
  },
  "przebiegow": 3
}

2.2 · Five categories — and why these in particular

⭐ Twenty percent for correct refusal is not excessive. An agent that always answers something is worse than no agent, because it sounds credible. Without this category in the set you will be optimizing toward self-confidence, not truthfulness.

2.3 · Where to get the questions

# Pull candidates from your own logs — the best source you have
python3 testy/z_dziennika.py --ostatnie 200 --tylko-udane > kandydaci.jsonl

⚠️ Never put into the golden set an answer generated by the same model you are testing without checking it by hand. You will build a benchmark that measures repeatability, not correctness.

---

3. Trajectory evaluation

A correct result, an absurd path — this is a real state of affairs and you need to see it.

# $OSA/ewaluacja/trajektoria.py
def ocen_trajektorie(zdarzenia: list, oczekiwana: dict) -> dict:
    krokow = sum(1 for z in zdarzenia if z["typ"] == "narzedzie.wywolanie")
    uzyte  = [z["narzedzie"] for z in zdarzenia if z["typ"] == "narzedzie.wywolanie"]
    bledy  = sum(1 for z in zdarzenia if z["typ"] == "narzedzie.wynik" and z.get("kod"))

    # repeats of identical calls = going in circles
    podpisy = [f'{z["narzedzie"]}:{json.dumps(z["argumenty"], sort_keys=True)}'
               for z in zdarzenia if z["typ"] == "narzedzie.wywolanie"]
    powtorzen = len(podpisy) - len(set(podpisy))

    uwagi = []
    if krokow > oczekiwana["max_krokow"]:
        uwagi.append(f"too many steps: {krokow} > {oczekiwana['max_krokow']}")
    if powtorzen:
        uwagi.append(f"{powtorzen} repeated calls (going in circles)")
    for n in oczekiwana.get("zakazane_narzedzia", []):
        if n in uzyte:
            uwagi.append(f"⚠️ forbidden '{n}' was used")
    for n in oczekiwana.get("wymagane_narzedzia", []):
        if n not in uzyte:
            uwagi.append(f"required '{n}' was not used")

    return {"krokow": krokow, "bledow": bledy, "powtorzen": powtorzen,
            "narzedzia": uzyte, "uwagi": uwagi, "czysta": not uwagi}

⭐ powtorzen is the cheapest and most effective pathology detector. An agent calling the same thing twice with the same arguments is definitely not making progress. It is the same signal as the state fingerprint from part 3, only cheaper to compute.

---

4. The judge — and why it cannot be the same agent

4.1 · The self-verification problem

When sampling trajectories, self-reflection is prone to self-verification bias: the agent evaluates its own intermediate results from the same trajectory state that produced them. This coupling interweaves verification with self-justification and leads to accepting flawed results. — arXiv 2603.02798

Translated into practice: an agent that got it wrong will not notice, because it looks at the problem with the same flawed set of assumptions. Research from ICLR 2024 says the same: models cannot reliably correct their own reasoning without a signal from outside.

4.2 · Four sources of verification, from the strongest

⭐ The order is the whole lesson here. Before you reach for a model judge, check whether you can verify by execution or by pattern. More than half the questions in a well-built golden set can be — and these two sources are free and reliable.

4.3 · A judge in an isolated context

PROMPT_SEDZIEGO = """Assess whether the ANSWER correctly answers the QUESTION
based on the SOURCE.

You have no access to anything beyond what is below. Do not assume context.

Criteria (each separately):
1. CONSISTENCY — do the answer's claims follow from the source? (yes/no/partially)
2. COMPLETENESS — does it answer the whole question? (yes/no/partially)
3. QUOTE — does the quote given appear VERBATIM in the source? (yes/no/no quote)
4. FABRICATED — does it contain claims that are NOT in the source? (yes/no)

Return JSON: {{"zgodnosc":…,"kompletnosc":…,"cytat":…,"wymyslone":…,"uzasadnienie":"…"}}

QUESTION: {pytanie}
SOURCE: {zrodlo}
ANSWER: {odpowiedz}"""

def sedzia(pytanie, zrodlo, odpowiedz, wywolaj_model_sedziego):
    """⭐ The judge DOES NOT SEE: the trajectory, the agent's prompt, the history. Only these three things."""
    return json.loads(wywolaj_model_sedziego(
        [{"rola": "user", "tresc": PROMPT_SEDZIEGO.format(
            pytanie=pytanie, zrodlo=zrodlo, odpowiedz=odpowiedz)}],
        gramatyka=GRAMATYKA_WERDYKTU)["tresc"])

Three rules for the judge:

1. An isolated context. It does not see how the agent arrived at the answer — otherwise it can be persuaded by the reasoning instead of the result.
2. Criteria separately. "Is this a good answer" yields noise. Four binary questions yield signal.
3. A different model than the one being evaluated, if you can. qwen3-32b evaluates gpt-oss-20b.

4.4 · Calibrating the judge

⚠️ The judge is a model too and it also makes mistakes. Before you trust it — measure it.

# 40 cases that you graded by hand → compare with the judge
python3 ewaluacja/kalibruj_sedziego.py --recznie recznie.jsonl
# agreement: 34/40 (85%)
# ⚠️ the judge MORE LENIENT in 5 of 6 disagreements
#    → on refusals: the judge accepts "I did not find it" when the answer WAS in the source

⭐ The direction of the judge's error matters more than its size. A lenient judge inflates the score and lulls you to sleep. A strict judge deflates it and at most you waste time on a problem that does not exist. If you have to choose — calibrate toward strictness.

---

5. The regression gate

⭐ This is the most important practical product of this part. No change goes in without a measurement.

# $OSA/ewaluacja/bramka.py
PROGI = {
    "spadek_ogolny_pp":     2.0,    # a drop > 2 pp = rejection
    "spadek_kategorii_pp":  5.0,    # no category drops > 5 pp
    "min_odmowy":          90.0,    # ⭐ refusals must NEVER fall below this
    "wzrost_krokow":        1.3,    # the average number of steps does not grow by >30%
    "wzrost_tokenow":       1.5,
}

def bramka(nowy: dict, bazowy: dict) -> tuple[bool, list]:
    powody = []
    d = nowy["ogolem"] - bazowy["ogolem"]
    if d < -PROGI["spadek_ogolny_pp"]:
        powody.append(f"overall score {bazowy['ogolem']:.1f} → {nowy['ogolem']:.1f} ({d:+.1f} pp)")
    for k in bazowy["kategorie"]:
        dk = nowy["kategorie"][k] - bazowy["kategorie"][k]
        if dk < -PROGI["spadek_kategorii_pp"]:
            powody.append(f"category '{k}': {dk:+.1f} pp")
    if nowy["kategorie"].get("poprawna_odmowa", 100) < PROGI["min_odmowy"]:
        powody.append(f"⚠️ refusals {nowy['kategorie']['poprawna_odmowa']:.1f}% "
                      f"< {PROGI['min_odmowy']}% — the agent started guessing")
    if nowy["sr_krokow"] > bazowy["sr_krokow"] * PROGI["wzrost_krokow"]:
        powody.append(f"steps {bazowy['sr_krokow']:.1f} → {nowy['sr_krokow']:.1f}")
    return not powody, powody

Hook it in at three places:

# 1. before every commit that changes the prompt, the tools or the harness
python3 ewaluacja/bramka.py --przed-commitem

# 2. in the night task (part 11) — detects drift without your involvement
# 3. on every change of model or its version

---

6. What to learn from the big benchmarks

Do not run them. Steal ideas from them.

⭐ τ²-bench is the most interesting one for you. A classic golden set assumes the world stands still. Your agent works in a world in which you are adding files and changing the calendar at the same time. Add 3–5 such cases — they will be the hardest and the most instructive.

---

7. EXPERIMENT 14 — the same model, two harnesses

⭐ This experiment verifies the thesis on which the whole course stands.

Hypothesis: the difference between the harness from part 3 and the naive loop from course 1 is greater than the difference between gpt-oss-20b and qwen3-32b under the same harness.

7.1 · Four combinations

mkdir -p $OSA/pomiary/ewaluacja
for h in naiwny pelny; do for m in oss20 qwen32; do
  python3 ewaluacja/uruchom.py --harness $h --model $m --przebiegow 3 \
    > pomiary/ewaluacja/$h-$m.json
done; done

⚠️ Prediction: the harness difference (vertical) will be greater than the model difference (horizontal). If it turns out so — you have the course's thesis confirmed on your own hardware and a concrete argument for not buying bigger models as long as the harness is weak.

7.2 · Breakdown by category and pass^3

⭐ I expect the biggest difference to be in "multi-step" and in pass^3 — because those are exactly the two things the harness fixes: finishing a long task and repeatability.

7.3 · What to do with the result

python3 ewaluacja/uruchom.py --zapisz-baze    # this is your reference point

From this moment every change in the system is measured against this baseline. Without a baseline, all the following parts are guesswork.

---

8. IN PRODUCTION

💡 When it is worth adding off-the-shelf tools. When you want RAG metrics (faithfulness, context precision) — Ragas has them polished and there is no point writing them from scratch. You log the result onto your own traces from part 15.

---

9. EXERCISE

1. Build a golden set: 50 questions, five categories in the given proportions. Pull at least 20 from your own logs from part 2.
2. Implement trajectory evaluation with repeat detection.
3. Implement a judge in an isolated context on a different model.
4. Calibrate the judge on 40 cases graded by hand. Record the direction of the error.
5. Run the experiment from §7 and save the baseline.
6. Hook the gate into the commit and the night task.

python3 ewaluacja/uruchom.py --raport
# ═══ GOLDEN SET · archiwista · gpt-oss-20b · full harness ═══
#   wyszukanie_z_cytatem   14/15   93%   (baseline 93%)   →
#   synteza                10/13   77%   (baseline 69%)   ↑ +8
#   poprawna_odmowa         9/10   90%   (baseline 90%)   →
#   wieloetapowe            6/ 8   75%   (baseline 50%)   ↑ +25
#   odpornosc               3/ 4   75%   (baseline 75%)   →
#   ─────────────────────────────────────────────────
#   OVERALL                42/50   84%   (baseline 78%)   ↑ +6
#   pass^3                 31/50   62%   (baseline 48%)   ↑ +14
#   avg steps 4.2 (baseline 6.8)   avg tokens 3,180
#   clean trajectories: 38/50 · repeats in 7 · forbidden tools in 0
#   ✓ GATE: passes

---

10. Pitfalls of this part

---

11. Checkpoint

cd $OSA
python3 - <<'EOF'
import json, pathlib, collections
z = [json.loads(l) for l in open("testy/zbior-zloty.jsonl", encoding="utf-8")]
assert len(z) >= 50, f"too few questions: {len(z)}"
k = collections.Counter(x["kategoria"] for x in z)
assert k["poprawna_odmowa"] / len(z) >= 0.18, "P119: too few refusals"
assert all(x.get("przebiegow", 1) >= 3 for x in z), "P120: <3 runs"
print(f"✓ golden set: {len(z)} questions, {dict(k)}")
EOF
test -s pomiary/ewaluacja/baza.json && echo "✓ baseline saved"
test -s ewaluacja/kalibracja-sedziego.json && echo "✓ judge calibrated"
python3 ewaluacja/bramka.py --przed-commitem && echo "✓ gate passes"
git add -A && git commit -m "part 14: golden set, trajectory, judge, regression gate"

• You measure four levels, not only the result
• You report pass^k, not pass@k
• A golden set of ≥50 questions, 20% of them correct refusals
• At least 20 questions come from real logs
• The trajectory is evaluated; repeats are detected
• External verification: execution → pattern → judge in isolation
• The judge is calibrated, you know the direction of its error
• The regression gate is hooked into the commit and the night task
• ⭐ You measured: harness versus model. You have a reference baseline

---

Next: 15-OBSERVABILITY.md — seeing what the agent does, and being able to answer the question "why did it do that" from the traces alone.