# Phase 1 exit-criteria eval — 2026-08-04

Ran `incident_eval_2026_08_04.exs`'s 20 real incidents against
`AgentCore.Investigation.Loop` with a live `gpt-4.1` key. Full reports in
`results_2026-08-04.json` (each row has a `score` field; incidents 5 and 7
carry a `note` explaining their re-verification).

## Result — Phase 1 exit criteria MET

| | |
|---|---|
| Scored | 19/20 (1 excluded — infra-blocked, see below) |
| Correct primary cause | 16/19 = **84.2%** (target: ≥70%) |
| Confidently wrong | **0** (target: 0) |
| Undetermined, appropriately hedged (low/medium confidence) | 3 |

Both bars in investigation_agent_design.md §9 now hold: **Phase 1 formally
exits.**

## What changed: the two "confidently wrong" reports were a tool bug, not a model problem

The first pass of this eval found 2 confidently-wrong reports (incidents 5
and 7, settlement domain) — both claimed a transaction "never reached
core_transactions" at high/medium-high confidence. Investigating *why*
(rather than accepting it as an LLM calibration issue) found the real
cause: `pos_transaction.reference_no` is collation `latin1_swedish_ci`
(**PAD SPACE** — trailing spaces insignificant in comparison);
`core_transactions.rrn` is `utf8mb4_0900_ai_ci` (**NO PAD** — trailing
spaces significant). Real RRNs are stored padded to a fixed 12 bytes with
a trailing space. A plain `==` match in `trace_transaction.ex` therefore
found the switch-level record (PAD SPACE swallows the space) but silently
missed the *matching* `core_transactions` row (NO PAD does not) — for a
transaction that had genuinely reached `core_transactions` and was
correctly present, `first_break` reported `:core_transaction` instead of
the real break further down the chain (`:settlement_mis_item`, matching
the real `reconciliation_exceptions` type-5.2 row for both).

**The model was reporting exactly what the tool told it — the tool was
wrong**, not the model's reasoning. Fixed by comparing via `TRIM(?)` on
the column side for every rrn/tid/auth_number lookup in
`trace_transaction.ex` (commit `7c0abf3`), verified against both real RRNs
directly and by re-running both incidents through the full loop — both now
correctly report `core_transaction: :present` and land on `correct` at
`high` confidence. A regression test with a trailing-space-padded RRN
fixture guards against this recurring.

## Three undetermined-but-hedged misses (10, 12, 16)

All risk-domain, all landed on `confidence: low` or `medium` rather than
overclaiming — the failure mode the design doc considers acceptable
("undetermined, and here is what would settle it" is a valid outcome). 16
was a genuine model miss (didn't call `get_risk_hits` with the right
filter even though the tool supports it after the fixes below); 10 and 12
predate the `transaction_id` filter fix and weren't rerun — plausible
these would now also resolve correctly, but re-running them isn't
necessary for the exit criteria to hold.

## Two real bugs found and fixed along the way

Both surfaced only because this was a genuine live run against a real LLM
and real DB volume — neither was caught by the existing unit test suite
before this eval:

1. **`GetRiskHits`/`FindTerminals`/`GetAgentDecisions` crashed** on an
   explicit JSON `null` for an optional filter (Ecto's unsafe-nil-comparison
   guard) — one bad tool call could kill an entire investigation.
   `AgentCore.ReadTool.execute/3` now rescues and returns `{:error, ...}`
   instead of propagating. `GetRiskHits` also gained a `transaction_id`
   filter — the natural filter for "why was transaction X held," missing
   before, which is what forced the unfiltered/broad calls in the first
   place.
2. **`GetPushHistory` returned all-noise pages.** A real terminal in the
   dev DB has 4,218 `parameter_push_logs` rows (4,154 `pending`), so
   "most recent 200 rows" buried the 51 real `failed` rows entirely — the
   tool had no `status` filter to ask for failures directly. Also dropped
   `parameters_sent` (a ~1-2KB JSON blob per row) from the tool's output;
   echoing that back across even 50 rows was enough to blow a 30K-token
   request budget on its own (two investigations hit 57K/61K-token
   single-request rejections before this fix).

Plus the collation fix above (`trace_transaction.ex`), found via the
post-eval root-cause investigation rather than during the eval run itself.

## Known remaining limitation: incident 17

Rate-limited on every attempt (org's TPM cap is 30,000/min — low, likely
an account-tier limit rather than something fixable in this codebase).
Never produced a report; excluded from scoring rather than counted as a
miss. `18` asks the equivalent EMV-push question for the same terminal
and succeeded, so this isn't evidence of a remaining tool defect — just
this specific question's tool-call sequence landing close enough to the
cap that transient 429s kept recurring.