Cases
606
3 checkpoints
Your eval says the model failed. It won't say why, or whether your fix worked. EvalVitals probes the model, finds the structure behind its failures, proposes a mechanism, tests it on cases the analysis never saw, then builds a repair and validates it against the unmodified baseline. A repair that fails escalates to a more invasive class of fix and tries again — so each pass either improves the model against a measured baseline, or returns an honest account of why it couldn't.
Five stages, and no hand-off between them. The agent writes and runs its own analysis code, selects the statistical tools, proposes the hypotheses, generates the repair candidates and works out which tier a given mechanism needs. Two stages can send the loop backwards — a refuted hypothesis returns to probing, and a repair that fails to beat the baseline escalates a rung instead of shipping.
You supply the question and the ceiling. Everything between is unattended.
L1 to L4.analysis.py as it ran, the normalised records, every figure and table, and the run's event log.One investigation, run end to end and committed to the repository: three Qwen3-VL checkpoints asked whether an object is present in a COCO image. Every number below comes from that run's report file — it is a single example, not an aggregate or a benchmark claim.
In this run, attention focus share separated hallucinations from correct rejections at AUC 0.82. Of four candidate signals, one survived adjudication. The other three did not, and the report says so.
Repairs are ordered by how deeply they cut into the model. Each rung buys causal reach and costs deployability. The ceiling is yours to set — escalation is recommended, never automatic.
The confirmed mode decides which repair is worth paying for. In the worked example — hallucinations show over-concentrated attention — that points at grounding, not more data. Ordered by cost:
Design space, not shipped code.
Repairing a model is an experimental problem, so the loop runs an experimental protocol. Each pass states a falsifiable claim about the mechanism, predicts what would follow if it were true, and tests that prediction on data reserved before the analysis began.
Guessing is expensive because most guesses are wrong and you pay full training cost to find out. Testing a hypothesis on held-out cases costs a fraction of a fine-tune — so the statistics are what keep the GPU bill down. By the time a repair reaches parameter space, the mechanism has already survived a test it could have failed.
| Approach | How a fix is judged | What survives the session |
|---|---|---|
| Read the failure table | Spot-check on the same cases that prompted it | nothing |
| Ask a model to explain | Plausibility of the narrative | A chat transcript |
| Iterate on the benchmark | The same benchmark that surfaced the problem | A score history |
| EvalVitals | Held-out cases, e-BH corrected, paired against the unmodified baseline | Every hypothesis, verdict and line of generated code |
Every hypothesis, generated tool, verdict and fix outcome is written to a structured event log against a published JSON Schema, alongside the analysis code that actually ran. A reviewer can re-run it and point at the line that produced a claim.
The guard that makes this work is the order of operations — the split is taken before exploration, so nothing found during analysis was tuned on the rows that later judge it:
A surviving mechanism then routes itself. The loop maps the confirmed hypothesis to the cheapest tier that could causally address it, works upward when candidates fail, and hands back the next move — so the decision about what to try next is made from evidence rather than by someone's hunch:
Raising the ceiling stays a human call — the one deliberate stop in an otherwise unattended loop, because that is the step that spends a cluster.
A run may end inconclusive — on one committed example, zero of four signals cleared adjudication. Knowing a mechanism is unvalidated, before spending a cluster on it, is the cheapest result the loop can return.
The buyer is anyone who ships or depends on an open-weight model and cannot currently answer “why did it fail, and did the fix work?”
Find and repair the failure modes your benchmark score hides, and ship the evidence bundle alongside the model card.
Clinical, financial and safety-critical settings need a documented mechanism and a verified repair, not a moved metric.
The aggregate barely moved but a slice regressed. Locate the slice, identify the mechanism, and test whether reverting or retraining fixes it.
Compare candidates on the failure modes that matter to your product rather than on a public leaderboard.
Turn a customer-visible incident into a diagnosed mechanism and a repair that is measured against the model you were already running.
Re-test a reported mechanism and its intervention on held-out cases and across model scales, before adopting it.
The loop is a productised version of what we do as research: find the mechanism behind a model's failure, then intervene on it and check the intervention held.
CIFAR AI Chair and Canada Research Chair at UBC, IC at Google GenAI. Research spans trustworthy machine learning, optimization, large language models and AI agents.
Graduate student at UBC. Internships at AWS and Google on customer-facing projects, which is where the gap between a benchmark score and a shipped system becomes obvious.
Graduate student at UBC. Former Alibaba researcher, building Alibaba’s top-ranked open-source model Ovis and developing on Qwen.