Your eval says the model failed. It won't say why, or whether your fix worked. A score is a temperature reading; what you actually need is model health — where the model is failing, what mechanism drives it, how deep a repair has to cut, and whether it held. EvalVitals probes the model, finds the structure behind its failures, proposes a mechanism, tests it on cases the analysis never saw, then builds a repair and validates it against the unmodified baseline. A repair that fails escalates to a more invasive class of fix and tries again — so each pass either improves the model's health against a measured baseline, or returns an honest account of why it couldn't.
Five stages, and no hand-off between them. The agent writes and runs its own analysis code, selects the statistical tools, proposes the hypotheses, generates the repair candidates and works out which tier a given mechanism needs. Two stages can send the loop backwards — a refuted hypothesis returns to probing, and a repair that fails to beat the baseline escalates a rung instead of shipping.
You supply the question and the ceiling. Everything between is unattended.
L1 to L4.analysis.py as it ran, the normalised records, every figure and table, and the run's event log.Repairs are ordered by how deeply they cut into the model. Each rung buys causal reach and costs deployability. The ceiling is yours to set — escalation is recommended, never automatic.
The confirmed mode decides which repair is worth paying for. In the worked example — hallucinations show over-concentrated attention — that points at grounding, not more data. Ordered by cost:
Design space, not shipped code.
EvalVitals verifies a diagnosis before it becomes a repair. A senior researcher can list ten plausible mechanisms for a failure; an agent will list fifty. Fluent explanations are easy to produce, and the loop tells you which of them hold on your data.
| Approach | How the hypothesis is formed | How it is tested | What the conclusion rests on |
|---|---|---|---|
| Hire an experimentalist | Experience and intuition, a handful of candidates at a time | An ablation designed after seeing the data, weeks of senior time per model | The analyst — one researcher's reading of the data, and the ablation they chose to run |
| Let an agent enumerate | Dozens of plausible candidates at once | A full fine-tune for each one you can afford | The budget — whichever candidates you could afford to test |
| EvalVitals | Drawn from published repair results and the memory of past runs | Cross-validated while exploring, then decided once on a separate sealed set | The data — a measured effect on reserved cases, reproducible from the run log |
A run can end with a question rather than a repair — on the committed example, one signal of four survived adjudication, and every candidate below L4 finished level with the baseline. Learning that a mechanism remains open, before a cluster pays for it, is the cheapest result the loop returns.
The buyer is anyone who ships or depends on an open-weight model and cannot currently answer “why did it fail, and did the fix work?”
Find and repair the failure modes your benchmark score hides, and ship the evidence bundle alongside the model card.
Clinical, financial and safety-critical settings need a documented mechanism and a verified repair, not a moved metric.
The aggregate barely moved but a slice regressed. Locate the slice, identify the mechanism, and test whether reverting or retraining fixes it.
Compare candidates on the failure modes that matter to your product rather than on a public leaderboard.
Turn a customer-visible incident into a diagnosed mechanism and a repair that is measured against the model you were already running.
Re-test a reported mechanism and its intervention on held-out cases and across model scales, before adopting it.
You staff senior engineers to sit inside each customer's failures. Run the loop first and the engagement opens with a diagnosed mechanism instead of spending three weeks arriving at one.
The loop is a productised version of what we do as research: find the mechanism behind a model's failure, then intervene on it and check the intervention held.
CIFAR AI Chair and Canada Research Chair at UBC, IC at Google GenAI. Research spans trustworthy machine learning, optimization, large language models and AI agents.
Graduate student at UBC. Internships at AWS and Google on customer-facing projects, which is where the gap between a benchmark score and a shipped system becomes obvious.
Graduate student at UBC. Former Alibaba researcher, building Alibaba’s top-ranked open-source model Ovis and developing on Qwen.