Skip to content

About this practice

Status
draft
Endorsement
none yet
Last reviewed
2026-09-16

Best practice 8

Evaluate / benchmark agentic tools before trusting their outputs

Practice

A tool being available, popular, or well-described is not evidence that it is correct for your task. Before you rely on its output, run it on cases where you know the answer, check it across repeated runs, and note where it breaks. Do not let a model from the same family grade the results for you; where a model does act as judge, check a sample of its verdicts yourself, against reference answers and criteria fixed before you see the results. Keep that evidence; it is what justifies trusting the result.

Publish evaluation evidence for what you ship: how the tool was tested, on what tasks, with what error rate and known failure modes. Say who or what judged the outputs, and how much of that judgement was human. Make it easy for adopters to reproduce your evaluation. Benchmark scores on general tasks do not tell a user whether the tool is reliable for theirs.

Require evaluation before reliance for tools that feed into results or decisions, proportionate to the stakes. Fund shared benchmarks and evaluation practice so adopters are not each testing blind. Support and fund independent evaluation efforts and community-driven challenges on held-out data.

Reasons

Whether a tool is safe to connect and whether it is correct are different questions, answered by different people. Registration and vetting (BP03) concern security at connect time; provenance and auditability (BP07) concern the ability to trace answers after execution. Evaluation (this practice) concerns accuracy and reliability at use time. Human oversight (BP09) decides where people check or intervene in a workflow, during or after a run; evaluation decides whether the tool is fit for the task at all, before reliance and again when the tool or the task changes. Goodhart's law states that a metric that becomes the target ceases to be a good metric; LLM leaderboards today decide on highest-volume investments for frontier AI companies. Evaluation is the task of users, adopters, and community; only they know the task and possess the necessary stakes and incentives to desire an objective evaluation. The evidence is clear that this cannot be skipped: a single benchmark score does not predict deployment reliability, agents vary run to run, and confabulationConfabulationA confident, fluent output from a language model that is not grounded in fact, also called hallucination. Because it reads like a correct answer, it is caught by checking against sources, not by judging how plausible it looks. is common. A tool that looks capable in a demo can be wrong in ways that only representative testing reveals. The measuring instrument matters too. A language model asked to judge outputs favours its own generations; this self-preference has been measured. Models from the same family share training data and failure modes, so this practice also treats a same-family judge as not independent. That step is a precaution: the study tested models judging their own outputs, not their relatives'. The failures log records an agent granting its own top verification status. Some human judgement, in proportion to the stakes, belongs in the evidence. It makes the evaluation independent only when it uses reference answers and criteria fixed before the results are seen. A person reviewing outputs without either tends to accept plausible output (automation bias, see BP09).

Examples

  • A team adopts a tool because it tops a public leaderboard, then finds it wrong on their own data; the leaderboard measured a different task from theirs.
  • Before relying on a tool, the adopter runs it on cases where the answer is known, checks it across repeated runs, and records where it breaks; that evidence is what justifies trusting the result.
  • A tool that looked convincing in a demo is wired into a study without testing, and a confabulated value reaches an analysis before anyone checks it.
  • A provider ships evaluation evidence with the tool (tasks, metrics, error rate, known failure modes) that an adopter can reproduce, not only a feature description.
  • Red-teaming and hallucination/confabulation testing are run as the safety-and-correctness side of the same evaluation, not a separate afterthought.
  • Evaluations follow an established checklist (for example REFORMS for ML-based science, or a reproducibility checklist), so different groups' results are comparable.
  • A team scores its literature agent with a judge model from the same vendor family and reports a high pass rate; a sample re-checked by hand shows the judge waved through the same misreadings the agent made, so the score measured agreement, not correctness.

Sources

  • ELIXIR TF Agentic AI: agenda and rolling best practice (2026). Cautions and gaps: how to assess whether a skill or MCP server is useful and safe is recorded there as an open question and a candidate future practice.
  • REFORMS reporting standards (2024). Consensus reporting standards for machine-learning-based science.
  • NeurIPS Paper Checklist. Reproducibility and evaluation reporting practice.
  • The GenAI Divide (2025). Enterprise field evidence that buyers who "benchmark tools on operational outcomes, not model benchmarks" succeed far more often; general benchmark scores do not predict fitness for a specific task (§6.2, pg. 20). Out-of-domain business report; cited as external context.
  • EU AI Omnibus (2026). Art 4a. The amended AI Act sets a conditional legal basis for processing special-category data to detect and correct bias, an evaluation dimension, subject to strict safeguards (bp8-a2).
  • LLM Evaluators Recognize and Favor Their Own Generations (Panickssery, Bowman & Feng 2024). Measured self-preference in LLM evaluators: a model scores its own outputs higher than human annotators do, and the bias grows with its ability to recognise its own text. A judge from the same family as the system under test is not independent evidence (bp8-a6).

Change history

  • 2026-09-16: Challenge #32. bp8-a5 now measures against an independent reference; new bp8-a6 says a same-family model judge is not independent and that some human judgement is part of the evidence, in proportion to the stakes, grounded in Panickssery, Bowman & Feng (2024) and the single-witness entry in the failures log. bp8-a4 and Reasons add the boundary with BP09 by purpose, and new bp8-a7 states when to benchmark: BP08 decides whether a tool is fit for the task, before reliance and again when the tool or task changes; BP09 decides where people check or intervene. Reasons separate the measured finding (a model favours its own outputs) from the precaution drawn from it (same-family judges), and say a human check is independent only against reference answers and criteria fixed in advance. Tabs and Examples extended to match.
  • 2026-07-27: Added the EU AI Omnibus (2026) as a qualifying source on bp8-a2 (conditional legal basis for special-category data in bias detection; Art 4a).
  • 2026-07-27: Added The GenAI Divide (MIT NANDA 2025) as a supporting source on outcome-based evaluation over benchmark scores (bp8-a3).
  • 2026-07-27: Renumbered from BP07 to BP08 on inserting the new BP01 (match the method to the task).
  • 2026-07-27: Rewrote Examples as concrete scenarios (actor, action, outcome), including anti-patterns; kept the labelled instance (REFORMS).
  • 2026-07-26: Created. New practice for adopter-side evaluation of agentic tools, distinct from vetting at registration (BP03) and from self-audit of one's own agent (BP04). The ELIXIR task force flagged evaluation as an unresolved open question; red-teaming and hallucination-rate testing fold in as its safety and correctness dimensions.

Discussion