Skip to content

About this practice

Status
draft
Endorsement
none yet
Last reviewed
2026-07-27

Best practice 1

Match the method to the task; do not default to an agent or a frontier LLM

Practice

Before reaching for an agent, ask whether a script, a fixed workflow, or an existing specialized model already does the job as well or better. Keep the frontier model for the open-ended, judgement-heavy steps where its flexibility is the point. A method that fits the task is usually cheapest to run and easiest to reproduce.

Expose deterministic operations and specialized models as first-class tools. Incentivise the use of classical workflows where they are well established and functional. Do not assume every capability should be reached through a general model; the same demand-and-importance logic that governs where to invest in agent interfaces (BP02) applies to method choice.

Fund method choice and evaluation, not model spend. "Total token spend" is not a useful productivity metric. The goal is the cheapest reliable method that meets the need; defaulting to frontier models raises cost and reproducibility risk without a matching gain. Support the evaluation that tells adopters when a simpler method suffices.

Reasons

An agent is a general instrument, and general instruments are rarely the most efficient way to do a fixed job. Vendor engineering guidance is explicit: start with the simplest approach, and add agentic complexity only when a task needs flexible, model-driven decisions at scale. Agents trade latency and cost for that flexibility. When real cost is measured, complex agents often fail to beat simple baselines. On well-specified tasks, a specialized or fine-tuned model can be more accurate than a larger general model, at a fraction of the cost. The reverse choice has a record of concrete harm in science: a frontier model asked to supply references fabricates a large share of them, and treating a model as a scientific instrument without method rigour produces over-optimistic results that do not reproduce. You cannot pick the right method without first understanding the task. Understanding the task includes understanding the net benefit expected from it. Controlled trials show forecast gains from agentic tools are frequently miscalibrated and vary by task and user, so state the expected benefit against a baseline with human oversight cost deducted, then measure the realised outcome against it.

Examples

  • A team needs to extract one field from a fixed report format every week; a short parser does it deterministically, and the agent that was drafted for the job would have cost more and varied run to run.
  • A group classifying thousands of records fine-tunes a small model on labelled examples; it beats a zero-shot frontier model on their data and runs on their own hardware.
  • A literature step genuinely needs open-ended judgement across messy sources, so an agent is the right tool; the routine lookups it depends on are exposed as deterministic tools it calls rather than reasons about.
  • A scientist asks a chat model for supporting references and pastes them into a manuscript; several do not exist, a failure that a bibliographic search against a real index would not have produced.
  • A scientist uses a literature agent that is validated against a real index to autonomously find evidence for their opinion piece. The cited papers really exist, but on review, it becomes clear that the agent only read the abstracts of the papers, citing works for sentences they do not really support. The opinion piece is rejected.
  • A pipeline reports strong accuracy that collapses on re-run because the model was applied where the evaluation leaked; a simpler method with proper train/test separation would have been both honest and reproducible.
  • A group expects a large speedup from an agent and, before committing, writes down the expected net benefit against its current baseline with review and correction time subtracted; for its experienced users the projected gain is marginal, so it keeps the existing workflow and re-checks the estimate on a small pilot.

Sources

Change history

  • 2026-07-27: Added the Expectation–Realisation Gap review (Lobentanzer 2026) as downweighted context on heterogeneous, miscalibrated benefit, with a Reasons and Examples note that method choice includes stating and re-measuring expected net benefit with oversight cost deducted (bp1-a1, bp1-a3).
  • 2026-07-27: Added The GenAI Divide (MIT NANDA 2025) as a supporting source on task-based method fit (bp1-a1).
  • 2026-07-27: Created as BP01. Method-selection practice at the general end of the record; grounds the disciplined stance that an agent or frontier model is chosen, not assumed. Existing BP01 through BP09 renumbered to BP02 through BP10.

Discussion