You Scored the Model. You Never Scored the Work.

A VP of operations at a fifty-person services firm rolled her team over to the highest-scoring model on the benchmark last quarter.

Opus 4.8. Top of the leaderboard the week the news dropped. She read the score, watched two thought-leaders sing it on LinkedIn, called a meeting on Monday, and switched the team’s default agent to the new model across the board. Intake triage. COI generation. Renewals chase. Quote drafts. All of it. The whole menu. “If it’s the best, it’s the best,” she told me. “Why would we use anything else for client work?”

Six weeks later the model bill was four times what it had been on the previous default. Turnaround time on intake triage was identical. The renewals chase was, if anything, a hair slower because the team had stopped trusting the longer, more cautious output and was now editing every draft. One of her account managers had quietly switched back to the cheap model for her own folder. “It just gets through the queue,” she said, when I asked.

The benchmark had scored the model. Nobody had scored the work.

The decision that never got made

Here’s the thing. The leaderboard is a measurement. Like all measurements, it has a job, and the job is narrow. It scores a model against a fixed bank of problems. Coding tasks, math reasoning, long-context retrieval, agentic tool use. The leaderboard tells you which model is sharpest at that set of problems on that day.

It does not tell you which model belongs on your line, cooking your dishes, in your kitchen, for your operators.

That second question is the only one that actually matters in a working business. And the answer almost never comes out of a benchmark. It comes out of the work.

A renewals chase is not a coding problem. It is a routine pattern with a low cost-of-being-wrong, a high volume, and a tolerance for some looseness because the human operator is going to see every reply before it sends. The work has a profile. The work has a posture. The work has a budget. None of that lives on a leaderboard.

When you skip that profile and pick the model off the benchmark, you are scoring the wrong thing. The number on the leaderboard says the model is sharper. Right? So why would you pick anything else. The answer is that sharper is one of many properties, and your work might need cheaper, or faster, or more predictable, or less verbose, or more willing to refuse uncertain answers instead. Sharper is not a free upgrade. Sharper costs money. Sharper costs latency. Sharper sometimes costs the small-but-real virtue of saying “I am not sure” in a tone the team trusts.

You scored the model. You never scored the work.

Where this lives in the kitchen

This is a Four D’s problem, and specifically it is a problem at the DECIDING stance. The Four D’s describe how the Chef stands relative to a given dish. DOING is hands on the line. DISPATCHING is writing the recipe. DIALING is refining the recipe at the pass while the station cooks. DECIDING is the one most operators skip when they upgrade a model. DECIDING is the stance where you step back, evaluate performance, and decide if the dish stays on the menu and which station gets to cook it.

A model upgrade is a DECIDING-stance move. It is not a DOING move. The Chef is not in the pot. The Chef is at the table, with a clipboard, deciding which dish gets routed to which station, with which budget, with which review burden.

Most operators who roll out a new model are confusing DECIDING with DOING. They are treating the upgrade as if the question is “which model should I use to write this email.” That is a DOING-stance question. Which tool do I pick up right now. DECIDING asks a different question. Which dishes on the menu are worth running through which station, given what each station costs and what each dish is worth. That is the question the leaderboard cannot answer for you. The leaderboard does not know your menu.

This is why DECIDING is the hardest stance to occupy. It does not look like work. There is no output. The Chef sits, looks at the menu, and crosses dishes out or routes them differently. The work is not picking up the spatula. The work is the decision that the spatula was not the right tool. And operators feel weird about getting paid to not pick up the spatula, so they default to picking it up, and the menu never gets cleaned.

The five questions that score a dish

Here’s the thing. There is a tool for this. It looks a lot like a Prep List, only narrower, applied to model selection instead of to candidate systems. You run each workflow through it before you decide which model gets the job.

1. What is the cost of being wrong by one increment? A misdated renewal letter is a low-cost miss. A misquoted policy premium is a high-cost miss. The cost-of-being-wrong sets the floor for how much accuracy you have to buy.

2. What is the volume? High-volume work amortizes the cost of a smart model badly. You are buying the leaderboard one query at a time, and the bill compounds in a hurry. Low-volume work tolerates the spike. The same query at twenty a day costs nothing. The same query at twenty thousand a day costs four engineers’ salaries.

3. Can the output be defined cleanly enough to grade? If the work has a clear good-looks-like spec, a cheaper model can usually hit it with enough prompting. If the work is a fuzzy use-your-judgment call, the sharper model earns its keep. The clearer the spec, the cheaper the model can be.

4. What is the review burden the human is willing to absorb? If every output is going to be edited anyway, the sharper model is paying for polish nobody is asking for. If the output is going to fire automatically, the sharper model is paying for caution, which might be exactly what you want.

5. Will the model say “I don’t know” when it should? This one almost nobody scores. Two models can have the same benchmark score and behave completely differently when uncertain. One will hallucinate a confident answer. The other will refuse the call or hand it back. For agentic work that escalates to a human on uncertainty, the second model is literally worth more, regardless of where it sits on the leaderboard.

Run those five on a workflow. The model you should be using becomes embarrassingly obvious. Half the time, it is not the one on top of the leaderboard. The leaderboard scored general capability. You needed fit.

What the team had been doing without saying it

The account manager who quietly switched back to the cheap model for the renewals queue was running this Prep List in her head. She had not written it down. She had not used the words. She knew the work. She knew the cost-of-being-wrong was low because she edited every reply. She knew the volume was high. She knew the spec was clean. She knew her own review burden. And she knew the cheap model said “I’ll send the standard reminder with the policy number” faster and with less polite-but-irrelevant context than the sharper one did.

She had been making the right call before anyone gave her a framework for it. “It just gets through the queue,” was a five-word version of the work scored fit higher than capability and the model that fit won.

The trap most operators fall into is undoing that kind of judgment from above. “We’re standardizing on Opus. It’s the best.” What that sentence does is overrule the operator who knew the work, in favor of a measurement that did not know the work, on the assumption that the measurement is leadership and the judgment is preference. It is exactly backwards. The measurement is the abstraction. The judgment is the data. Yeah. There it is.

The Monday move

Pull one workflow. Just one. The one your team runs the most. Open a notebook. Write the workflow’s name at the top.

Now answer five questions, in order. What does it cost when this output is wrong by one increment. How many times a day do you run it. Can you write good-looks-like in one sentence. How much review is the human going to do anyway. How does the model behave when it is unsure.

The model the workflow wants almost always declares itself before you finish the fifth question. Sometimes the answer is the heavy model, and you are right to pay for it. More often the answer is the cheap model, with a tighter prompt, and a review pass on a small sample. The bill goes down. The team gets faster. Nothing on the leaderboard changed. The change happened on your menu.

Does that make sense? You can run this on any agent chain you have built, any default model assumption your team is making, any line of the model-cost report your CFO is asking about. The leaderboard ran a benchmark. The benchmark scored the model. You score the work.

Your workflow doesn’t care which model topped the leaderboard.

Source influence: Nate B Jones on model routing by workflow, not by benchmark. Distilled and operator-translated.

Framework spine: The Four D’s, specifically the DECIDING stance. Read the full framework. Sister piece: Pick the Decision. Latency Answers Itself. (the dashboard cousin of the same operator habit).