Taste-Bench: AI Agents Flunk Long-Horizon Strategy Decisions

By Sophie Lindqvist

Even the strongest AI model tested on the new Taste-Bench benchmark picks the correct strategic path on just under 60 percent of simple binary questions. That model is GPT-5.6 Sol, and according to the benchmark's results, it is the best performer in the field. Thinking longer doesn't improve the outcome.

That is the headline finding from Taste-Bench, a new benchmark designed to test how well AI agents make strategic choices during long-running tasks. The number matters because it lands well below what most enterprises would accept from a system entrusted with multi-step, high-stakes work.

What does Taste-Bench actually measure?

The benchmark's focus is strategic judgment over time, not one-shot question answering. Its core test is deliberately minimal: a binary choice. Which path should the agent take, given a long-running task with consequences that unfold over many steps?

That simplicity is the point. A binary question strips away everything else — formatting, tool use, retrieval, verbosity — and isolates one capability: the ability to judge which of two options serves the task better in the long run. When the best model gets it right less than six times out of ten on that stripped-down test, the weakness it exposes is hard to explain away.

The benchmark's name points at the underlying skill. "Taste" here means the judgment call a person makes when they choose one direction over another without a complete proof that it's right. Experienced professionals do this constantly. The benchmark asks whether AI agents can do it too, and the answer so far is: not reliably.

Takeaway: Taste-Bench isolates strategic judgment, and even the simplest version of that test defeats today's best agents.

Why does "thinking longer doesn't help" matter so much?

The single most consequential line in the results is that extended reasoning doesn't fix the problem. Thinking longer doesn't help, the benchmark's creators report.

This cuts against the industry's dominant playbook. Over the past product cycles, the main answer to agent errors has been more compute at inference time — models that reason step by step, check their work, and spend more tokens before answering. That approach has lifted scores on math and coding benchmarks substantially.

Taste-Bench suggests strategic choice behaves differently. If additional reasoning time doesn't move the needle on a binary decision, the bottleneck probably isn't effort or chain length. It's something closer to the model's underlying judgment about which direction is worth pursuing — the "taste" the benchmark is named for.

Takeaway: More inference-time thinking doesn't buy better strategic decisions, which challenges the industry's favorite fix.

How far below 60 percent are we talking?

Just under 60 percent is the ceiling, set by GPT-5.6 Sol. Two facts frame that number.

First, a coin flip on a binary question lands at 50 percent. The best agent in the field sits only a few points above random chance when the question is stripped to two options. Second, every other model tested performs worse than GPT-5.6 Sol, which means the field as a whole clusters somewhere between coin-flip territory and that just-under-60 ceiling.

For context on why this matters commercially: agents are being marketed today as autonomous workers that can plan, execute, and adapt across long tasks — deployments where each step commits resources and narrows future options. A decision success rate near 60 percent on simple either/or strategic questions is a poor foundation for that pitch.

Takeaway: The best score is barely above a coin flip, and everything else in the field scores lower.

What separates Taste-Bench from other agent benchmarks?

Most agent benchmarks measure throughput-style capabilities: can the model complete a task, use a tool, book the reservation, write the code, pass the test suite? Success is usually binary and immediate.

Taste-Bench measures something upstream of execution. Before an agent completes anything, it has to choose what to attempt. Those choices compound over a long-running task — a bad early direction wastes everything that follows it. The benchmark tests exactly that compounding juncture.

This distinction maps onto a known gap in the field. Models are strong at tasks with clear, checkable answers and weak when the value of a choice only becomes apparent later. Taste-Bench formalizes that gap with a number attached.

Takeaway: Taste-Bench scores the decision before the task, not the task itself — and that's where agents break down.

What should buyers and developers take from this?

The practical reading is cautionary. If an AI agent will make strategic direction calls during extended work, this benchmark says those calls will be wrong roughly four times out of ten even with the best available model — and slower, more expensive reasoning won't reduce that rate.

That has design implications. Teams deploying agents for long-running work can treat the strategic-decision layer as a known weak point: keep humans in the loop on directional choices, constrain agents to well-defined execution lanes, and avoid architectures where one early binary mistake cascades unchecked through the rest of a task.

It also has evaluation implications. Organizations shopping for agent platforms should ask vendors how their systems perform on long-horizon strategic judgment specifically, because standard completion-style benchmarks won't surface this failure mode.

Takeaway: Treat agent strategy calls as a known failure point and design around them, rather than expecting reasoning time to close the gap.

What happens next?

The benchmark gives researchers a concrete target: raise the binary strategic-choice score above the just-under-60 percent ceiling that GPT-5.6 Sol currently holds, and do it through something other than longer thinking. Until a model clears that bar convincingly, "autonomous agent" will remain a description of execution speed, not judgment quality.