A team of AI agents sounds like the obvious shortcut: a lead agent breaks the job into pieces, helpers work in parallel, and the result comes out better. A new set of measurements from benchmark provider Vals AI shows that this math often does not work out for coding. The teams cost up to five times as much as a single agent, yet improved results measurably in only one of four comparisons. If you use tools like Claude Code or Codex, the findings tell you fairly precisely when subagents are worth it.
Key takeaways
- Vals AI had GPT-6 Sol and Claude Opus 5.5 build 50 web apps each, once alone and once as a team with up to five concurrent subagents.
- Teams cost 1.8 to 5.1 times as much. They were reliably better in only one configuration: GPT-6 Sol at medium reasoning effort, up 7.3 points.
- At its highest setting, Opus 5.5 cost $122 per app as a team instead of $23.77 and took a median of 174 minutes instead of 75.
- Anthropic’s own measurements point to the real benefit of teams: they finish faster, not necessarily better. Beyond about ten agents, quality gains flatten out.
- Rule of thumb: raise a single agent’s reasoning effort first, and use teams only when time is tight and the work splits cleanly.
What Vals AI measured
The study is based on Vals AI’s Vibe Code Bench. Models receive a product specification and have to build a complete web application from it. The finished app is deployed automatically, and browser agents then click through its intended workflows. The score is the share of passed UI tests, averaged across 50 apps. The study, titled “Do Agent Teams Pay Off?”, was published on October 9 and written by Katrina Drozdov.
In team mode, a lead agent could launch up to five subagents at once, and those could not spawn helpers of their own. Both leads got the same short instruction: split the work into independent parts, delegate, integrate, and verify. GPT-6 Sol ran through OpenAI’s Agents API, Opus 5.5 through Claude Code in headless mode, meaning without a user interface. Each was tested at medium and maximum reasoning effort, the setting that controls how long a model thinks before it answers.
The results are sobering and clear. At medium effort, GPT-6 Sol improved from 77.6 to 84.9 percent. That is the only difference that holds up statistically. At maximum effort, Sol scored 89.0 percent alone and 90.4 percent as a team. Opus 5.5 reached 91.5 percent (medium) and 89.8 percent (maximum) on its own, and 91.2 and 93.2 percent as a team. According to Vals AI, none of the Opus differences is statistically significant.
Where the money goes
On the cost side, the differences are stark. With Sol at medium effort, one app cost $1.22 for a single agent and $3.07 for a team. With Opus 5.5 at its highest setting, the price rose from $23.77 to $122 per app, a factor of 5.1. According to Vals AI, most of that is cached input: every subagent resends its own context with each model call. The more helpers there are, the more often the same text gets billed, even if caching makes it cheaper.
One comparison the study draws itself makes this especially clear: a maximum-effort Opus team scored only 1.6 points higher than a single Opus agent at medium effort, but cost roughly 30 times as much. It was not faster either. The Opus teams worked in sequential waves: the lead first wrote a shared contract file with the database schema, routes, and file ownership, followed by builder, feature, and testing agents. That took time, a median of 174 minutes versus 75. The Sol teams split the work early, ran in parallel, and got somewhat faster as a result.
Where teams actually won is revealing. For Sol, the advantage was concentrated in harder apps, about 15 points there versus about one point on easier ones. Single agents often left out features on those apps even though they reported their tests as passing. The team lead tested the combined result in the browser itself and found bugs at the seams between separately built parts. The extra layer of review was worth more than the extra labor.
Anthropic’s data: faster, not necessarily better
Anthropic confirms that teams mainly save time in its system card for Opus 5.5, published on September 22. On 166 tasks from ProgramBench, where agents must rebuild a program using only its compiled binary and documentation, a fixed five-agent team reached a score of 0.6 about 2.7 times faster than a single agent, but used more tokens to get there. On shorter research tasks, teams were actually slower without time pressure, because coordination eats time. The picture only flipped with a tight time budget: given half the time a single agent normally needs, the five-agent team beat its result at roughly 2.8 times the speed.
Anthropic also tested large teams of up to 100 agents that spent 24 hours building a knowledge base for a company or working on a math proof in the proof language Lean. Scores rise with team size, but with diminishing returns. According to The Decoder, which analyzed the charts, the knowledge base score climbed from 0.53 with one agent to 0.70 with ten, but only to 0.74 with 100. The large teams’ gain lies in speed: they reach a given level within a few hours instead of at the end of the day. One side observation stands out: the 100-agent team working on the proof organized itself into two tiers with twelve sub-leads, while the knowledge base team stayed flat.
This fits what Anthropic wrote about its own research system back in June 2025. There, multi-agent systems used about 15 times as many tokens as a regular chat, and token usage alone explained 80 percent of the performance differences. Anthropic explicitly named coding as a field where less work can truly be parallelized than in research. A similar picture emerged just recently with Claude Science’s ultraviolet sky map, where the work consisted of many similar, independent subtasks.
What this means for your own work with agents
For anyone using Claude Code, Codex, or similar tools, three practical rules follow. First, before adding subagents, look at reasoning effort. For GPT-6 Sol, the highest setting gave a single agent 11.4 more points, more than any team, and at $4.82 per app it cost less than most team variants. Second, teams pay off when a task breaks into genuinely independent pieces, such as parallel research on several questions, reviewing many files, or separate modules with clear interfaces. Third, when time is the bottleneck, a team can be the right choice even if it costs more.
Vals AI points out the study’s limits itself. Each configuration ran only once, the models ran in different software environments, and the results apply first and foremost to building complete web applications. Part of the team advantage might also simply come from the delegation instruction that only the teams received.
Bottom line: the value is in review, not in numbers
The measurements contradict the idea that you just need to throw enough agents at a problem. Where teams really helped with coding, the reason was a second reviewer that caught incomplete results. That suggests a cheaper path than five parallel helpers: one strong agent at high reasoning effort, followed by a targeted review pass. Agent teams remain a tool for speed and for work that divides well. Anyone using them as a quality dial is, for now, mostly paying for the coordination between agents.