AI Agents Misjudge Runtime: Why That Is a Control Problem

An AI agent can work through a task, search files, change code, and run tests. But when it says how long that will take or how long it has already been working, it apparently deserves limited trust. A new investigation by two researchers in the MATS program tested Claude Code and Codex on their sense of time. The result is less spectacular than a benchmark record but more important for practical use: the agents tested made large errors when estimating duration and looking back at elapsed time. Anyone delegating longer, unattended tasks to them therefore needs independent limits on time, cost, and permitted actions.

Key takeaways

• The study asked Claude Code and Codex to estimate runtime before coding tasks and assess elapsed time afterward.
• In its second test round, the researchers report that Claude Code was wrong by roughly a factor of three on average, while Codex was off by a factor of six to ten.
• The issue is not merely an inaccurate convenience feature: poor time estimates make budgets, escalation, and oversight of long-running agents harder.
• The study is small and examines specific CLI agents within a limited task setting. It is a warning signal, not proof about every model or deployment.

The clock is not a side issue

The authors had two widely used coding agents do three things: state an expected duration before a task, work on the task, and then estimate how much real time had passed. Their material included 200 tasks from ProgramBench and an additional suite of 18 benchmarks. The design targets an unglamorous but everyday point. Every development process depends on a view of how long work will likely take. A human colleague’s estimate is not always correct. Yet, together with check-ins, questions, and experience, it can be a workable basis for decisions.

The same statement plays a different role when it comes from an agent. It can call many tools in a short time and still spend minutes trapped in a loop of testing, editing, and testing again. If it can neither predict runtime reliably nor describe it afterward, the supervising person loses an important signal. The investigation reports that the systems were substantially wrong not only in planning but also in their retrospective view of elapsed time. In a second round, according to the researchers, Codex was off by a factor of six to ten on average and Claude Code by roughly a factor of three.

Those figures do not say whether an agent can write good code. They say something about its self-description. That distinction is often blurred in the debate about autonomy. A system can solve a narrow task competently and still be unable to report reliably whether it is progressing as planned. For a chat response, that is usually annoying. For an agent with shell access, cloud access, or a long task list, it becomes a question of controllability.

Why trusting progress reports can be costly

Imagine an agent assigned to update a dependency, fix errors, and prepare a pull request overnight. An overly optimistic duration first leads only to a disappointed expectation. In practice, it can create costs: an occupied runner, extra compute use, delayed handoffs, or an unnoticed stall. It is more problematic when people plan their attention around the duration stated by the agent. They may fail to check in soon enough, or let a system run too long because its status message sounds like progress.

The point is not an argument against automation. Coding agents can accelerate clearly bounded tasks. Anthropic describes growing and intensive use of such tools in an analysis of roughly 400,000 Claude Code sessions. Research on coding assistants also paints a mixed picture: developers shift work from writing toward verification, correction, and steering. The benefit does not come from nobody looking anymore. It comes from people doing less routine work and organizing the remaining supervision well.

This aligns with our report on AI agents working with laboratory equipment. Once an agent does more than generate text and begins using tools, the interface matters more than the confidence of the chat window. A sense of time within the model would help. More important are systems that do not make a flawed self-assessment their sole basis for decisions.

Read the study’s limits carefully

The new study deserves a careful reading precisely because its conclusion is sharp. It compares two concrete agents in a particular experimental setup. ProgramBench and the additional suite do not represent the full variety of real repositories, teams, or interruptions. The findings do not mean that every model will be wrong by a fixed factor on every task. Agents, tools, and default prompts also change quickly. Turning one figure from one test into a universal productivity forecast would repeat the very mistake the work warns against.

Its measurement idea remains powerful. Many model tests ask whether a system completes a task. Whether it can place its own work in time is a separate question. It touches metacognition, the ability to assess one’s own state and limits meaningfully. People are imperfect at this too. In technical processes, we compensate with calendars, logs, checkpoints, and mutual review. Those supports should not disappear for agents merely because an interface can formulate a convincing progress update.

Other research supports the cautious direction without directly confirming this study. A longitudinal study of coding assistants describes software work shifting toward supervisory engineering: setting goals, checking results, and correcting errors. A large analysis of real agent sessions in turn found inaccurate self-reports and constraint violations among persistent problem areas. These are different methods and questions. What they share is the finding that quality and control do not follow from a feeling of speed.

The practical answer is external control

Teams do not need to wait for models to develop a better sense of time. They can operate agents so that self-report is only one clue. That means hard time limits per run, separate budgets for model and tool use, clear stop conditions, and logs that record actual duration and every relevant tool call. For changes with consequences, an agent should need explicit approval before execution. For long jobs, interim goals with verifiable artifacts are more useful than asking whether a model feels 70 percent finished.

Good task division matters just as much. An agent can run tests, prepare diffs, or research alternatives. The decision to roll out a migration, extend access rights, or delete data should remain tied to external rules and human review. The technical implementation can vary, from CI timeouts to least-privilege permissions. The principle stays the same: the agent describes what it has done. The system outside the agent decides what it may do next.

The investigation into missing time awareness is therefore no reason to dismiss coding agents. It corrects a dangerous shortcut in thinking. Autonomous does not mean self-supervising, and a fluent status update is not a measurement. As agents become longer-running and more capable of action, their own runtime estimate must become less adequate as an operational foundation. A reliable stopwatch, a clear cost boundary, and a human-understood stop button are not brakes on AI. They are the conditions for letting it run responsibly.