AI agents appear capable of taking on a substantial share of demanding physics calculations. In the TheoPhysEval benchmark, GPT-6-Astra running in Codex produced correct final answers for 49 of 50 problems, according to the research team. Posted on arXiv on October 9 and now covered by specialist media, the paper points to a concrete application: assistance with precisely defined calculations in theoretical physics.
Key takeaways
- TheoPhysEval contains 50 problems that were unpublished at the time of testing and have verifiable answers. Major unresolved foundational questions are explicitly excluded.
- GPT-6-Astra in Codex solved 49 problems with web access; Opus 5.5 in Claude Code solved 46.
- The evaluation mainly checks numerical final answers. It does not automatically certify the quality of every step in a derivation.
- The tests can, in principle, be reproduced. However, access to the dataset requires a request, and each agent was evaluated only once per test setting.
Why these problems matter for research work
An AI agent in this context combines a language model with a working environment that provides tools. Rather than simply composing a response, it can search the literature, write programs, and run calculations. That makes a difference when a task requires both an understanding of physics and numerical work.
The collection focuses on these clearly defined steps. Someone repeating a known calculation under modified assumptions does not necessarily need a new theory. They do, however, need to understand the assumptions and adapt the calculation correctly. This work can be time-consuming within a research project even when its broader question has already been established.
Reliable assistance at this level could therefore be valuable to a research group. It might allow the group to explore more variations of a calculation or create a second implementation for comparison. This is our practical inference from the test design, rather than a productivity gain measured by the benchmark. Whether an agent can fulfill that role in a particular field would need to be tested on that field’s tasks.
The distinction between completing a subtask and making a scientific contribution also matters when evaluating AI research reports. Here the objective is narrower and therefore easier to check: a specified problem should produce a verifiable result.
How to read the accuracy and cost figures
The study compares a realistic setting with internet access against an isolated setting with local computational tools. Without web access, Astra and Opus each returned 46 correct answers. The researchers interpret this as evidence of extensive memorized knowledge of the literature. This comparison alone, however, cannot neatly attribute individual pieces of knowledge to particular training sources.
The top score of 98 percent is a result from this specific setup. Different problems, other tools, or another run could produce different results. A gap of a few solved problems is therefore not a general measure of the quality difference between the products. Users should consider the model and its working environment together, because both influence the outcome.
The cost table lists $116.28 for Astra’s full realistic run. This figure is calculated from token consumption and API rates. Dividing it across the 50 assigned problems gives approximately $2.33 per problem; it is not a price for arbitrary physics research. The token bill does not establish complete project costs including supervision, computing infrastructure, and scientific review.
The figure nevertheless makes a limited experiment look plausible. A group does not have to change its entire workflow for an initial comparison. It can select a few known calculations, track costs separately, and check whether the additional implementation actually saves time. Only that observation connects a low model price with a benefit in day-to-day research.
Keep the final answer, derivation, and grading standard distinct
The researchers also converted symbolic expressions into numerically verifiable answers. This makes comparison more straightforward and avoids adding a language model as a judge. A correct final answer, however, addresses a narrower question than a fully checked derivation. Anyone building on the solution still needs to understand its assumptions, intermediate steps, and range of validity.
There was another revealing finding: for several Astra answers initially marked as wrong, the authors found errors in their own reference solutions after rechecking them. Once those were corrected, the agent’s answers proved right. This supports the value of an additional perspective from AI, while also showing why the grading standard itself needs to remain open to scrutiny.
For reproduction, the dataset page offers task packages with instructions, a Docker environment, configuration, and a verifier. Its instructions require Python 3.12 or newer and Docker, and use Harbor to run the evaluation. File access requires a request and subsequent approval. This is a documented research setup, rather than a chat test that can be used immediately without preparation.
Anyone repeating it should retain both correct and incorrect answers along with their calculations. Cases that require adapting a known formula to changed conditions are especially informative. They reveal whether the system incorporates the change or simply continues down a familiar solution path.
The next benefit lies between an idea and a finished result
TheoPhysEval offers an encouraging finding for precisely defined scientific assistance. The study does not measure the independent selection of important research questions or establish equal progress across every subfield. Nor does “unpublished” mean that all underlying methods are new or that influences from earlier AI use have been ruled out.
The natural next step is a comparison using real subtasks from the relevant field. An agent can already provide scientific value by producing an independent calculation, making another variation feasible, or uncovering a discrepancy in a reference answer. Recognizing that progress does not require claiming that the next foundational theory will now be discovered automatically.