NVIDIA Shows How AI Agents Build Virtual Experiment Environments

On October 8, 2026, NVIDIA showed how AI agents can turn an idea into an interactive simulation. The interesting part is less a polished robot video than the connection between programming, physics, and results that can be checked. For developers, this opens a path toward building virtual experiments more deliberately and trying changes directly.

Key takeaways

• AI agents can connect simulation components and write application code around them.
• Omniverse provides separate tools for scene data, physics, and sensor images.
• OpenUSD holds the virtual world together as an editable scene description.
• Several libraries are prerelease software. A useful starting point is a small, measurable experiment.

What a natural-language request can produce

In NVIDIA’s latest project collection, GPT-6 Astra connects a warehouse environment, a humanoid robot, and interactive controls, among other examples. Another experiment has simulated robots perform sports movements. NVIDIA reports 64 successful crossings of a single hurdle in 100 simulation trials. That is a result for this experimental setup, not a success rate for real robots.

A useful shift is visible here: the agent does more than explain what a simulator might look like. It works on the software used to investigate an idea. Our assessment is that this could be especially valuable when the effort of building the experimental environment has prevented a technical question from being explored. A small team might, for example, first determine which aisle dimensions are worth investigating for its planned setup.

Such an experiment should answer a narrow question. “Build an intelligent factory” contains too many unclear goals. “Show whether this body fits through the specified passage” can instead be broken down into inputs, visible states, and an outcome. The difference lies in how the task is defined: a simulation becomes useful when it determines transparently what counts as success in that particular experiment.

A virtual world needs several components

The Omniverse developer page describes a toolkit whose functions can be embedded individually in applications. Physics calculations, rendering, and shared scene data are separate responsibilities. The agent skills offered there provide instructions for bounded workflows, such as preparing simulation assets or creating a viewer. They are aimed at developers who want to connect existing components.

OpenUSD provides the shared description of the scene. According to the project’s documentation, it can organize objects, materials, and other scene properties across applications. Changes can sit in separate layers without overwriting the original data. That is useful for an experiment: the starting scene remains intact while a variation introduces a different arrangement of objects, for example. OpenUSD describes the world; that alone does not mean motion will be calculated correctly.

For motion, the ovphysx library provides a standalone connection to the PhysX physics engine. Its documentation lists both CPU and GPU simulation, without requiring a full Omniverse Kit installation. However, the interface is explicitly described as prerelease software that is not yet mature. Anyone experimenting with it should therefore record the package versions used and review updates deliberately.

ovrtx handles rendering and simulation of sensors such as cameras, lidar, and radar. Virtual sensor data serves a different purpose than an image for human viewers. ovrtx is also currently prerelease software. The shared ovstage runtime layer supplies scene data to several Omniverse libraries. This separation helps when reasoning about errors: correct geometry paired with an unsuitable sensor setting can still produce an unusable measurement.

How to build a manageable starting point

In our view, a good first step would be an existing minimal example rather than a completely new environment. The ovrtx repository provides starting points for Python and C. A developer could first render a known scene, change the camera position, and check whether the output reflects the change. Physics or automated editing by an agent would come afterward. That keeps the contribution of each component visible.

The next step is to give an asset appropriate properties. SimReady Foundation defines requirements and profiles for different use cases. Checking a profile establishes whether an asset meets the specified technical requirements. We see this as a useful initial filter: unexpected motion later on is difficult to assess if the input data was never clearly established.

A useful task might read: use the existing scene, change only the position of the obstacle, and record whether the bodies touch. The agent should then show the change it made and the corresponding output. This is a suggested approach, not a test we conducted. Its benefit would be comparability: the baseline and the variation differ in one deliberately chosen place.

This incremental approach could be particularly interesting for educational projects or technical prototypes. Teams that already have scenes and sensor data could then examine which discrepancies matter for their task. We discussed the broader challenge of moving from virtual to real environments in our article on World Labs and robot training in simulations. An additional experimental environment expands the possibilities, but does not replace measurements on the actual system.

The benefit would be a question that is easier to answer

The examples reveal a development path: AI can help translate a technical hypothesis into executable software. Whether that saves time also depends on how much rework the environment needs. The projects presented do not support a general promise about cost or speed.

The next useful benchmark would therefore be a small, repeatable experiment: the same scene, the same change, and a result that is easy to understand. If that works, the environment can be expanded to address further questions. That is the practical prospect these tools offer: more ideas could make the transition from a sketch to an experiment that can be checked.