NVIDIA Cosmos 3 releases four models for simulation

In this article you will see:

• What each of the four Cosmos3-Nano checkpoints does.
• How controls, images and history form an interactive simulation.
• What the 16 billion trainable parameters and the disclosed license mean.
• Which limitations and requirements to watch for before adapting the models.

What the four NVIDIA Cosmos 3 models do

A world model is a system that predicts how a scene can change when it receives controls and previous observations. NVIDIA's four models specialize this idea to generate driving data or simulate the response of an environment to robot actions.

Cosmos3-Nano-Transfer-Auto generates driving scenes for synthetic data. The page says the model can produce up to 11 camera views, including fisheye lenses, and combine RGB with LiDAR, varying weather, lighting, time of day and traffic movement.

Cosmos3-Nano-Sim-Auto is the streaming variant. It generates synchronized camera and LiDAR segments from history and updated driving controls. For closed loop interaction, NVIDIA says a simulation runtime and a WSM renderer are also required.

Cosmos3-Nano-Sim-Transfer transforms spatial controls from a simulator into RGB observations. The simulator continues to compute motion, contacts and physical state, while the model generates the visual appearance that a policy can receive.

Cosmos3-Nano-Sim-Bimanual predicts next observations from two arm robot actions. NVIDIA cites the YAM and AgiBot bodies and shows sequences that can be driven by a policy or by joysticks operated by a person.

How the Cosmos3-Nano checkpoints represent the world

The four checkpoints are post trained examples from the Cosmos 3 Nano family and have 16 billion trainable parameters. The publication makes the weights available under the OpenMDW1.1 license, whose terms allow commercial and non commercial use, subject to the license conditions.

In automotive generation, WSM scene maps specify scene structure and traffic movement. Per camera captions and text prompts guide the appearance; map controls and initial RGB inputs can serve as reference for the result.

The described architecture is a Mixture-of-Transformers. It connects separate text processing and visual generation networks by attention, predicting each segment based on the prompt, the controls and the generated history.

NVIDIA also publishes a four step training recipe: bidirectional adaptation, teacher forcing, consistency distillation and self forcing. In the final step, training uses the model's own generated history, because a small error can become context and grow in subsequent frames.

The thumbnail shows examples of synchronized observations across different cameras and LiDAR. The official Cosmos Labs replay accompanies the announcement and demonstrates NVIDIA's world model family.

What to check before adapting Cosmos 3

The checkpoint choice depends on the workflow. To produce offline variations of driving scenes, Transfer-Auto is the starting point described by NVIDIA. For streaming, Sim-Auto adds generation in short segments, but relies on simulation components to close the control loop.

For robotics, Sim-Transfer leaves the physics calculation in the simulator and generates the visual observation, while Sim-Bimanual predicts the outcome of two-arm actions. Separating who calculates physics from who generates the image helps define where sensors, controller and policy come in.

The publication lists Linux, NVIDIA GPU, Docker and NVIDIA Container Toolkit among the setup requirements for Sim-Bimanual. The vLLM-Omni example was tested on a GB200, but the page stresses that this does not define a minimum memory requirement.

Before a long training run, NVIDIA itself recommends checking the environment, validating a sample and testing a short run. Afterwards, quality should be evaluated on held-out episodes. A short run confirms that the pipeline runs, not that the model has converged.

This care applies to other NVIDIA tools. The guide to TensorRT Edge-LLM on DGX Spark documents local inference, while the NVIDIA Agent Toolkit brings together components to connect agents to enterprise data.

The NVIDIA Cosmos 3 announcement does not include a model ranking

The publication presents the four checkpoints, intended uses, license and adaptation paths. It does not disclose a comparative table that allows saying one checkpoint beats the others in quality or speed.

Results should be read according to each task. Generating driving scenes, streaming cameras and LiDAR in real time, rendering outputs from a simulator and predicting bimanual movements are different problems, with different inputs and criteria.

Visual demonstrations should also not be treated as physics validation. For use in driving or robotics, the checkpoint needs to be tested with relevant scenes, controls and hardware, plus held-out data that was not used in fine-tuning.