ToyBlocks: A Controlled Benchmark for Robots

In this article you will see:

• What ToyBlocks is and what tasks the environment offers.
• How the demonstration set and generalization tests were put together.
• What the results show in simulation and on the real robot.
• Why the physical sample still limits the conclusions.

What ToyBlocks evaluates

ToyBlocks is a simulation environment for robot learning, not a language model nor a ready-made robot. The team presents it as a “foundation world” that allows generating scenes, tasks and training data under controlled conditions.

The first version works with blocks on a table and four manipulation tasks: stacking, unstacking, flipping and rotating objects. The visual simplicity helps isolate perception, planning and motor control problems without mixing the result with complex environments.

The goal is to let researchers vary the scene systematically and repeat the same tests. This makes it easier to observe whether a policy learned a skill that works in new configurations or just reproduced examples seen during training.

The AI2 published 85,336 expert demonstrations, with about 1,600 hours and 144 million frames. These data are demonstrations for training. They do not correspond to the number of episodes used in each performance evaluation.

How the benchmark measures generalization

ToyBlocks separates familiar scenes from configurations that change the spatial composition of the blocks. The evaluation seeks to identify whether the policy keeps working when the relationship between objects does not exactly match the examples used in training.

One of the charts in the AI2 paper compares training strategies. In the two tasks shown, the in-domain rate was 83% and 90%, while initial generalization was 5% and 29%. The combination of strategies reached 81% and 89%, respectively.

The figure shows that the final rate depends on the training protocol, not just the name of the algorithm. AI2 compares co-training, delta actions, intermediate training and other configurations. The caption says that the gains are percentage points and that the combination was evaluated separately.

This kind of reading also appears in software agent evaluations. The FeatureLoop separates tasks with progressive requirements, while the NVIDIA Vera benchmark analyzes agent loops in a specific infrastructure setup.

What the tests show outside simulation

AI2 also took a hierarchical policy from the virtual environment to a Franka FR3 robotic arm with a Robotiq gripper. In the results published by the team, simple tasks with Molmo 2 and ManiFlow achieved 55.5% success in simulation, with 111 successes in 200 episodes, and 50.0% on the real robot, with 15 out of 30 episodes.

Both percentages use per-episode evaluation units. The physical sample of 30 episodes is small, so the research itself says more trials are needed to establish statistical confidence.

The study also measures atomic skills of ManiFlow, such as individual manipulation actions. On this per-command measure, the result was 80.0% in simulation, with 990 successes out of 1,237 commands, and 81.1% on the real robot, with 43 successes out of 53 commands, according to the AI2 sim-to-real evaluation figure.

Per-command and per-episode rates are not interchangeable. An individual skill can work frequently while the full task sequence fails at some step, so the research shows both metrics separately.

The paper also describes letter search tests with blocks. Half of the prepared scenes had no possible solution, and the agent was evaluated both on solving the task and on correctly identifying that it was infeasible. This test examines reasoning about the scene, in addition to physical execution.

Limits for interpreting ToyBlocks results

The numbers come from experiments published by the team that created the environment. They are not an independent evaluation of all robots, policies, or manipulation scenarios available.

The initial benchmark is also limited. It uses blocks, tabletop tasks, objects, and configurations defined by the authors. The result does not automatically predict performance in homes, factories, or robots with different sensors and mechanisms.

The physical sample of 30 episodes requires caution, and the metric denominators change depending on the evaluation. To compare policies, it is important to record whether the rate is per command or per episode, how many attempts were made, which scenes were new, and which criteria defined success.

A practical application of ToyBlocks is to test a hypothesis in stages: train on known scenes, set aside unseen configurations for evaluation, and then repeat part of the tasks on real hardware. The difference between stages shows where performance is lost, without summarizing everything into a single number.