ThinkingBox boosts consistency of Qwen3.8-27B

On October 22, 2026, at 7:24pm Brasília time, Microsoft and Hugging Face published results from ThinkingBox Training, which uses final database state checks as a reward to tune agents.

After 50 GRPO steps, Qwen3.8-27B went from 38 to 107 tasks approved in all 20 attempts on ThinkingBox-Bench, which includes 507 tasks. Artigo conjunto.

What it is: a post-training experiment evaluated by the authors. What it is not: an independent model ranking or proof of performance on any real task.

In this article you will see:

how ThinkingBox-Bench verifies the agent's outcome
which metrics changed after tuning
why consistency and coverage need to be read together

How ThinkingBox-Bench verifies 507 tasks

The benchmark gives the agent a backend and MCP tools, then checks the final database state. The evaluator verifies whether the outcome was completed, instead of rewarding only tool calls that look correct.

ThinkingBox Training diagram: select tasks, check signals in the database, turn results into reward, train the agent and evaluate on ThinkingBox-Bench. — Imagem: Microsoft e Hugging Face

Credit: Microsoft and Hugging Face.

The authors evaluated 507 tasks 20 times each, for a total of 10,140 attempts. Training used tasks separate from the benchmark reserved for evaluation.

What changed in Qwen3.8-27B after 50 steps

Tuning with GRPO, a reinforcement learning technique that compares groups of responses, raised pass@1 from 51.70% to 60.78%.

Tasks approved in all 20 attempts: 38, or 7.50%, before; 107, or 21.10%, after.
Tasks solved at least once in 20 attempts: 453, or 89.35%, before; 446, or 87.97%, after.

Across the metrics, tuning raised repeatability but slightly reduced tasks solved at least once. See also NVIDIA Vera's test in agent loops and OpenAI's report on the critical PHP failure.

What the ThinkingBox result does not prove

The study is from the authors, not independent. The benchmark tasks were kept out of training, but the sets share the ThinkingBox structure, user simulation, final state checking and some tools.

For that reason, the result does not demonstrate that Qwen3.8-27B is better than other models in general. It shows that, in this scenario, tuning raised consistency measured across repeated attempts.

Frequently asked questions about ThinkingBox and Qwen3.8-27B

How does ThinkingBox-Bench work?

It evaluates the final state of a backend after the agent executes a task with tools, using executable checks on database records.

What does 20 out of 20 mean?

It is the number of tasks the model completed correctly in all 20 recorded attempts.

Did fine-tuning improve all metrics of Qwen3.8-27B?

No. The average success rate and tasks approved in all attempts increased, but pass@20 fell from 89.35% to 87.97%.

Does ThinkingBox-Bench prove that Qwen3.8-27B is the best model?

No. It is an evaluation by the authors on 507 tasks and not an independent ranking or proof of general superiority.

What is GRPO in this experiment?

It is a reinforcement learning technique that compares groups of responses. In the experiment, fine-tuning was done for 50 steps using rewards verified on the final state.

More consistency came with lower coverage on ThinkingBox-Bench

The clearest advance was in tasks approved in all 20 attempts. Since pass@20 fell from 89.35% to 87.97%, the correct read is higher repeatability with a small loss of coverage, not an improvement across all metrics.