On November 3, 2026, the Allen Institute for AI presented StateBench, an evaluation to measure whether models follow state changes over long sequences.
StateBench measures state tracking in long context
StateBench measures four levels of state tracking and shows how accuracy drops in long sequences, with explicit methods and limitations.
Marcelo Santiago
Criador do bridgecode
The biggest AI newsletter in the World
Get AI articles in your inbox
Technical curation on models, agents and coding tools. No spam, unsubscribe anytime.
The result is not a general ranking: it isolates a skill that also appears when code updates variables. The Ai2 article publishes the method, the results and the limitations.
In this article you will see:
- What StateBench calls state tracking.
- How the four difficulty levels are compared.
- What happened with pretrained models and models trained for the task.
- Why the score does not measure all of a model's capability.
What StateBench measures in long sequences
State tracking is keeping the current value of entities that change over a text. In code, this can be following the value of a variable after several assignments, instead of copying only the first or last occurrence.
StateBench separates four levels. At the R-trivial level, a value does not change. At aperiodic, there are reassignments and it matters to find the last one. At soluble, each update depends on the previous value, like adding points to a scoreboard.
The general finite level adds assignment by reference. A variable can receive the current value of another, which creates a chain of dependencies between updates.
For a programmer, this looks like reassigning variables or swapping their values in several steps.
The authors keep the number of variables, the distinct values and the points in the sequence where assignments occur the same.
What changes between comparable tasks is the type of update. This helps isolate the difficulty the benchmark intends to measure.
The paper relates the hierarchy to theoretical distinctions from Krohn and Rhodes theory.
You do not need to master this theory to understand the practical use of the set. It explains why the authors do not treat all state changes as a single problem.
How Ai2 set up the comparisons
The paper tests pretrained models with different sizes and architectures, including OLMo 3 and OLMo Hybrid, both with 7 billion parameters, in addition to smaller models.
The starting models were trained on the same dataset described by the team.
None of these pretrained models had seen contexts above 64 thousand tokens during training, according to the authors.
The evaluation then observes behavior on longer input sequences and at different tracking levels.
In another experiment, the team trained from scratch 275 million parameter models on sequences of up to 256 assignments. Then they evaluated sequences of up to 64 thousand assignments.
This test is 250 times larger than the maximum length used in training, measured in assignments, not tokens.
The technical report, the dataset and the code are public. This allows researchers to inspect the examples and the implementation.
This transparency does not automatically turn the results into a measure of performance in real applications. See also the dataset and the code.
What the results show about frontier models
In the authors' evaluation, Gemini 3.8 Flash, Claude Opus 5.5, GPT-6 Sol and GPT-6 Luna performed well on the two simplest levels. Accuracy dropped when tasks required accumulating updates or following references between variables.
On longer general finite problems, the team reports accuracy of 13% or less for the four models. That claim applies to the hardest level and the longest sequences in the test, not to every type of question or model use.
Image credit: Ai2.
In models trained directly on the task, hybrids and Gated DeltaNet variants generalized better than Transformers on some levels.
A Gated DeltaNet variant maintained near ceiling performance on simpler tasks and, on the hardest level, went beyond the length seen during training.
This does not mean one architecture wins in every scenario. The study itself notes that pretrained models and models trained from scratch are in different conditions.
Results also depend on the specific update form tested.
Limits that prevent treating StateBench as a general ranking
Sequences are controlled and keep the set of variables fixed.
In real conversations, documents or repositories, new people, objects and symbols can appear mid context. The paper says the benchmark does not yet cover that situation.
The task also does not measure code quality, tool use, information retrieval or general problem solving.
A low result on general finite points to a weakness in that test, it does not prove the model is unable to track any state in any application.
The frontier model results are the models and configurations evaluated in the paper.
Do not treat them as a universal scoreboard or as a current table of all available systems. The main goal is to compare types of state tracking under controlled conditions.
There are open questions about the exact position of some architectures in the theoretical hierarchy and about the effect of numerical precision and attention layers.
The report describes these issues as open work, not as settled conclusions.
To compare with another type of evaluation, see the blog post about DeepSeek V4.1 Flash on CursorBench.
CursorBench evaluates programming tasks; StateBench isolates state updates. The benchmarks category brings together analyses with other methods and scopes.
How to use StateBench to guide your own evaluation
If your application depends on long context, turn reported failures into test cases close to your workflow.
Include simple updates, cumulative chains and references between variables, instead of measuring only the maximum window size.
Use the same inputs, instructions and grading rules when comparing versions. Record which cases failed and why.
A synthetic benchmark helps locate a specific capability, but it needs to be complemented by real, safe examples from your product.
To compare this slice of state memory with another agent specific evaluation, see FeatureLoop and the tests with evolving requirements. The two studies observe different problems and should not be combined into a single ranking.
Frequently asked questions about StateBench
What does StateBench measure?
StateBench measures how models track variable updates across four difficulty levels, from values that never change to dependencies between variables.
What does general finite mean in StateBench?
General finite includes reference assignments, where the current value of a variable depends on the value of another and on previous changes.
What was the models' accuracy on the hardest tests?
Ai2 reports 13% or less on the longest general finite problems for the four frontier models tested. This result does not represent all tasks these models can do.
Does StateBench prove that Transformers cannot track states?
No. The benchmark uses controlled sequences with a fixed number of variables and does not cover all real world contexts, such as conversations where new entities appear.
What does the 250 times longer test mean?
No. The 250 times factor compares sequence lengths measured in assignments: models were trained up to 256 and evaluated up to 64 thousand assignments.
StateBench highlights the difference between context and state memory
The main strength of StateBench is separating tasks that look similar but require different operations. Context window size alone does not guarantee that a model will track an entire chain of changes.
To choose a system, combine this reading with tests of the work it will actually do. Do not turn an isolated score into a performance promise for code, documents or production conversations.
Instead of only following benchmarks like StateBench, split your project across agents and coordinate the flow on bridgecode.
Source: Ai2
Tags:#StateBench#Ai2#benchmark#contexto longo#rastreamento de estado#Gated DeltaNet
Developer and creator of bridgecode, an agentic development environment for orchestrating squads of AI agents. Helps people going from vibe coding to professional development move from idea to shipped product.
Related articles
Benchmarks
ToyBlocks: A Controlled Benchmark for Robots
AI2 introduces ToyBlocks, a simulation environment with four tasks to study generalization and physical transfer in robotics.
5 min
Benchmarks
CloudOpsBench: AI agents in cloud operations
CloudOpsBench measures how AI agents perform real cloud operations tasks. The best agent solves 60.6% of the 45 tasks on AWS. See the numbers.
7 min
Benchmarks
SuperCLUE November 2026: different leaders by category
SuperCLUE published in November 2026 a table with 19 models and different leaders in each category. The split ranking is more useful as a shortlist map than as a final verdict.
6 min