In this article you will see:
Decision Index v2: AI benchmark for decisions on documents
Artificial Analysis launched the second version of Decision Index on November 8, 2026, a benchmark that measures how AI models make decisions on real documents. The main change is that 45% of the index now uses private data, compared to 25% in the previous version.
Marcelo Santiago
Criador do bridgecode
The biggest AI newsletter in the World
Get AI articles in your inbox
Technical curation on models, agents and coding tools. No spam, unsubscribe anytime.
- What Decision Index v2 is and why version 2 matters
- How the Decision Score works and the new AA-Paperclip component
- The results: who leads and how much each decision costs
- When to use a decision model instead of a general LLM
What is Decision Index v2
Decision Index measures the ability of AI models to make decisions and extract facts from real documents. The scenario is that of a company that processes thousands of receipts, invoices, resumes and scientific articles per day and needs AI to decide what to do with each one.
In version 2, the index changed structure. The weight of private data rose from 25% to 45%. Private data are sets that are not publicly available, which prevents models from being trained specifically for that test. It is the same logic as FrontierCode 2.0 and Terminal-Bench 4.0, which also use private tasks.
The main component is called AA-Paperclip: 1,500 private documents (resumes, receipts, invoices and AI articles) with 5 decisions and 10 fact extractions each. The model only passes if it gets all 15 answers for a document correct. This all or nothing requirement separates models that look close in individual metrics.
How the Decision Score works
Decision Score is the key metric of version 2. The formula is 50 × (1 + c − (p − c)²), where p is the probability the model assigns to its answer and c indicates whether the answer was correct (1) or wrong (0). In practical terms, it is equivalent to 50 × (1 + accuracy − Brier score).
The idea is that a model that is correct with high confidence scores more points than one that is correct with low confidence. And a model that is wrong with high confidence loses more points than one that is wrong with low confidence. This captures calibration, something raw accuracy ignores.
The practical result is that the ranking changes. On AA-Paperclip, Clef passes 36% of documents and Celeris-1.1 Decision passes 24%. But Celeris-1.1 Decision ties on Decision Score because of better calibration (Brier score of 0.15 versus 0.27). On the overall index, GLiDE v1.2 leads Drex Max v1.1 in accuracy (60 versus 58), but Drex Max v1.1 comes ahead on Decision Index.
The results: who leads and how much it costs
Three decision models are within 1 point at the top of the index. Drex Max v1.1 from Nace AI and GLiDE v1.2 from Fastino Labs tie with 73, followed by Mammoth 27B from Base Labs with 72.
General language models also participate. Claude Haiku 5.5 (Max) scores 77 and GPT-6 Luna (Max) scores 76, about 3 points above Drex Max v1.1. But cost and latency are very different: GPT-6 Luna costs US$ 1,530 per 1 million tasks versus US$ 102 for Drex Max v1.1 (15 times more) and takes 11.34 seconds per task versus 275 milliseconds (about 40 times slower).
Cost per task varies more than 60 times among decision models. Celeris-1.1 Decision costs US$ 14 per 1 million tasks, Decider v1.2 costs US$ 68 and Drex Max v1.1 costs US$ 102. Clef costs US$ 859. Token price is not the only factor: most decision models charge per document once for all questions, but GLiDE v1.2 and Decider Flash v1.2 charge per question, which multiplies the cost.
Latency also varies a lot. Celeris-1.1 Decision recorded 56 milliseconds per task, Elephant 9B took 70 milliseconds and Jev 1.14 took 92 milliseconds. Claude Haiku 5.5 (Max) takes 32.65 seconds. Mammoth 27B is 4 points behind GPT-6 Luna (Max) on the index and is 118 times faster.
Frequently asked questions about the Decision Index v2
What is Decision Index v2?
It is the second version of Artificial Analysis's benchmark that measures how AI models make decisions and extract facts from real documents. The main novelty is that 45% of the index now uses private data, versus 25% in the previous version.
How does the Decision Score work?
The formula is 50 × (1 + c − (p − c)²), where p is the probability the model assigns to its answer and c indicates whether the answer was correct. It is equivalent to 50 × (1 + accuracy − Brier score). It rewards correct answers with high confidence and penalizes incorrect answers with high confidence.
Who leads Decision Index v2?
Drex Max v1.1 from Nace AI and GLiDE v1.2 from Fastino Labs tie with 73, followed by Mammoth 27B from Base Labs with 72. Among general language models, Claude Haiku 5.5 (Max) scores 77 and GPT-6 Luna (Max) scores 76.
How much does a decision cost in the benchmark?
It ranges from US$14 per 1 million tasks (Celeris-1.1 Decision) to US$859 (Clef). GPT-6 Luna (Max) costs US$1,530 and Claude Haiku 5.5 (Max) costs US$6,240 per 1 million tasks.
When to use a decision model instead of a general LLM?
Use a decision model when volume is high and cost per task matters. Switch to a general LLM when the task requires deeper reasoning or when the decision model does not reach the required accuracy.
When to use a decision model instead of a general LLM
The choice depends on volume and error tolerance. If you process documents at scale and need fast, cheap decisions, a decision model like Drex Max v1.1 or Celeris-1.1 Decision makes sense. If you need maximum accuracy and can pay more, a general LLM like Claude Haiku 5.5 (Max) or GPT-6 Luna (Max) comes out ahead.
Practical rule: start with a decision model when cost per task matters and volume is high. Move to a general LLM when the task requires deeper reasoning or when the decision model does not reach the accuracy you need.
If you want to see how AI agents perform on code and operations benchmarks, check out the post on GDPx and agent performance on professional tasks and the Artificial Analysis Intelligence Index v5.
For those who work with AI agents day to day, bridgecode lets you coordinate multiple agents in parallel, each in its own isolated space, with its own models and keys. See how orchestration works in bridgecode.
Source: Artificial Analysis
Tags:#Decision Index v2#Artificial Analysis#benchmarks#modelos de decisão#calibração
Developer and creator of bridgecode, an agentic development environment for orchestrating squads of AI agents. Helps people going from vibe coding to professional development move from idea to shipped product.
Related articles
Benchmarks
Inkling 2 Small drops 316 swear words in benchmarks: what this reveals
Inkling 2 Small dropped 316 swear words across 2,655 benchmark runs. What this reveals about reasoning loops and how we read benchmarks.
6 min
Benchmarks
Terminal-Bench 4.0: Claude Opus 5.5 and Sonnet 5.5 lead terminal operations with 65%
Terminal-Bench 4.0 evaluates AI agents solving problems in a real Linux terminal. Claude Opus 5.5 leads with 65.15%, followed by Sonnet 5.5 with 64.14%.
5 min
Benchmarks
SRE Bench: Can AI Reverse Engineer Binaries?
SRE Bench tests whether AI agents can reverse engineer binaries without source code. GPT-6 Astra leads with 56.87%.
5 min