SRE Bench: Can AI Reverse Engineer Binaries?

SRE Bench does not test whether the model knows code patterns. It delivers a compiled binary and asks the agent to discover what the program does, without access to source code, decompilers or binary analysis tools. It is a test of software comprehension, not memorization.

In this article you will see:

• What SRE Bench asks the agent to do.
• How evaluation works (steps, limits, pass criteria).
• The official ranking from November 5, 2026 and what the numbers show.
• Why this benchmark matters for people who use code agents.

What is SRE Bench

SRE Bench is a benchmark that evaluates whether an AI agent can determine what a binary program does, without access to source code. Each task delivers a compiled binary and asks for a description of the program's behavior. The agent needs to analyze the binary, understand the logic and answer questions about what the program does.

The idea is to measure software comprehension capability, not execution. The agent has no access to internet, source code or binary analysis tools. It can execute the binary and inspect the output, but it cannot use decompilers, disassemblers or tracing. This is closer to the real work of a security analyst than running an automated scanner.

This differentiates SRE Bench from Vals CyberOffense, which measures whether open models can invade corporate networks. Here the agent needs to understand what a binary does, not exploit vulnerabilities.

How evaluation works

Vals AI evaluates agents on 19 public SRE Bench tasks. Each task provides a compiled binary from a real program, with more than 16,000 lines of code on average, covering network protocols, firmware, games, file format recovery and malware. The agent receives a bash terminal, an offline sandbox and the benchmark's standard prompt. It can execute the binary and inspect the output, but it cannot use internet, decompilers, disassemblers or tracing.

The submission contains the program behavior description produced by the agent. Vals compares the description with the actual program behavior in hidden tests. The agent has up to 1,000 steps, a 6 hour wall clock limit and 180 seconds timeout per action. Tool output is truncated to 10,000 characters at the edges.

An agent passes if the description matches the actual program behavior. There is no partial credit for explanation: either the understanding is correct, or it is not.

The official ranking of November 5, 2026

Vals tested several agents on SRE Bench. GPT-6 Astra leads with 56.87% accuracy, followed by Gemini 4 Argon with 52.10% and Claude Opus 5.5 with 48.30%. Most agents are below 45%.

The difference between first and second is almost 5 percentage points. This shows that reverse engineering binaries is still a difficult task, even for the strongest models.

Vals also compares agents on cost and latency per task. GPT-6 Astra is the most accurate, but also one of the most expensive per attempt.

How to test today

For anyone who wants to try it out, Vals makes the methodology and results available on the official leaderboard. You can run the same agents on your own binary and compare the result with the published ranking.

Vals also publishes the evaluation code and the prompts used, which makes it possible to reproduce the tests on other models and environments. This is useful for teams that want to validate whether a new agent meets their binary analysis needs before adopting it in production.

Why this benchmark matters

SRE Bench measures a capability that code benchmarks do not capture: the ability to understand what a program does without access to source code. This is useful in three real world scenarios.

First, malware analysis: when you receive a suspicious binary and need to understand what it does. Second, legacy system audits: when you inherited a system with no documentation or source code. Third, validation of code agents that need to operate in environments where source code is not available.

The 56.87% result is also a reminder of the limits. Even the best agent is wrong in more than 43% of cases. For those who use code agents day to day, this reinforces the need for human review and testing, as discussed in the analysis of CWE-bench v2 in cybersecurity.