How to test AI agents in cloud operations with CloudOpsBench

CloudOpsBench is an open source benchmark that measures how AI agents perform real cloud operations tasks. In addition to publishing results, the project releases the 45 tasks from version 1.0 for anyone who wants to test their own agent. This guide shows how to clone the repository, validate the tasks, and run them against an emulator or against real AWS.

If you build agents that operate infrastructure, testing with real tasks is the only way to know if the agent actually solves problems or just looks like it does. CloudOpsBench provides tasks written by DevOps engineers, with automated evaluation scripts and a documented lifecycle contract.

In this article you will see:

• What CloudOpsBench provides and what you need to bring
• How to clone and validate the tasks
• Emulator versus real AWS: how to choose
• How to submit results to the leaderboard
• Security and cost considerations

What CloudOpsBench provides

The repository contains 45 AWS tasks. Each task includes a setup script, natural language instructions for the agent, automated result checks, and a reference solution. Task logic and checks are identical regardless of the execution environment.

The project does not include an execution harness. You bring your own: the component that runs the setup, invokes the agent, and executes the checks. The repository documents the lifecycle contract in docs/seeded-tasks.md, which defines how to integrate your harness.

CloudOpsBench publishes a leaderboard with tested models. Results show resolution rate, time, tokens and cost. For details on the numbers, see the CloudOpsBench: AI agents in cloud operations.

How to clone and validate the tasks

You need Git and Python 3.11 or higher. The validation command checks the task structure, but does not run agents or create cloud resources.

$ git clone https://github.com/CloudOpsBench/CloudOpsBench.git
$ cd CloudOpsBench
$ python3 scripts/validate.py

If validation passes, the 45 tasks are ready to use. The repository is under Apache 2.0 license, so you can use it in commercial and internal projects.

Emulator versus real AWS

Tasks run identically in two environments: an AWS emulator like Vera or a real AWS account. The choice depends on your goal.

The emulator is the recommended option for development and iteration. It does not charge for resources and has no risk of modifying real infrastructure. Use the emulator to tune the agent prompt, test different models and debug failures.

Real AWS is required when you want to measure agent behavior under production conditions. Some tasks depend on services that the emulator does not reproduce faithfully. It is also the only way to submit results to the official leaderboard.

To use real AWS, create an isolated test account with limited scope credentials. Tasks can modify or delete resources and generate costs. Clean up resources after each run. Never provide real AWS credentials to emulator containers.

How to submit results to the leaderboard

The CloudOpsBench leaderboard accepts submissions from any agent configuration. To submit, you need to run the 45 tasks on the full test set and send the results through the project website. Each submission includes resolution rate, execution time, tokens consumed and total cost.

Before submitting, run the tasks in the emulator to validate that your harness works correctly. Submissions with a broken harness pollute the leaderboard and hinder comparison between models.

Security and cost considerations

The highest effort configurations in published results consume 60 to 98 million tokens per run, with costs between US$ 100 and US$ 400. For development, use medium or high effort settings, which cut cost by 50% or more with a 10 to 20 point drop in resolution rate.

Never run decommissioning tasks in a production account. Resource removal tasks delete buckets, queues, certificates and volumes. Always use an isolated test account.

To compare with other UI automation tools, see the 5 tips to operate desktop with agent-desktop.

Frequently asked questions about testing agents with CloudOpsBench

What do I need to run CloudOpsBench? Git and Python 3.11 or higher. The command python3 scripts/validate.py checks the task structure without creating resources.

Can I run without an AWS account? Yes. Use an emulator like Vera. The checks are identical in both environments.

Do I need my own harness? Yes. CloudOpsBench provides the tasks and checks, but not the component that runs the setup, invokes the agent and runs the checks. The integration contract is in docs/seeded-tasks.md.

How much does it cost to run on the emulator? Nothing. The emulator does not charge for resources. The cost of AI model tokens is the only expense.

How do I submit results to the leaderboard? Run the 45 tasks on the full test set and send the results via the project website, including resolution rate, time, tokens and cost.

Conclusion

CloudOpsBench offers a practical and reproducible way to test AI agents on real cloud operations. With 45 verified tasks, a free emulator and Apache 2.0 license, it is a good starting point for anyone who wants to measure their own agent's capability before putting it into production.

Start with the emulator, validate the harness, adjust the prompt and only then submit to the leaderboard. The difference between an agent that seems to solve and one that really solves only shows up in tasks with real side effects.

Did CloudOpsBench catch your attention? Put this new tool to work in an AI agent squad, each one in its role.

Source: GitHub

PYEOF_PLACEHOLDER = None