AWS open-sources aws-bench to score AI agents on cloud tasks
A new research-preview benchmark gives model builders a reproducible way to measure how well AI agents handle real AWS infrastructure work.
What is aws-bench and how does it measure AI agent performance on AWS?
aws-bench is an open-source benchmark from AWS, currently in research preview, that evaluates how accurately and efficiently AI agents complete tasks on AWS infrastructure. Each test case pairs a natural-language query with a defined cloud resource state and a verified answer, giving researchers a consistent, reproducible scoring baseline.
Why AWS built it
Model providers and AI researchers building agents that operate on AWS have lacked a shared, objective standard for measuring performance. Without one, teams can't easily compare agents, diagnose specific failure modes, or demonstrate incremental improvement over time.
AWS says the test cases were derived from analysis of real AWS usage patterns, covering three broad categories: investigation, troubleshooting, and infrastructure creation. That grounding in actual usage is intended to make benchmark results meaningful beyond synthetic or toy scenarios.
How the benchmark works
Every test case in aws-bench consists of three elements: a natural-language prompt, a pre-defined cloud resource state for the agent to operate against, and a ground-truth answer used to score the response. This structure allows any agent or foundation model to be evaluated on identical conditions, making cross-model comparisons valid.
The release ships with a CLI tool that handles three phases of a benchmarking run:
# Typical aws-bench workflow (from README)
aws-bench setup # instantiate testing environment
aws-bench run # execute and score evaluation
aws-bench reset # restore resource state between runs
The reset capability is notable: it lets researchers run repeated evaluations without accumulated cloud state interfering with subsequent test cases.
Who the benchmark is for
AWS positions aws-bench primarily for two audiences. Foundation model providers can use it to identify gaps in how their models reason about AWS services and to track whether fine-tuning or other improvements are actually moving the needle. AI researchers building agent harnesses, such as tool-selection logic, memory systems, or multi-step planners, can use it to isolate whether gains come from the underlying model or from the harness design.
Because the benchmark is open-source and publicly available on GitHub, third-party labs and independent researchers can run it without any special AWS partnership or access arrangement.
What it does not cover
The announcement describes a research preview, which typically signals that the scope, task categories, and scoring methodology may still evolve. AWS has not published specific numbers on how many test cases the initial release includes, what AWS services are represented, or how scoring is weighted across task types. Researchers evaluating whether aws-bench suits their needs will want to review the README directly for those details.
Broader context
Benchmarks for AI agents operating in real computing environments have become an active area, as generic language-model evaluations don't capture the tool-use, multi-step planning, and error-recovery behaviors that cloud infrastructure tasks demand. By releasing its own benchmark tied to AWS-specific tasks, AWS is also signaling that it wants performance on its platform to become a tracked, publicly comparable dimension of foundation model evaluation.
aws-bench is available now on GitHub, with setup instructions in the repository's README.