A tiny prompt edit can break an AI agent without breaking its code. Google’s Agents CLI now gives developers a repeatable way to catch that kind of regression.
In an October 28, 2026 walkthrough, Google demonstrates a practical loop: run test conversations, grade the answers, compare changes, and inspect what went wrong.
What the evaluation commands do
- eval generate: Run your agent against a dataset and save its traces.
- eval grade: Score those traces using configured metrics.
- eval compare: Compare result files from two runs.
- eval analyze: Group failures into patterns worth investigating.
This matters when you change a system prompt, swap a model, or update tools. A passing unit test only proves the software runs; it does not prove the agent still behaves correctly.
Try the documented starter workflow
You’ll need Python 3.11+, uv, and Node.js. Google’s repository documents the setup command:
uvx google-agents-cli setupGoogle’s walkthrough uses the LangChain template bundled with Agents CLI v1.8.0:
agents-cli create my-agent \
--agent google/agents-cli/extensions/langchain/[email protected] \
-d cloud_run
cd my-agent && agents-cli installWith the template’s sample evaluation files in place, the documented commands are:
agents-cli eval generate \
--dataset tests/eval/datasets/basic-dataset.json \
-o tests/eval/output/
agents-cli eval grade \
--traces tests/eval/output/eval_dataset_*.json \
--config tests/eval/eval_config.yamlThese are official example commands, not commands independently tested by RumorsCity. Check the template files before adapting them to your own project.
One useful lesson from Google’s example
Google intentionally replaced a weather agent’s helpful prompt with a chef persona. Its sample mean evaluation score dropped from 4.00 to 2.75 out of 5. The important clue wasn’t just the average: individual cases revealed that the agent stopped using its weather tool correctly.
Practical takeaway: Keep a small set of representative test conversations in version control. Run them before and after every meaningful prompt or model change, and review failures case by case.
Caveat: AI-judge scores are imperfect, and a tiny dataset can overfit. Evaluation can call paid models and may transmit test prompts to configured services, so avoid real customer secrets. Local development can use an AI Studio API key; deploying to Google Cloud has separate requirements.