Developer experience items found while using the library from a consumer project.
A failing test should show the workspace and the agent's last message
When a then step fails, the workspace has already been cleaned up and the agent's final message is not part of the error. A flaky run cannot be diagnosed.
Suggested fix: an opt-out for cleanup (AGENTS_GWT_KEEP_WORKSPACES=1 or keepOnFailure), the workspace path in the failure output, and a one-line log of the agent's final message from executing_the_agent.
Treat "the agent asked a question and changed nothing" as a failure
In headless mode an agent that asks for clarification exits normally with an untouched workspace. The test then fails on a later assertion (missing file) with no hint.
How we saw it: on a vague prompt, haiku asked "could you clarify what project you'd like me to describe?" in 3 of 7 runs. Once the agent's message was visible the cause was obvious, and a self-contained prompt went 3 for 3 at half the cost.
Suggested fix: snapshot the workspace before and after the run; when nothing changed, fail early and quote the agent's message. Add a README note that headless prompts must be self-contained.
Small helpers
executing_the_agent_with(prompt): sets the prompt and runs, so one test can drive several runs in the same workspace.- An agent descriptor for writing a scenario once and running it against every configured agent: default model, conventions file name (
CLAUDE.md or AGENTS.md), where a skill or rule lives, whether the result reports cost. The consumer project has a working shape.
Size: all small.
Developer experience items found while using the library from a consumer project.
A failing test should show the workspace and the agent's last message
When a
thenstep fails, the workspace has already been cleaned up and the agent's final message is not part of the error. A flaky run cannot be diagnosed.Suggested fix: an opt-out for cleanup (
AGENTS_GWT_KEEP_WORKSPACES=1orkeepOnFailure), the workspace path in the failure output, and a one-line log of the agent's final message fromexecuting_the_agent.Treat "the agent asked a question and changed nothing" as a failure
In headless mode an agent that asks for clarification exits normally with an untouched workspace. The test then fails on a later assertion (missing file) with no hint.
How we saw it: on a vague prompt, haiku asked "could you clarify what project you'd like me to describe?" in 3 of 7 runs. Once the agent's message was visible the cause was obvious, and a self-contained prompt went 3 for 3 at half the cost.
Suggested fix: snapshot the workspace before and after the run; when nothing changed, fail early and quote the agent's message. Add a README note that headless prompts must be self-contained.
Small helpers
executing_the_agent_with(prompt): sets the prompt and runs, so one test can drive several runs in the same workspace.CLAUDE.mdorAGENTS.md), where a skill or rule lives, whether the result reports cost. The consumer project has a working shape.Size: all small.