Agree that a spec has to be able to fail. The step we would add sits before the acceptance test: the questions the spec never asked. An executable test pins what someone thought to write down, but the expensive misses are the cases nobody listed, like what happens on a retry or a partial refund. Who writes the acceptance tests in your setup, the developer or the agent? If the agent turns the Markdown into tests, its misreading goes into the test as well, and then the test passes.
The process I recommend has two gates, each with AI doing the work and a human approving:
STEP 1: Acceptance Criteria: In Wave 1, AI reads the informal specs, asks questions, and generates acceptance criteria. In Wave 2, the team (PO, developers, QA engineers) writes, refines, and approves the acceptance criteria during backlog refinement. The waves may repeat, ultimately finishing with human approval. This activity requires domain judgment and critical thinking.
STEP 2: Acceptance Tests: AI translates the approved criteria into acceptance tests. A developer then reviews them, checking that each test is a correct 1:1 mapping from its acceptance criterion. This activity is a mechanical check.
On your "questions the spec never asked": in step 1, both AI and humans ask questions, and those questions feed the refinement. The acceptance criteria are written from what both sides are aware of. Neither side will catch everything, and both can miss questions (e.g. what happens on retry or partial refund). Step 2 makes sure the tests say exactly what the criteria say, so AI can't quietly change the meaning.
Thank you, that is a clear split, and making the second gate a mechanical 1:1 check is a neat way to stop meaning drifting. On the first gate, one thing that has helped us: have the AI sort its questions into the ones the team must answer and the ones it would otherwise settle silently with a default. Refinement then spends its time on the second list, which is where the retry and partial-refund cases tend to hide. In your experience, do the AI’s questions or the team’s catch more of those?
"On the first gate, one thing that has helped us: have the AI sort its questions into the ones the team must answer and the ones it would otherwise settle silently with a default."
- I find that helps me too, I always ask AI to generate both questions and recommended answers, so that we miminize how much humans need to answer...
"In your experience, do the AI’s questions or the team’s catch more of those?"
- In my experience, though I must admit, I still find it hard to say this, but that AI generates excellent questions and brainstorming the scenarios. However, don't take this as a rule, I still cannot say with certainty, who will catch more of these, and this is why I see human involvement as must-have.
This is how I view the value of human involvement:
1. Biggest value is in acceptance criteria refinement
2. Second value is in checking mapping acceptance criteria to acceptance tests
Questions with recommended answers is a good habit: the team approves or corrects instead of starting from blank. Your second point is the one I would underline. A test can pass while mapping to the wrong criterion, and nobody notices because both look finished. Thanks for laying out the two gates, it is a clear model.
Acceptance tests provide a binary response to the question: does our system meet requirements? Acceptance tests are simimulatenously both the specification and the test, two-in-one.
Agree that a spec has to be able to fail. The step we would add sits before the acceptance test: the questions the spec never asked. An executable test pins what someone thought to write down, but the expensive misses are the cases nobody listed, like what happens on a retry or a partial refund. Who writes the acceptance tests in your setup, the developer or the agent? If the agent turns the Markdown into tests, its misreading goes into the test as well, and then the test passes.
The process I recommend has two gates, each with AI doing the work and a human approving:
STEP 1: Acceptance Criteria: In Wave 1, AI reads the informal specs, asks questions, and generates acceptance criteria. In Wave 2, the team (PO, developers, QA engineers) writes, refines, and approves the acceptance criteria during backlog refinement. The waves may repeat, ultimately finishing with human approval. This activity requires domain judgment and critical thinking.
STEP 2: Acceptance Tests: AI translates the approved criteria into acceptance tests. A developer then reviews them, checking that each test is a correct 1:1 mapping from its acceptance criterion. This activity is a mechanical check.
On your "questions the spec never asked": in step 1, both AI and humans ask questions, and those questions feed the refinement. The acceptance criteria are written from what both sides are aware of. Neither side will catch everything, and both can miss questions (e.g. what happens on retry or partial refund). Step 2 makes sure the tests say exactly what the criteria say, so AI can't quietly change the meaning.
Thank you, that is a clear split, and making the second gate a mechanical 1:1 check is a neat way to stop meaning drifting. On the first gate, one thing that has helped us: have the AI sort its questions into the ones the team must answer and the ones it would otherwise settle silently with a default. Refinement then spends its time on the second list, which is where the retry and partial-refund cases tend to hide. In your experience, do the AI’s questions or the team’s catch more of those?
"On the first gate, one thing that has helped us: have the AI sort its questions into the ones the team must answer and the ones it would otherwise settle silently with a default."
- I find that helps me too, I always ask AI to generate both questions and recommended answers, so that we miminize how much humans need to answer...
"In your experience, do the AI’s questions or the team’s catch more of those?"
- In my experience, though I must admit, I still find it hard to say this, but that AI generates excellent questions and brainstorming the scenarios. However, don't take this as a rule, I still cannot say with certainty, who will catch more of these, and this is why I see human involvement as must-have.
This is how I view the value of human involvement:
1. Biggest value is in acceptance criteria refinement
2. Second value is in checking mapping acceptance criteria to acceptance tests
Questions with recommended answers is a good habit: the team approves or corrects instead of starting from blank. Your second point is the one I would underline. A test can pass while mapping to the wrong criterion, and nobody notices because both look finished. Thanks for laying out the two gates, it is a clear model.
Only acceptance tests can tell us if it's working properly.
Acceptance tests provide a binary response to the question: does our system meet requirements? Acceptance tests are simimulatenously both the specification and the test, two-in-one.