Test design — not executedCoding agents
How to evaluate AI pull-request review
Use a bounded task with observable results. The candidates below are research starting points, not tested recommendations.
Source-linked research and editorial guidance. Capabilities are not independently tested unless a profile explicitly includes a reproducible test record. Sources checked November 13, 2026.
- Inputs
- A fixture pull request with known defects, acceptable changes and repository conventions.
- Expected deliverable
- A set of findings with file references, severity, rationale and proposed remedies.
- Procedure
- Seed known bugs
- Include benign changes
- Run the review without revealing the answer key
- Adjudicate each finding
- Repeat on a second change
- Approval boundary
- Keep review comments separate from automatic code changes and merges.
- Acceptance criteria
- Measure defect recall, actionable precision, false positives, latency and reviewer time.
- Failure modes
- Counting every comment as useful, treating style opinions as defects and measuring only the number of findings.
Candidates to investigate
Editorial topic relevance — task support not verified. Research starting points — none of these has been run against this task.
Sources
- coderabbit.ai (opens in a new tab)
- coderabbit.ai/pricing (opens in a new tab)
- qodo.ai (opens in a new tab)
- devin.ai (opens in a new tab)
- cognition.com (opens in a new tab)
- devin.ai/pricing (opens in a new tab)
- cognition.com/blog/series-e (opens in a new tab)
- southparkcommons.com/companies/cognition-ai (opens in a new tab)
- docs.devin.ai/work-with-devin/devin-mcp (opens in a new tab)
Something out of date? Suggest a correction