Test design — not executedCoding agents

How to evaluate AI pull-request review

Use a bounded task with observable results. The candidates below are research starting points, not tested recommendations.

Source-linked research and editorial guidance. Capabilities are not independently tested unless a profile explicitly includes a reproducible test record. Sources checked November 13, 2026.

Inputs
A fixture pull request with known defects, acceptable changes and repository conventions.
Expected deliverable
A set of findings with file references, severity, rationale and proposed remedies.
Procedure
  1. Seed known bugs
  2. Include benign changes
  3. Run the review without revealing the answer key
  4. Adjudicate each finding
  5. Repeat on a second change
Approval boundary
Keep review comments separate from automatic code changes and merges.
Acceptance criteria
Measure defect recall, actionable precision, false positives, latency and reviewer time.
Failure modes
Counting every comment as useful, treating style opinions as defects and measuring only the number of findings.

Candidates to investigate

Editorial topic relevance — task support not verified. Research starting points — none of these has been run against this task.

Sources

  1. coderabbit.ai (opens in a new tab)
  2. coderabbit.ai/pricing (opens in a new tab)
  3. qodo.ai (opens in a new tab)
  4. devin.ai (opens in a new tab)
  5. cognition.com (opens in a new tab)
  6. devin.ai/pricing (opens in a new tab)
  7. cognition.com/blog/series-e (opens in a new tab)
  8. southparkcommons.com/companies/cognition-ai (opens in a new tab)
  9. docs.devin.ai/work-with-devin/devin-mcp (opens in a new tab)

Something out of date? Suggest a correction