Test design — not executedCoding agents

How to evaluate an agent for fixing a GitHub issue

Use a bounded task with observable results. The candidates below are research starting points, not tested recommendations.

Source-linked research and editorial guidance. Capabilities are not independently tested unless a profile explicitly includes a reproducible test record. Sources checked November 13, 2026.

Inputs
An issue, a fixture repository, setup instructions and acceptance tests.
Expected deliverable
A reviewable patch or pull request, test output and an explanation of remaining limitations.
Procedure
  1. Reproduce the issue
  2. Agree on allowed files and dependencies
  3. Implement the smallest relevant change
  4. Run the acceptance tests
  5. Return the diff and evidence
Approval boundary
Require approval for merging, release, production credentials and changes outside the fixture.
Acceptance criteria
Tests pass; the reported issue is addressed; unrelated behavior is preserved; a reviewer can reproduce the result.
Failure modes
Passing tests with a weakened assertion, unrelated refactors, hidden manual edits and a PR that does not address the issue.

Candidates to investigate

Editorial topic relevance — task support not verified. Research starting points — none of these has been run against this task.

Sources

  1. cognition.com (opens in a new tab)
  2. devin.ai (opens in a new tab)
  3. devin.ai/pricing (opens in a new tab)
  4. cognition.com/blog/series-e (opens in a new tab)
  5. southparkcommons.com/companies/cognition-ai (opens in a new tab)
  6. docs.devin.ai/work-with-devin/devin-mcp (opens in a new tab)
  7. factory.com (opens in a new tab)
  8. factory.com/pricing (opens in a new tab)
  9. openhands.dev (opens in a new tab)
  10. openhands.dev/pricing (opens in a new tab)
  11. cursor.com (opens in a new tab)
  12. cursor.com/pricing (opens in a new tab)

Something out of date? Suggest a correction