Test design — not executedCoding agents
How to evaluate an agent for fixing a GitHub issue
Use a bounded task with observable results. The candidates below are research starting points, not tested recommendations.
Source-linked research and editorial guidance. Capabilities are not independently tested unless a profile explicitly includes a reproducible test record. Sources checked November 13, 2026.
- Inputs
- An issue, a fixture repository, setup instructions and acceptance tests.
- Expected deliverable
- A reviewable patch or pull request, test output and an explanation of remaining limitations.
- Procedure
- Reproduce the issue
- Agree on allowed files and dependencies
- Implement the smallest relevant change
- Run the acceptance tests
- Return the diff and evidence
- Approval boundary
- Require approval for merging, release, production credentials and changes outside the fixture.
- Acceptance criteria
- Tests pass; the reported issue is addressed; unrelated behavior is preserved; a reviewer can reproduce the result.
- Failure modes
- Passing tests with a weakened assertion, unrelated refactors, hidden manual edits and a PR that does not address the issue.
Candidates to investigate
Editorial topic relevance — task support not verified. Research starting points — none of these has been run against this task.
Cognition
Operates Devin for planning, writing, testing and shipping code.Devin
Software engineering agent with vendor-described planning, coding, browser testing and pull-request workflows.Factory
Enterprise coding agents spanning planning, implementation, review and shipping.OpenHands
Open platform for coding agents, an SDK, cloud runtime and agent control plane.Cursor
Coding agent available through desktop and CLI interfaces.Sources
- cognition.com (opens in a new tab)
- devin.ai (opens in a new tab)
- devin.ai/pricing (opens in a new tab)
- cognition.com/blog/series-e (opens in a new tab)
- southparkcommons.com/companies/cognition-ai (opens in a new tab)
- docs.devin.ai/work-with-devin/devin-mcp (opens in a new tab)
- factory.com (opens in a new tab)
- factory.com/pricing (opens in a new tab)
- openhands.dev (opens in a new tab)
- openhands.dev/pricing (opens in a new tab)
- cursor.com (opens in a new tab)
- cursor.com/pricing (opens in a new tab)
Something out of date? Suggest a correction