Recommendation: start with bounded repository tasks that have explicit acceptance criteria, reproducible checks, and a reviewable change boundary. Require isolated execution, narrowly scoped permissions, and human approval for consequential actions. Measure review, rework, stability, and delivery outcomes alongside coding time.
This living whitepaper covers task selection, repository context, execution environments, testing, review, security, maintainability, and measurement. It distinguishes product documentation, benchmark results, causal research, observational evidence, and practical engineering recommendations.
Start with work your team can assess
Choose a task with a clear expected behaviour and manageable review surface. A narrowly scoped defect, documented interface change, dependency update, or test improvement provides a more useful trial than an instruction to improve an entire application. Record acceptance criteria before the agent starts.
A large study of 33,000 agent-authored pull requests found that documentation, CI, and build-update tasks had higher merge success than performance and bug-fix tasks. Non-merged changes were more likely to be larger, touch more files, and fail project validation. The study analysed five coding agents across GitHub and was submitted on 21 January 2026, so it is observational evidence about selected public repositories and cannot stand in for a controlled comparison of delivery performance. See Where Do AI Coding Agents Fail?, submitted 21 January 2026.
Select representative work, including incomplete documentation, integration boundaries, awkward conventions, and slow or unreliable tests. A trial made entirely of isolated examples will not answer whether the agent can operate safely in the repository that matters.
Make the development environment reproducible
Give the agent the same documented setup and verification commands that a new engineer would need. Specify where implementation rules live, which generated files should be left alone, and how the relevant tests run. Report failed environment setup separately from failed implementation.
Use an isolated checkout, non-production test data, short-lived credentials, and a network policy appropriate to the task. Separate access needed to develop the change from authority to release it. Require an explicit maintainer decision before changing public interfaces, adding runtime dependencies, modifying deployment configuration, or accessing production systems.
Execution controls should be observable. Retain session logs, tool calls, test output, changed-file lists, and commit history where possible. If the environment permits parallel agents, isolate workspaces and make ownership of each change explicit.
Keep the agent’s task and evidence together
A useful handover includes the requested behaviour, changed files, commands that ran, results obtained, and remaining uncertainty. Ask the agent to distinguish executed checks from checks it recommends. The reviewer should be able to reproduce the evidence without reading the full conversation.
For repository work, begin with a simple sequence such as inspect, plan, change, verify, and review. Add delegation or parallel work only when the team can observe state, prevent conflicting edits, and recover from partial completion.
Longer-running agents change the operating model. METR’s MirrorCode update, dated 10 April 2026, reported preliminary success on some weeks-long coding tasks, including reimplementation of a 16,000-line codebase. METR also notes that the benchmark concerns software reproduction, a specific task type. This is capability evidence, not production evidence. It supports testing wider task boundaries while keeping requirements, review and maintainability checks in place.
Define expected behaviour before reading the patch
Write down what must become true, what must remain true, and which failure cases matter. For access control, include requests that should be rejected as well as those that should succeed. For data transformation, include missing, malformed, and boundary inputs.
Derive at least some checks from requirements, known defects, invariants, or externally observable behaviour rather than from the generated implementation alone. If the same misunderstanding appears in both code and tests, a passing suite may only confirm internal consistency.
A regression test should show the reported failure before the fix and the expected behaviour after it. Inspect test changes for removed assertions, weakened configuration, or reduced coverage. A successful build is evidence for the checks that ran; it does not show that every relevant behaviour was validated.
Give automated review a defined role
An automated reviewer can provide another set of observations, but the team still needs to decide which findings are valid. Use it to investigate specific risks such as inconsistent authorisation checks, changed error paths, unsafe deserialisation, missing transaction handling, compatibility regressions, or insufficient test coverage.
Measure the work created by review as well as the defects it finds. Record accepted, rejected, duplicate, and unactionable findings. Comment volume is not a quality measure.
The 21 January 2026 study of failed agentic pull requests found that rejection could also result from weak reviewer engagement, duplicate pull requests, unwanted feature implementation, and agent misalignment. These are workflow failures as well as code failures. Define who owns review, what evidence a pull request must contain, and when an agent should stop rather than continue iterating.
Check the whole change at the relevant boundaries
Choose verification based on the effect of the change. An interface adjustment may need a consumer check. A schema migration may need compatibility and rollback exercises. A visual change needs inspection in the rendered interface. An unrelated unit suite should not be described as validation of all these behaviours.
Security review should include instructions, repository context, tools, extensions, and external services. The OWASP Top 10 for LLMs and Generative AI Applications, 2025 edition, lists Prompt Injection and Excessive Agency among its risks. OWASP presents this as a risk and mitigation framework; it does not set a software delivery standard. The page identifies the 2025 version and was current in 2026.
Ask whether untrusted issue text, documentation, test fixtures, source comments, or dependency metadata can influence actions outside the intended task. Limit write access, network reach, credential availability, and tool selection. Treat connected tools and extensions as part of the security boundary.
Read productivity claims in their original setting
A study’s task, participants, tools, date, and definition of success determine what its result can tell you. METR’s early-2025 randomised study found that experienced open-source developers took 19% longer with AI tools in that setting. See the METR study, published 2025. Its follow-up, dated 24 February 2026, reported that selection effects, task choice, concurrent agents, and unreliable time reporting made the later estimate weak evidence for the size of any speedup. See METR’s experiment-design update, 24 February 2026.
METR’s Task Substitution and Uplift, dated 8 May 2026, separates speed on old tasks, speed on newly selected tasks, and change in value. Agents may make previously unattractive work feasible, changing the task mix rather than simply reducing time for the same work.
Treat benchmark capability, self-reported speed, and vendor adoption figures as hypotheses to test on your own corpus. Keep the tested tools and study dates attached to every number.
Follow a change through the delivery process
Choose outcomes before the trial begins. Record elapsed time from task readiness to acceptance, review time, change size, rework, rollback or incident activity, follow-up fixes, and operational effects. Keep the boundaries consistent so an apparent improvement is not simply a change in what gets counted.
DORA’s ROI of AI-assisted Software Development report was last updated 22 April 2026. Its framing explicitly allows for an initial productivity dip during rollout and directs teams towards calculations and discussion about adoption economics. It supports decision-making but does not estimate the causal impact of agents in any particular organisation.
DORA’s 2025 State of AI-assisted Software Development describes AI as an amplifier of existing organisational strengths and weaknesses. Pair every speed measure with quality or stability measures, and inspect batch size, review queues, rework, and delivery performance. A faster implementation stage can coexist with slower review or weaker operational outcomes.
What we are watching
We are watching independent evidence on sustained agent work in established repositories, especially ambiguous requirements, cross-service changes, migrations, and operational constraints. The 21 January 2026 study of failed agentic pull requests is a useful addition, but its abstract does not establish causal effects on quality or delivery outcomes. Follow-up analyses and replications matter.
Further priorities are measurement designs that remain credible when developers select AI-suitable tasks, use multiple agents concurrently, or shift towards work that was previously uneconomic. Security work on prompt injection, excessive agency, connected-tool trust, and sandbox escape paths remains directly relevant.
For the next trial, define a bounded task, prepare a reproducible and isolated environment, give the agent only the permissions it needs, choose checks that can expose shared misunderstandings, and measure the work through review and delivery. Expand scope only when evidence from your own team supports it.