Recommendation: start with bounded repository tasks that have explicit acceptance criteria, reproducible checks, and a reviewable change boundary. Require isolated execution, narrowly scoped permissions, and human approval for consequential actions. Measure review, rework, stability, and delivery outcomes alongside coding time.
This paper separates product documentation, benchmark results, causal research, observational evidence, and engineering recommendations.
Start with work your team can assess
Choose a task with clear expected behaviour and a manageable review surface. A narrowly scoped defect, interface change, dependency update, or test improvement provides a more useful trial than an instruction to improve an entire application. Record acceptance criteria before the agent starts.
Select representative work, including incomplete documentation, integration boundaries, awkward conventions, and slow or unreliable tests. Include changes where the intended result must be demonstrated quantitatively, such as performance or infrastructure work. A trial made entirely of isolated examples will not answer whether the agent can operate safely in the repository that matters.
Mergeability alone is a weak selection criterion: accepted changes can still create follow-up maintenance, weak validation, or operational risk.
Make the development environment reproducible
Give the agent the documented setup and verification commands that a new engineer would need. Specify where implementation rules live, which generated files should be left alone, and how relevant tests run. Report failed environment setup separately from failed implementation.
Use an isolated checkout, non-production test data, short-lived credentials, and a task-appropriate network policy. Separate development access from release authority. Require a maintainer decision before changing public interfaces, adding runtime dependencies, modifying deployment configuration, or accessing production systems.
Repository configuration is part of the execution boundary. OWASP’s Q3 2026 exploit roundup, published 8 October 2026, describes cases where opening a cloned repository in an AI coding tool could execute code without a conventional installation step. Review agent instructions, editor configuration, hooks, package scripts, and workflow files as executable supply-chain material.
Retain session logs, tool calls, test output, changed-file lists, and commit history where possible.
Keep the agent’s task and evidence together
A useful handover includes the requested behaviour, changed files, commands that ran, results obtained, and remaining uncertainty. Ask the agent to distinguish executed checks from recommended checks. The reviewer should reproduce the evidence without reading the full conversation.
For repository work, begin with inspect, plan, change, verify, and review. Add delegation or parallel work only when the team can observe state, prevent conflicting edits, and recover from partial completion.
Longer-running agents make state capture, permission expiry, interruption, and recovery more important. METR’s Frontier Risk Report, published 19 May 2026, describes coding agents working autonomously on internal research tasks for tens of minutes and occasionally hours, which is capability evidence from one setting.
Define expected behaviour before reading the patch
Write down what must become true, what must remain true, and which failure cases matter. For access control, include requests that should be rejected as well as those that should succeed. For data transformation, include missing, malformed, and boundary inputs.
Derive some checks from requirements, known defects, invariants, or externally observable behaviour rather than from the generated implementation alone. If the same misunderstanding appears in both code and tests, a passing suite may only confirm internal consistency.
A new observational study, Testing with Coding Agents, submitted 26 October 2026, analysed 114,134 merged pull requests across 391 Java repositories. Within repositories with mixed adoption, agent-supported pull requests were more likely to add tests and less likely to modify or delete existing tests. The study found no clear difference in most structural characteristics or production-change coverage, and reported that test quality remained uncertain. Its data covers January 2025 to April 2026, and its repository signals do not establish causation.
A regression test should show the reported failure before the fix and expected behaviour after it. Inspect test changes for removed assertions, weakened configuration, or reduced coverage. A successful build is evidence only for the checks that ran.
Give automated review a defined role
An automated reviewer can provide additional observations, but the team decides which findings are valid. Use it to investigate inconsistent authorisation checks, changed error paths, unsafe deserialisation, missing transaction handling, compatibility regressions, or insufficient test coverage.
Measure the work created by review as well as the defects it finds. Record accepted, rejected, duplicate, and unactionable findings. Comment volume is not a quality measure.
A study of proposed pull-request alternatives, submitted 26 October 2026, reviewed 50 grey-literature documents from 31 organisations or individuals; most reported approaches retained pull requests in some form. When review happens in agent sessions, preserve durable records of requirements, decisions, evidence, and approvals.
Check the whole change at the relevant boundaries
Choose verification based on the effect of the change. An interface adjustment may need a consumer check. A schema migration may need compatibility and rollback exercises. A visual change needs inspection in the rendered interface.
Performance changes require measured baselines, representative workloads, and relevant trade-offs. The October 2026 study of agentic performance pull requests, submitted 2 October 2026, found that agent-authored optimisation pull requests were merged less often than human-authored ones in its open-source sample. Agents relied more on static reasoning and less on benchmarks, while about half of validated pull requests in both groups reported no quantitative performance metric. The sample is observational and drawn from selected public repositories.
Security review should include instructions, repository context, tools, extensions, and external services. OWASP’s Agent Control Standard, dated 1 September 2026, defines middleware hooks for runtime policy enforcement; assess any implementation on its own evidence.
Read productivity claims in their original setting
A study’s task, participants, tools, date, and definition of success determine what its result can tell you. Coding speed, accepted pull requests, and end-to-end delivery performance are different measures.
METR’s early-2025 randomised study, published 2025, found that experienced open-source developers took 19% longer with AI tools in that setting. Its experiment-design update, dated 24 February 2026, reported that selection effects, task choice, concurrent agents, and unreliable time reporting made the later estimate weak evidence for the size of any speedup.
METR’s Task Substitution and Uplift, dated 8 May 2026, separates speed on old tasks, speed on newly selected tasks, and change in value, since agents may make previously unattractive work feasible.
GitHub’s Octoverse 2026 report, published in October 2026, says more than 40% of detected pull requests began with an agent first draft in September 2026, up from 17% in March. The figures measure adoption, and the report notes its agent signals are incomplete.
Treat benchmark capability, self-reported speed, vendor adoption figures, and repository telemetry as hypotheses to test on your own corpus. Keep tested tools and study dates attached to every number.
Follow a change through the delivery process
Choose outcomes before the trial begins. Record elapsed time from task readiness to acceptance, review time, change size, rework, rollback or incident activity, follow-up fixes, and operational effects. Keep boundaries consistent so an apparent improvement is not simply a change in what gets counted.
DORA’s ROI of AI-assisted Software Development report, last updated 22 April 2026, allows for an initial productivity dip during rollout and directs teams towards calculations about adoption economics. It supports decision-making but does not estimate the causal impact of agents in any particular organisation.
DORA’s 2025 State of AI-assisted Software Development describes AI as an amplifier of existing organisational strengths and weaknesses. Pair implementation measures with review and stability measures: faster implementation can coexist with a larger review queue, weaker tests, or more post-merge repair.
DORA’s software delivery performance metrics guide, last updated 5 January 2026, provides established measures and definitions, though no single metric demonstrates value on its own. For an agent trial, add task-level measures such as intervention count, permission escalations, validation completeness, and time spent correcting generated work.
What we are watching
We are watching independent evidence on sustained agent work in established repositories, especially ambiguous requirements, cross-service changes, migrations, infrastructure, and operational constraints.
Further priorities are measurement designs that remain credible when developers select AI-suitable tasks, use multiple agents concurrently, or shift towards work that was previously uneconomic. We also track security work on executable repository configuration and connected-tool trust.
For the next trial, define a bounded task, prepare an isolated environment, give the agent only the permissions it needs, choose checks that can expose shared misunderstandings, preserve the review record, and measure work through acceptance and delivery. Expand scope only when evidence from your own team supports it.