The Specific Finding That Should Concern Enterprise Leaders
A Harvard study released this week found that AI coding agents produce 30% more code volume while resolving zero additional issues. That is not a marginal finding. It is a structural indictment of how enterprises are currently evaluating AI integration. The measurement apparatus being used, lines of code produced, is tracking the wrong variable entirely, and the gap between what is being measured and what actually matters has direct consequences for how organizations are deploying these tools and structuring the jobs around them.
This finding arrives alongside a separate report that Apple has already deployed an AI phone support assistant to replace the functions of 5,000 AppleCare advisers, with job cuts currently paused but displacement proceeding through task redistribution. Procurement, finance, and accounting departments are undergoing similar back-office agent integration. The common thread across these cases is that organizations are instrumenting for throughput while the outcome variables they actually care about, problem resolution, service quality, decision accuracy, are moving independently of the throughput metrics being tracked.
Why This Is a Classic Measurement Substitution Problem
Organizational theorists have a name for this pattern. When outcomes are difficult to observe directly, organizations substitute observable proxies. The proxy becomes the target, and the original outcome drifts. In the coding agent context, code volume is legible and easy to report upward. Issue resolution rates require attribution logic that is harder to establish cleanly, especially when human and agent contributions are interleaved. So organizations optimize for the legible metric and declare progress.
The Harvard finding suggests this substitution is already producing a specific kind of organizational delusion: leaders believe AI agents are generating productivity gains because the throughput numbers are real, even as the outcome numbers fail to move. This is not a technology failure in the narrow sense. It is a governance failure. The organization lacks the measurement infrastructure to distinguish activity from accomplishment.
The Awareness-Capability Gap, Applied to Enterprise AI Governance
My dissertation research on Algorithmic Literacy Coordination proposes a distinction between algorithmic awareness and algorithmic capability. Algorithmic awareness means knowing that a system operates according to certain rules or tendencies. Algorithmic capability means being able to respond effectively to those rules to produce better outcomes. Research consistently shows that awareness does not translate automatically to capability (Kellogg, Valentine, and Christin, 2020). The Harvard coding agent finding maps this distinction onto enterprise AI governance precisely. Firms are aware that AI agents produce code at scale. They are not yet capable of evaluating whether that code production serves organizational goals, because the organizational infrastructure for that evaluation does not exist.
Hatano and Inagaki (1986) distinguish between routine expertise, which executes known procedures reliably, and adaptive expertise, which applies principles flexibly to novel situations. Enterprise AI governance right now looks like routine expertise applied to a situation that demands adaptive expertise. The routine is: deploy the agent, track the output metric, report the gain. The adaptive challenge is: redesign the measurement architecture so that output metrics and outcome metrics are aligned. Most organizations have not made that redesign because it is organizationally expensive and requires cross-functional coordination that procurement, finance, and engineering rarely achieve quickly.
What the Back-Office Agent Wave Is Actually Revealing
The back-office agent wave described in this week's reporting, spanning procurement, finance, and accounting, is not primarily a labor displacement story, though it is that too. It is a story about organizational readiness to manage systems that produce high-volume, low-visibility outputs. An AppleCare adviser who mishandles a customer interaction is visible and attributable. An AI phone support assistant that systematically misroutes a category of calls produces diffuse, hard-to-attribute degradation in service outcomes. The organization may not detect the problem until it shows up in churn data, which lags the cause by months.
Rahman (2021) describes how algorithmic systems create invisible constraints that workers navigate without full visibility into the rules governing their situation. The Harvard study suggests that the inverse problem is now emerging at the organizational level: managers are navigating AI agent deployments without full visibility into whether the outputs those agents produce are connected to the outcomes they were hired to deliver.
The Practical Implication for Organizations Deploying Agents Now
The governance correction here is not complicated conceptually, though it is difficult operationally. Organizations need to instrument for outcome variables before or alongside deploying AI agents, not after. Issue resolution rates, not code volume. Customer problem solved on first contact, not calls handled per hour. Decision accuracy over time, not reports generated per week. The Harvard study is useful precisely because it makes visible a measurement trap that was already operating but had not yet been named clearly. The question for enterprise leaders is whether they will use it to redesign their evaluation frameworks, or explain it away as an outlier finding that does not apply to their specific deployment context.
Based on what organizational theory tells us about how institutions respond to inconvenient measurement findings, I would not assume the redesign happens automatically.
Roger Hunt