The Study and What It Actually Shows
A study reported this week found that one in four business executives cannot explain the outputs their AI systems produce, while a majority of these same leaders rely on AI for consequential financial tasks including expense management and payments. This is not a finding about junior employees experimenting with new tools. This is a finding about decision-makers who have formally adopted AI into core operational workflows while lacking the structural understanding to evaluate what that AI is doing. The gap between adoption and comprehension here is not incidental. It is, I would argue, the defining organizational problem of this particular moment in AI deployment.
This Is Not an Awareness Problem
The instinctive response to this finding will be to call for more AI literacy training, and that response will largely miss the point. The executives in this study are almost certainly aware that AI systems exist, aware that those systems can produce errors, and aware that responsible oversight is expected of them. Awareness was never the constraint. The constraint is the absence of what I would call structural schema: an accurate internal model of how the system produces outputs, what kinds of errors it is prone to, and under what conditions its outputs should be treated with suspicion. Kellogg, Valentine, and Christin (2020) identified precisely this distinction in their work on algorithmic management. Workers in algorithmically governed environments regularly develop surface awareness of the systems controlling their work without developing the deeper structural understanding needed to respond adaptively when those systems behave unexpectedly.
Folk Theories Are Not Schemas
The Algorithmic Literacy Coordination framework I am developing in my dissertation draws on Gentner's (1983) structure-mapping theory to distinguish between folk theories and structural schemas. A folk theory is an individual's working impression of how a system operates, assembled from partial observations and inference. A schema is an accurate representation of the system's underlying relational structure. The 25% figure from this study suggests something important: senior executives, even those who have formally integrated AI into their workflows, are operating on folk theories. They have built intuitive impressions of what the AI generally does, impressions sufficient for routine use, but insufficient for the kind of adaptive judgment required when outputs are anomalous, stakes are high, or the system is operating outside its training distribution.
This distinction matters because folk theories are self-confirming under normal conditions. When AI outputs are reasonable, there is no signal that the underlying mental model is inadequate. The deficit only surfaces under pressure, which is precisely the wrong moment to discover it.
The Organizational Theory Angle
There is an organizational structure problem layered on top of the cognitive one. Rahman (2021) described the "invisible cage" dynamic in platform work, where workers are governed by algorithmic systems whose logic they cannot inspect and whose criteria they cannot directly observe. That analysis was developed in the context of gig workers, but the structure applies here with uncomfortable precision. When a senior executive approves a financial output they cannot explain, they are operating inside a governance structure they do not understand. The formal authority runs upward through the organizational hierarchy. The actual epistemic authority runs through the model. These two authority structures are misaligned, and that misalignment is invisible as long as the outputs appear reasonable.
Why Procedural Training Will Not Fix This
The training response that most organizations will deploy in response to findings like this one is procedural: checklists for reviewing AI outputs, approval workflows, flagging criteria. This is the equivalent of teaching platform workers which buttons to press rather than teaching them why the platform responds the way it does. Hatano and Inagaki (1986) distinguished between routine expertise, competence in executing known procedures, and adaptive expertise, the ability to respond effectively in novel and uncertain conditions. Procedural AI governance training produces routine expertise. The conditions under which that expertise is most needed, namely anomalous or high-stakes outputs, are precisely the conditions under which routine expertise fails.
What This Finding Demands
The study's finding is not primarily a training story. It is a governance story about how organizations have structurally separated the authority to adopt AI from the competence to oversee it. Closing that gap requires schema induction at the executive level: not courses about what AI is, but structured learning experiences that develop accurate internal models of how specific systems produce specific outputs. Gagrain, Naab, and Grub (2024) found that algorithmic literacy, properly understood, requires engagement with structural features of systems rather than surface familiarity. One in four executives cannot explain their AI's outputs. That number should be read as a baseline measurement, not a headline.
Roger Hunt