The Specific Event
OpenAI recently banned a cluster of China-linked ChatGPT accounts found to be running a covert influence operation targeting U.S. debates about data centers and artificial intelligence infrastructure. The operators used the service to generate English- and Chinese-language posts, memes, and coordinated messaging designed to shape public opinion on a specific policy question. OpenAI identified and removed the accounts. The incident is being framed primarily as a platform governance story, which is accurate but incomplete. The more analytically interesting problem is structural: the operation succeeded at the production layer before it was caught at the distribution layer, and that sequencing tells us something important about where competence gaps actually live in algorithmically-mediated environments.
Why Platform Governance Is Not the Same as Platform Literacy
The standard response to influence operations is to strengthen detection and enforcement. OpenAI banned the accounts; case closed. But this framing conflates two distinct problems. The first is a governance problem: platforms need better systems for identifying coordinated inauthentic behavior. The second is a schema problem: the people and institutions targeted by this content largely lack the structural understanding required to recognize when algorithmically-generated material is shaping their information environment. These are not the same problem, and solving one does not solve the other.
Hancock, Naaman, and Levy (2020) introduced the concept of AI-mediated communication to describe interactions where artificial agents modify, generate, or evaluate content in ways that recipients cannot directly observe. The ChatGPT influence operation is a near-perfect empirical instance of this dynamic. The recipients of those posts on X did not have access to the production conditions of the content they were reading. They had no structural cues to distinguish synthetically generated opinion from organically produced opinion. That is not a failure of individual intelligence. It is a predictable outcome of operating without an accurate schema for how AI-generated content enters and circulates through social media environments.
The Awareness-Capability Gap Appears Again
There is a reasonable objection here: most people now know that AI-generated content exists. Awareness of synthetic media is no longer a niche concern. But awareness is not schema, and this distinction is central to my dissertation research on Algorithmic Literacy Coordination. Gagrain, Naab, and Grub (2024) draw a similar distinction in their work on algorithmic media use, separating surface-level awareness of algorithmic systems from the deeper structural understanding required to actually adjust behavior in response to those systems. Knowing that AI-generated posts exist tells you approximately nothing about how to identify one, how to evaluate the credibility of a source that may or may not be synthetic, or how to reason about the incentive structures that produce coordinated campaigns.
This is what I mean by the awareness-capability gap. The gap is not a knowledge deficit in the colloquial sense. It is a schema deficit. Users exposed to the "Data Center Bandwagon" campaign were not ignorant of AI. Many of them were actively engaged in debates about AI policy. Their engagement did not protect them because engagement without structural schema does not produce the kind of adaptive expertise that Hatano and Inagaki (1986) describe. Procedural familiarity with a platform, knowing how to post, how to reply, how to read a thread, does not transfer to the novel problem of detecting synthetic influence at the point of consumption.
What This Means for Organizational Theory
Kellogg, Valentine, and Christin (2020) observed that algorithmic systems at work create visibility asymmetries: the platform sees worker behavior in detail, while workers have only partial visibility into the platform's logic. The influence operation case extends this asymmetry into the civic domain. The operators of the campaign had precise knowledge of what the platform could detect and what it could not, at least for a period of time. The targets of the campaign had no equivalent structural knowledge about the production conditions of what they were reading. That is not just a political problem. It is an organizational design problem, one that concerns how institutions develop and distribute the kind of schema-level competence that would make populations more resilient to this class of manipulation.
Rahman (2021) describes algorithmic control as an "invisible cage," a structure that shapes behavior without announcing itself. The influence operation case adds a layer to this metaphor: the cage can be operated by external actors, not just platform owners, and users who lack structural schema cannot tell the difference. The policy conversation following this incident will focus on what OpenAI did to stop it. The more durable question is what kind of cognitive infrastructure would have reduced the campaign's effectiveness before detection. Those are different interventions, and right now, almost all institutional energy is going toward the first.
The Transfer Problem, Stated Plainly
If schema induction, teaching people the structural features of how AI-generated content is produced and distributed, enables transfer across novel influence contexts, then general literacy training of this kind should outperform platform-specific warnings or one-time content removals. That is a testable claim. The ChatGPT influence operation gives us a natural reference point: a population exposed to synthetically coordinated content, most of whom had general awareness of AI but lacked structural schema for detecting it. The outcome was predictable under my framework. Whether targeted schema training would have changed that outcome is an empirical question, and one worth taking seriously as these operations become cheaper and easier to run.
Roger Hunt