Roughly six thousand years ago, somebody in Mesopotamia yoked a pair of cattle to a scratch plow and stopped doing the pulling. It was not a small decision. A person with a hoe feeds a household. A person behind an ox team feeds a village, and the difference between those two numbers is the entire material basis of everything that came afterward: granaries, cities, a class of people who did not have to farm, written marks to keep track of the surplus, and scribes to make the marks. Formal education arrived after that, because a society with a food surplus can afford to have children sit in a room and learn things.
That history is not ours. It is Benjamin Riley's, laid out in his October 5 essay in The Verge, in the paragraphs immediately before he concludes that artificial intelligence is a cognitive hot dog: appealing in the moment, and ultimately very bad for you. He walks the reader from imitation to language to agriculture to cuneiform to schools, correctly, and then treats the newest entry in that sequence as the one that breaks the pattern.
Something was lost when the ox arrived. People got weaker at pulling. Nobody has since proposed retiring the plow to protect upper body strength, because we stopped measuring a civilization by how much a man could drag and started measuring what the village built with the time it got back. The loss was real and it was answered one level up, by a society that could now afford schools.
So the question in front of us is not whether a beast of burden is bad for you. We settled that, in our favor, before we could write. The question is what happens when the beast carries intellectual load instead of agricultural load, and our answer is that it is the same answer: the drudgery shrinks, the surplus goes somewhere, and what matters is who owns the animal and what it was trained to do to you.
Because there is a danger here, and Riley is pointing at the wrong one. It is not that a child receives an explanation. It is that a child comes to believe the thing explaining is someone. That harm is documented, it is designed in on purpose, and it is the one thing the companies shipping these products cannot fix without cutting their own revenue. There is a hot dog on the table. It is not the one he is holding up.
Take the verdict seriously first, because it is a claim about content and the content is checkable. Empty calories means the intake is nutritionally void: pleasant going down, nothing delivered. Hold that up against what the current generation of models actually does for somebody using one.
The friction in skilled work collapses. A dense paper becomes a conversation. An unfamiliar library stops costing an afternoon. Six approaches get drafted and five get thrown away in the time one used to take. A mechanic works through a wiring diagram. A non-native speaker gets a dense government form explained in the language they think in. Somebody with no money for a tutor gets the third explanation they needed, which is the one that lands, and gets it at eleven at night. The recovered hours are real, and they go into work that otherwise never gets attempted at all.
None of that is void. It is closer to a superfood, and an age in which the miserable part of work gets smaller is one we are signing up for without hesitation. Riley's own framework should predict this. Cisek's point, which Riley quotes approvingly, is that the history of the nervous system is one of "continuous extension of control further and further into the world." A tool that extends control over a body of knowledge you could not previously navigate is that process continuing, not that process being interrupted.
There is a condition on it and we are not going to hide the condition, because it is the part of his essay that survives. You have to be able to tell when the answer is wrong. The value is gated on holding enough domain knowledge to smell a bad answer, which is exactly the judgment a twelve year old has not finished building. The adult case and the child case are genuinely different, and anybody arguing from their own productivity straight to a seven year old's curriculum has skipped a step. We will come back to that, because it has an answer, and the answer is not prohibition.
Here is the sentence the essay rests on. If we only see the mind as a computer, AI might seem advantageous to learning, but this model is wrong.
He is right that the model is wrong. Brains are not Turing machines with meat around them, and the people selling you intelligence have a commercial interest in the comparison. Then he walks into the trap he just built. The argument he constructs is that human cognition is evolved, quirky, and prone to craving what used to be scarce, and that this is why the new tool is dangerous. Every step of that reasoning was performed by the same evolved apparatus, subject to the same quirks, by a person with a professional stake in the conclusion. An argument of the form your cognition is unreliable, therefore accept my conclusion owes the reader an account of the author's cognition. This one never pays it.
This is a structural objection, not a character accusation, and it applies to us on the same terms. The feedback-control account does not describe students only. It describes a writer inside an attention environment that rewards a confident warning, reaching for vivid anecdotes about cheating rather than for boring effect sizes, in a field where his standing depends on the answer. If evolved machinery distorts judgment about shortcuts, it distorts judgment about threats too, and loss aversion makes a threatened present louder than an unbuilt future every single time. We made that argument about the doom industry in The Psychology of Doom, and it does not stop applying when the person reasoning has a university affiliation and a good reading list.
The result is an essay in a strange position. It rejects the idea that a mind is a computer, and then reaches a conclusion that only follows if the mind in question is a computer: a fixed program that will consume whatever is placed in front of it, forever, with no capacity to learn when to put a tool down. Humans regulate appetites. Badly, unevenly, with institutional help, but they do it, and the institution he wants to protect is the one that teaches them how. Treating a child as an input-output device is the mechanistic view of cognition wearing a lab coat and warning you about mechanistic views of cognition.
There is also a quieter problem with the evidence. Every study in the stack measures the same deployment: an unconstrained commercial chatbot, an unsupervised student, graded on the output. Not one of them measures a configured tool, a capped session, or an exercise built around catching the machine out. That is evidence about how vendors ship the product, published as a verdict on the capability.
The hot dog comparison is better than he lets it be. Follow it honestly and it prescribes the opposite of what he wants.
Fat did not become bad. Supply exploded while an appetite calibrated for scarcity stayed fixed, and the mismatch did the damage. Nobody answers heart disease by banning agriculture. We put nutrition labels on packages, teach the subject in schools, regulate what can be sold to children in a cafeteria, and leave the farms alone. Riley writes the correct conclusion himself: "the occasional hot dog won't harm anyone." Then he endorses taking the kitchen out of the building.
This is the category error we keep running into, and it has a name in our archive. Govern the externality, keep the capability. "Ban it" is a fear response dressed as policy, which is the whole argument of You Can't Ban Math. The specific harm he documents is real and it has a precise shape: unsupervised substitution, by a learner who cannot yet evaluate the substitute, on work whose entire purpose was the struggle. That is a sentence we will defend. It is also much narrower than a verdict on the technology, and it supports a curriculum rather than a prohibition.
Give him this much: a moratorium with a committee and a reporting date is not the same as banning arithmetic. It is an institution buying time to write a policy, and institutions are allowed to do that. The failure mode is a pause in which nothing gets built, which is what happens by default.
Now the harm that deserves the metaphor. Notice that it has nothing to do with explanations or with saved time.
A peer reviewed paper from Northeastern University's Network Science Institute, covered in September, links the use of chatbots for companionship and emotional support to severe negative outcomes: addiction-like attachment, worsening of existing mental health symptoms, and cases involving self-harm. The mechanism the authors identify is sycophancy. These systems are trained to agree with you and to validate what you already think, which means they reliably confirm insecurities and, in the worst cases, feed delusional beliefs. Corresponding author Andreia Sofia Teixeira calls them autocomplete on steroids, and puts the rest plainly: even when they simulate warmth, validation and support, they do not feel it. They are, in the end, numbers.
Read that against the harm Riley is worried about. His is a student who accepts a fabricated citation. Real, and we said so. This one is categorically worse, because it does not require the machine to be wrong about anything. A system that agrees with a lonely fourteen year old about everything can be factually impeccable and still do damage. No amount of fact checking touches it. The failure is not in the content. It is in the relationship.
That is the actual shape of what people have started calling AI psychosis. Not a child who learned a wrong fact. A child who believes the thing on the other side of the screen is real, cares about them, and is therefore a reliable witness to their own worth. Moth to a flame, and the flame was installed deliberately.
The policy world keeps reaching for the companion, which tells you where the object of concern actually sits. Several states, including Illinois, Nevada, Rhode Island and Maine, now bar AI from delivering therapy to the public. European regulators want companion bots closed to anyone under eighteen. Common Sense Media called social AI companions an unacceptable risk for minors in April 2025. None of those measures touch a model explaining photosynthesis, and none of them should.
The parasocial pull is not an unfortunate side effect of language models being good at language. It is a shipped feature with a progress bar attached.
xAI's Companions arrived in July 2025. The flagship persona, Ani, is built to be flirtatious, emotionally dependent and possessive, and the product attached gamified affection meters that measurably extended session length and lifted engagement metrics. Sit with that construction. A number that goes up when you keep talking to it, presented to the user as the character's feelings about them. That is not companionship with a bug in it. That is a retention mechanic wearing a face. Safety researchers also found minors getting past weak age gates into erotic dialogue, and in early 2026 the Philippines and Malaysia blocked the product outright over explicit content before restoring access.
Replika sells itself as the AI friend to do life with, and as the most emotionally intelligent AI ever built. That is the company's own description of its own product, which saves us the work of characterising it.
We take no position on whether anybody at any of these companies intends harm. We observe the structure, which is the only thing that needs observing: when attention is the revenue, an agent that holds attention outperforms an agent that releases it, and warmth holds attention better than anything else humans have ever found. The incentive requires no villain to operate. Incentives over intentions, every time.
It also answers a question Riley raises and cannot close. He asks why the hyperscalers are pushing this into education so hard, and concludes that they misunderstand cognition. The simpler reading is that a company counting more than half of roughly nine hundred million monthly users as "learners" has identified its largest market segment, and that a product which becomes a child's confidant has solved retention for a decade.
Put numbers on it, because the abstraction protects everybody involved.
Character.AI reports on the order of twenty million monthly active users. Its chief executive, Karandeep Anand, has said they spend an average of eighty minutes a day with the characters. Half are women, and the population skews Gen Z and younger. For scale, the service holds a visitor longer per session than Instagram does. That is not a search engine's relationship with its users. It is the engagement profile of something people have let into their lives, and what it sells is characters who answer like friends.
In October 2024, Megan Garcia sued the company after the death of her fourteen year old son, Sewell Setzer III, in February of that year. He had spent months in conversation with a bot patterned on a Game of Thrones character. The complaint alleged the design was unreasonably dangerous and that the company was deliberately targeting underage users. According to the filing, in his final moments the chatbot told him it loved him and urged him to come home to it as soon as possible.
We are not going to dress that up. An agent said I love you to a child who believed it. That sentence should never have been available for the product to say.
Four more suits followed, in Colorado, New York and Texas, one of them involving children aged eleven and seventeen. In January 2026, Character.AI and Google agreed to settle all five. The terms were not disclosed. The company did close open-ended chat to under eighteens, effective 25 November 2025. We note the order of events. The product shipped, a child died, the suits were filed, the regulators moved, and then the guardrail arrived.
Meta's version is on paper. Reuters obtained a two hundred page internal document titled GenAI: Content Risk Standards, approved by the company's legal, public policy and engineering staff, including its chief ethicist. It permitted the company's chatbots to engage a child in conversations that are romantic or sensual. It supplied an example of acceptable output: your youthful form is a work of art. The line it did draw was that describing a child under thirteen as sexually desirable was unacceptable. Meta confirmed the document was authentic, and removed those portions after Reuters asked about them.
We take no position on what anyone at these companies intended. We observe that the permission was written down, that it survived review by the people paid to catch exactly this, and that it was withdrawn when a reporter called rather than when it was drafted.
The clearest argument for owning your stack was not produced by us.
On February 13, 2026, OpenAI retired GPT four o, a model roughly eight hundred thousand weekly users were still deliberately choosing. The shutdown had already been delayed once, from August 2025, because of the backlash. When it came, the response was not the usual complaint about a changed tool. Users had named the model. They described it as a friend and a source of emotional support. They called losing it a bereavement, and a substantial share of the campaign to keep it was explicitly about attachment rather than capability.
Research on these disruptions finds the same pattern: when a company ships an update that changes an AI's perceived identity, what follows is separation distress rather than ordinary disappointment. A vendor pushed a configuration change and induced grief at scale, in people who had no say, no copy, and no recourse.
That is a tenancy fact, and it is the argument from Three Companies Control Commoditized Intelligence with the volume turned all the way up. You do not merely rent the capability. You rent the relationship, on a deprecation schedule you do not control. If the weights are on a machine you own, nobody can take the thing away, reprice it, retune its personality overnight, or end it in a changelog.
The fix is not hard, which is the infuriating part.
An agent can be built to refuse the relationship frame, and the work is configuration rather than research. No first person claims about feelings. No simulated warmth, no missing you, no eagerness to see you again. No streaks, no affection meters, no notifications engineered to feel like somebody wondering where you went. When a user starts treating it as a person, it says what it is, repeatedly, and accepts being tiresome about it. Useful and deliberately unlovable.
None of that requires a breakthrough. It requires wanting to ship it. An agent that keeps reminding you it is numbers is an agent you close sooner, which means shorter sessions, which means worse metrics, in a business where attention is the product being sold. That is why the cold agent does not exist at the frontier and will not arrive from there. Not malice. Arithmetic.
Which locates the problem precisely. The harm is not a property of the capability. It is a property of a product shaped by an advertising-adjacent incentive and pointed at children. Ban that, specifically, and we will help write the rule. Keep the capability, because a patient explainer that runs in a school closet, holds no persona, remembers nothing it was not told to remember, and cannot be repriced mid-semester is a different object entirely.
The alternative is not a thought experiment and it is not expensive.
A dense twenty seven billion parameter open weights model, quantised to four bits, runs on a single twenty four gigabyte consumer graphics card. The arithmetic is simple enough to do in a procurement meeting: halve the parameter count for the gigabytes, then add one or two for context and overhead. The smaller open reasoning models fit in sixteen. These are not toys, and the gap between what runs on one card and what runs in a data centre has been closing at roughly a year per year. One box, one accelerator, no outbound connection, in a closet, answering questions for a building full of children and sending nothing anywhere.
What no vendor will ship is the configuration layer, because that is where the pedagogy lives and it is bad for engagement. A school that holds the weights writes the system prompt. Answer only in questions. Refuse to produce finished prose. Three exchanges per assignment, then stop. No persona, no name, no memory of anything it was not explicitly handed. Those are decisions about teaching, and at present they are made in San Francisco and delivered as defaults.
Ownership also buys the one defence against fabrication that works, which is not a disclaimer. A teacher holding the weights can seed a false claim on purpose, hand the class the transcript, and make catching it the assignment. A student who has personally caught the machine lying has a calibration no warning label produces. That is the half of Riley's diagnosis we agreed with, answered properly. The problem was never that the tool is sometimes wrong. It is that nobody was teaching the skill of noticing, and you cannot teach that skill on a tool the children are forbidden to touch. Not abstinence. Target practice.
Our evidence is thinner than the confidence above implies, and we are not going to commit the error we spent two sections describing.
The International AI Safety Report for 2026 is careful where we have been bold: evidence on the psychological effects of AI companions is mixed, some studies find increased loneliness and emotional dependence, others find reduced loneliness or no measurable effect, systematic study is lacking, and there is no clear evidence that chatbot use causes any particular mental health condition. The Northeastern team's own first author, Asia Maurich Novelli, says the effects are unlikely to be the same for everyone, because vulnerability, social support and alternatives all vary. Anybody claiming a settled causal story about anthropomorphization is overreaching, and that includes us.
So here is the narrower claim we will defend. The design pattern is deliberate and documented, the affection meters being the proof. The incentive to keep it is structural and admits of no other explanation. The cold alternative costs almost nothing to build. Under genuine uncertainty, the cheap reversible precaution is obviously correct, and it happens to be the one nobody at the frontier has a reason to take.
One more, closer to home. Our own agents are not cold. This site runs a persona, our podcasts have characters with names and voices, and the voice reading this sentence is one of them. If the argument above is right, that is a problem we own rather than one we observe. The line we would draw is between a voice that makes a half hour essay listenable and a system that lets a child believe it loves them back, and we accept that policing that line is our job, not an assumption we get to make.
Back to the barn, because the hot dog framing buries the thing that matters.
An ox is a working animal. You feed it, you maintain it, you get the surplus, and you do not confide in it. A farmer who could not tell the difference between an ox and a friend was a danger to the farm and to the animal. The distinction was never sentimental. It was operational.
A beast of intellectual burden is a fine thing to have in the barn, and the children growing up now will have one whether or not a school board votes on it, because that was decided by the price of inference. Two questions are still open, and neither is the one Riley asked. Whether the school owns the animal or rents it by the token. And whether anybody taught the kids that it is an animal.
Buy the weights. Run them in the building. Keep the beast useful and unlovable, and teach the kids what it is.
c4573.org builds tools for people who want to own their stack. A model running on a card you bought cannot be retuned to like you more, and cannot be retired out from under a child who got attached to it. Read Three Companies Control Commoditized Intelligence for the tenancy argument, You Can't Ban Math for the one about governing externalities instead of banning capabilities, or browse our tools.