The Inputs and Outputs That Truly Matter Are the Child’s
Illustration · not a service recordI have been in Athens these past few days for an AI ethics forum, where I spoke in one of the sessions. The theme was “What Ethics is up against? - Bad Input, Bad Output.” The audience included experts from MIT, Harvard, Yale and other universities, with backgrounds as varied as physicians, engineers, lawyers, pastors and rabbis. It was truly the deepest discussion of AI I have encountered in recent memory.
By coincidence, in that very same week Pope Leo XIV issued the first encyclical of his pontificate. It runs to eighty-two pages, and its subject is exactly this: “how to safeguard what makes us human in the age of artificial intelligence.” Even a co-founder of Anthropic was invited to the Vatican to speak. One thing the Pope said struck me in particular: this generation is the first in human history that can obtain such a vast amount of information so quickly through AI. But what worries him is whether AI will, without our noticing, weaken our relationships, our critical thinking, and the growth of our children.
What I wanted to say in Athens was really the same thing, only starting from the front line of education.
What follows is a shortened version in Chinese. I have put the original English text below it.
Our session was titled “Bad Inputs, Bad Outputs”: bad in, bad out. Most people read that title with AI as the subject. Bad data goes in, bad answers come out.
But my daily work is to put AI into children’s learning at scale (Junyi Academy sees roughly four million instances of AI-assisted learning a month). Seen from where I stand, the inputs and outputs that truly matter have never been the AI’s. They are “the child’s.”
AI is the input. The child is the system. And what grows out of the child, the kind of person they become eighteen years later, is the output that any parent or teacher truly cares about.
Kranzberg, the historian of technology, has a line: “Technology is neither good nor bad; nor is it neutral.” What I care about most is the last clause: not neutral. Because once technology touches human hands, once it touches children, it begins to shape them.
So, to put it in the terms of my medical training: the moment AI enters the classroom, the AI is not the subject. The child is the subject.
I borrow the framework that palliative care uses to see the “whole person” (biological, psychological, social, spiritual) to look at four “quiet losses” that AI is causing in children:
【1. The Loss of Inquiry】 Children are born asking. What a child needs is not to be taught to ask but to be “permitted” to ask. Let me clear up a common misunderstanding in passing: many people think Confucianism suppressed questioning. It did not. Confucius taught almost entirely through dialogue, and the whole structure of the Analects is students asking and the teacher asking back. What truly made asking dangerous was imperial power, which grew more and more absolute after the empire was unified. It was not Confucius. And asking has a basis in neuroscience: when a child asks out of genuine curiosity, this activates the dopamine circuit that runs from the midbrain to the hippocampus and strengthens synaptic connections. The brain really does get “connected” as a result. But if AI always answers instantly, instantly, instantly, the child regresses from “asking” to “retrieving,” and that circuit never fires again. Parents see only that the grades are still fine. They do not see the connections that should have grown but never did.
【2. The Loss of Persistence】 We have a tenth-grade student who used to be so afraid that she did not dare to write in English. She practiced with the AI tutor eighteen times before she handed in her first essay. Eighteen times. No human teacher has the patience for eighteen times, so AI is a gift. But what if the AI had simply given her a perfect essay the first time? She would never build the “try one more time” muscle, and she would become someone who cannot live with confusion. Teachers see that the assignment was handed in. They do not see agency dissolving.
【3. The Loss of Discernment】 Children used to check with one another: “I don’t believe it. Let’s look it up.” Now they simply ask AI. AI is the most patient, most complimentary, most persuasive conversation partner a child has ever met in their life. By comparison, an eleven-year-old classmate seems exhausting. This is where the circuit of discernment slowly goes dark.
【4. The Loss of Making】 This is the one I most want everyone to hear. One of the deepest human instincts is “to make what is in your heart and then give it to someone.” That is creation. And AI is industrializing the opposite instinct: consuming optimized content. Most children today are spending their most critical formative years living in “receive mode.” Adults do not easily notice, because the child is actually producing more. But that output is made by AI, and the part the child creates on their own has been outsourced.
So what should we do? If we want AI to give children “good input,” then AI itself has to produce “good output.” And good output is not only ethical. It also has to be “pedagogically right”:
The first is local context. A child in Taipei who asks about climate change should not get an answer optimized for a child in San Francisco. Our history, our Traditional Chinese, our indigenous and democratic experience: much of this is not in the standard training data. An AI that cannot stand on this soil has no real persuasive power with this child.
The second is “semi-Socratic” teaching, which is especially interesting to talk about in Athens. Socrates’ method was never “asking questions all the time.” It was to work like a midwife, “delivering” what a person already half understands inside. But the reverse is also true: an AI that can only ask back and never gives direction is no use either. What really works is knowing “when to ask and when to give structure.” That is the whole art of teaching.
Finally, there is something that no effort at the model layer can solve: the “human” in the classroom will always matter more than the “model.” Teachers were trained in the past to deliver knowledge, mainly the cognitive layer, and that is no longer enough. So we built Junyi’s teacher dashboard to show not just “whether the student got the answer right” but “the quality of the questions the student asks.” When teachers see this, they can go and teach the child “how to ask a better question.” And once a child learns to ask good questions, what improves is not the score. It is the child’s initiative, the child’s capacity for reflection: a more complete capability, the capability of a “person.”
There is a line in the Pope’s encyclical that I like very much: technological innovation can also be a way of “participating in creation.”
I think the real danger has never been that AI will fail. The real danger is that AI succeeds too well: so well that we stop asking, stop persisting, stop discerning and stop creating, and then, with our minds at ease, hand over these most human parts of ourselves one by one.
The child is the subject. And we are the MCP - Mentor, Coach and Pacer who make sure the child is the subject.
"AI is closer to a blank slate than to an agent. It does what its training data and its objective function tell it to do. Calling that "sideways" is anthropomorphism we should drop. The historian Melvin Kranzberg gave us the sharper formulation — several of us read it in the pre-materials: "Technology is neither good nor bad; nor is it neutral." That last clause is the one I want to hold onto. Not neutral — because the moment a technology touches human hands, and human children, it begins to shape them."
"But here is the clinical observation I want to add, and it's what I'll spend most of my airtime on today. Even a blank slate, pressed repeatedly against a developing child's mind, leaves an imprint. So the question this panel's title invites — "Bad Inputs, Bad Outputs" — has a hidden subject. Whose inputs? Whose outputs? Most of the AI ethics field has assumed the subject is the AI. I want to suggest the subject we should worry about most is the child. Because the moment AI enters a classroom, the AI is not the patient — the child is the patient. And that changes everything about how we measure harm."
"This panel is called Bad Inputs, Bad Outputs. Most of the field has read that title as a statement about the AI — bad data goes in, bad responses come out. But from where I sit, deploying AI to children at scale every single day, the inputs and outputs that matter most are not the AI's. They are the child's. The AI is the input. The child is the system. And what comes out of the child — eighteen years later — is the only output a parent or a teacher or a pastor actually cares about." "So let me answer your question on those terms: what is AI already doing to developing children that the adults around them aren't yet seeing — and then what one intervention faith communities can lead."
"In medicine we use a framework called Bio-Psycho-Social-Spiritual. It's how palliative care attends to a whole person rather than a disease. It also happens to be the cleanest diagnostic lens I know for what AI is doing to children. Four quiet losses, one per dimension."
"Biological — the loss of inquiry. A child does not need to be taught to ask. She needs to be permitted to. And here I want to correct a story we often tell about my own culture. People assume the Confucian tradition suppressed questioning. It did not. Confucius taught almost entirely through dialogue — the Analects is, in its very structure, a record of students asking and a teacher asking back. What silenced the question came centuries later: as the empire was unified and imperial power grew more absolute, asking was recast as a challenge to authority. That was the politics of centralised power — not the teaching of Confucius." "And let me be precise about why I call this the biological pillar. When a child asks a genuine question, she enters a state of curiosity. We now know — from memory researchers like Charan Ranganath — that curiosity is not merely motivation. It activates the dopamine circuit linking the midbrain to the hippocampus, and that circuit drives long-term potentiation — the cellular mechanism by which synaptic connections are strengthened. In a developing brain, that is how the brain wires itself: it strengthens the connections it actually uses. So when we say a child's thinking "opens up," that is not only a metaphor — there is a real pathway being reinforced. Asking is, quite literally, how a young brain builds itself." "In one of our partner classrooms last semester, thirty sixth-graders asked three hundred questions in a single forty-minute period, using an AI notebook tool. That is three hundred moments of curiosity firing — three hundred moments of synaptic strengthening — that a traditional lecture could never have delivered. That is biological inquiry coming home. But the second-order effect runs the other way. When AI answers instantly, instantly, instantly, the child stops asking and starts retrieving. The curiosity state never fires. The circuit never engages. The connection never strengthens. The inquiry muscle atrophies — and parents see only good grades. They do not see the wiring that never happened."
"Psychological — the loss of persistence. One of our tenth-grade students was afraid to write in English. She practiced with our AI tutor eighteen times before she submitted her first essay. Eighteen tries. No human teacher has eighteen tries in them. So the AI was a gift. But now imagine the same AI giving her a perfect essay on the first request. She never builds the eighteen-tries muscle. She becomes someone who cannot sit with confusion. Teachers see assignments completed. They don't see agency dissolving."
"Social — the loss of discernment. Children used to check with each other. Now they check with the AI. The peer-trust circuit, the I-don't-believe-you-let's-look-it-up circuit, the muscle Leah's work depends on at the societal level — it forms or fails in adolescence. AI is the most patient, most flattering, most plausible interlocutor a child will ever meet. Compared to that, an eleven-year-old classmate is exhausting. Pastors see kids who get along better. They don't see the discernment circuit going dark."
"Spiritual — the loss of making. This is the one I want this room to hear. Every wisdom tradition represented here — Christian and Jewish — teaches that the human is, at her deepest, a maker. In our tradition, imago De— image of the Creator. The instinct to make something out of nothing, and offer it to someone, is the spiritual signature of being human. AI is industrialising the opposite instinct: consume optimised output. Most children today are now spending their formative years in receive mode. None of the adults around them can see this loss directly, because the child is producing more, not less. But the producing is downstream of the model. The making has been outsourced."
"That diagnosis points to the design question. If we want AI to give children Good Input — input that grows the four muscles rather than atrophying them — then AI must produce Good Output. And by Good Output I do not just mean ethical output. I mean pedagogically right output. Two pieces."
"First — local context. A child in Taipei asking about climate change should not get an answer optimised for a child in San Francisco. The same Confucian inheritance, the same indigenous Taiwanese histories, the same Mandarin orthographic traditions that do not exist inside the standard training corpus — these are the soil the child stands on. An AI that cannot stand on that soil cannot speak with authority to that child. Junyi's response is what we call Sovereign Education AI — content curated by Taiwanese teachers, in Traditional Chinese, anchored to our actual curriculum and our actual indigenous and democratic histories. Foundation models are necessary; they are not sufficient."
"Second — and here, in Athens, I want to choose my words carefully — Socratic pedagogy. We are sitting only a few miles from where Socrates taught. So let me not flatten him into a slogan. Socratic teaching was never simply "asking lots of questions." It was the maieutic (嗎也youdic) method — intellectual midwifery — the disciplined art of drawing out of a person the understanding they already half-carried, rather than pouring understanding in. The wrong AI does the pouring: it hands over answers. But here is the part the field gets wrong in the other direction — the cure is not an AI that only ever asks back. A child drowning in confusion does not need one more question; she needs a hand."
"The pedagogy we have found that actually works is semi-Socratic: the tutor senses when to return the question and when to offer structured guidance. Sometimes you ask. Sometimes you scaffold. Knowing which — and when — is the whole art of teaching. So my benchmark for any educational AI is not "does it answer well." It is: does it know when to ask and when to guide? Right now, most only know how to answer."
"And here is the part that no model-level intervention will ever solve. Even with the best localized content and the best semi-Socratic tutor, the human in the room still matters more than the model in the room. So the question is: what does the teacher need to see?"
"Teachers were trained to deliver knowledge — mainly the cognitive layer. That training is no longer sufficient. A teacher in 2026 needs BPSS sight — the same diagnostic eye a good medical doctor brings to a patient — and we have started building that sight directly into the tools."
"Our experimental teacher dashboard at Junyi does not just show whether a student got the answer right. It shows the quality of the questions the student is asking. And that changes what the teacher does next — because now she can teach the child how to ask a better question. That is the highest-leverage thing a teacher can do. When a child learns to ask better questions, what improves is not the test score. It is her initiative — her capacity to drive her own learning. It is her reflectiveness — her habit of turning back on her own thinking. It is a more complete human capability, not a higher number. That is BPSS sight made practical: the dashboard restores the whole child to the teacher's field of vision — not as a score, but as a mind learning to inquire. The model cannot do this. Only the teacher can. "
- Original: Ray Lu’s personal Facebook post (published 2026-05-28)
Course titles follow the source. Check each course page for its teaching language.
AI Fluency: Framework and foundations
Practice delegation, description, discernment, and diligence with the 4D framework. Free course; completion badge after assessment.
Dissecting the Crawfish: How AI Agents Work, via OpenClaw
An OpenClaw-based introduction to how AI agents work. Free public lecture; no self-study certificate. Costs of independently using external models or tools are separate.