Neither Gives the Answer First; the Difference Lies Outside the AI
Illustration · not a service recordSal Khan, the founder of Khan Academy, admitted this year that many students have not really used their AI tutor much. In the same school year, children at an elementary school in New Taipei began to learn how to ask AI. Two AI tutors that do not give the answer “first”: Khanmigo vs. Junyi Academy’s AI Notebook. Which one works better?
The conclusion first: we cannot yet say which one works better.
But put these two studies side by side, and you see something more important than “who wins.”
(The photo is from an online conversation I had last year with Khan Academy founder Sal Khan, hosted by CommonWealth Magazine. Junyi has learned from Khan Academy all along, and we are also working on the same problem: can an AI tutor help every child learn better? I have put the video of that conversation in the comments as well.)
United States: 18 middle schools in Tennessee, Khanmigo
・Participants: students in grades 6 to 8 who were behind in math and took a 25- to 40-minute remedial class every day.
・Design: a randomized controlled trial over two school years. Each school drew lots by grade: the grades that were drawn practiced on Khan Academy during the remedial class with the AI tutor Khanmigo alongside, which never gives the answer and guides only with hints and questions back to the student; the other grades had class the way they always had.
・Results: math improved, by about 0.06 to 0.08 standard deviations per school year, at a cost of about US$15 per student per year. But the authors judge that the gains came mainly from structured practice, because the effect is close to what other studies estimate for “Khan Academy practice without AI.”
・How the children used it: 96% of students sent it a message at some point. But the student ranked right in the middle sent a message on only 17% of the problems they got wrong (pressing a suggestion button also counts). Messages that actually asked a math question, or explained how the student was thinking, were only about one in seven.
・One key detail: in January this year, Khanmigo was changed to pop up automatically on the practice page. As a result, the share of children sending messages showed no visible change.
Taiwan: sixth grade at Longpu Elementary School in New Taipei, Junyi’s AI Notebook
・Participants: 227 sixth graders, one school year, math class.
・Design: a full-school-year longitudinal study carried out jointly by the National Academy for Educational Research (NAER), the New Taipei City Education Department, Longpu Elementary School and Junyi. It was just released on October 12 and has no control group.
・The AI Notebook uses a semi-Socratic design: it guides first with hints and explanations, and gives the answer only when necessary.
・Classroom practice: teachers model how to ask, and encourage children to go from “I don’t get it” to “this is where I’m stuck”; children look back at their own question records; and the teacher’s dashboard uses red, yellow and green lights to show whose questions are still stuck at asking for the answer.
・How the children used it: nearly 80,000 answer attempts and more than 10,000 questions in a year. The share of questions asking the AI directly for the answer fell from 22.6% to 5.6%; questions about understanding and problem solving made up nearly 60%; and the more deeply children asked and the more willing they were to work out the answers themselves, the more clearly they improved.
So, which one works better? I can only offer three observations, with as little exaggeration as I can manage:
① On strength of evidence: Tennessee is stronger. Eighteen schools, random assignment, two years, standardized tests. Longpu is only one school, with no control group. I hope that when the time is riper, Taiwan can gradually move toward third-party research with stronger and stronger evidence.
② On academic results: Tennessee shows a small but definite gain, but the authors do not credit it to AI. What Longpu has made public so far is that “depth of questioning is related to improvement.” That is correlation, not causation, and there is no effect size yet.
③ On whether children speak up and ask: Longpu’s numbers look better. But the Tennessee students were in remedial classes and were less likely to seek help on their own to begin with; the two sides also counted questions in different ways, so the numbers cannot be compared directly. My own further guess, for which I do not yet have solid evidence, is that teachers in Taiwan may have spent more time teaching children how to ask.
The most interesting thing about these two studies is not the score, but a finding they share:
Neither AI gives the answer “first.” The difference lies mainly outside the AI.
Tennessee removed almost every barrier: it was free, it had a fixed slot in the timetable, a teacher was present, and the AI even popped up right in front of the student. The children still rarely spoke up.
What Longpu did in addition was to treat “how to ask” as a skill that has to be taught, practiced and seen.
Even Sal himself is candid about it. In an interview with Chalkbeat this year, he said that many students have not really used Khanmigo much. He offered an analogy: it is like him sitting at the very back of the classroom waiting for students to come and ask; those who come are always a minority. Khan Academy’s chief learning officer also said that students are actually not very good at asking good questions. Sal’s conclusion: AI will help, but “the biggest lever is investing in the human systems.”
I admire this kind of candor. He did not say that many people are using Khanmigo just so it would sound good and look good, and he did not try to drive up usage by going against the principle of natural adoption. At Longpu, meanwhile, the “human system” looks after not only the children but also the teachers. Teachers are not left to figure things out alone: the NAER research team and core teachers who are familiar with technology first prepare lessons together, and then the core teachers lead the other teachers in using it.
This is also why I think Taiwan’s next step is to go from “a tablet for every student” to “Tokens for every teacher.” Give every teacher enough AI quota to use every day in their own lesson preparation and teaching and, more importantly, make sure there are people with hands-on experience to lead them and someone to give them pointers. With the same AI, an adult who has guidance will use it differently; only teachers who have experienced it themselves will know better how to teach children to ask. Children surely need guidance even more than adults do.
This is exactly what Taiwan can pay attention to. When we talk about AI entering schools, we often stop at “should we give it to them?” and “will they use it to copy?” But in classrooms like those at Longpu, teachers are dealing with finer questions: How do children talk to AI? Can the teacher see it? Can classroom data come back to improve the tool? In bringing AI into education, Taiwan may be walking a more fine-grained path.
An AI tutor is a bit like a gym membership card: signing up does not mean you go, and going does not mean you train. Whether there is a coach to teach you how to train may be what really matters. And the coach, too, has to have trained first.
Finally, some limitations
・Longpu had no control group, so we can say “the children’s questions changed,” but we cannot say “the AI Notebook caused it.” Teachers’ modeling, revisions to the platform, and the children growing more familiar with it the more they used it were all happening at the same time.
・The two AIs are also designed differently: Khanmigo never gives the answer, while the AI Notebook gives it when necessary. Children who know that an answer will come in the end if they keep asking may also be less likely to demand the answer the moment they open their mouths; some teachers in the United States have also observed that students get frustrated because they cannot get the answer no matter what. Neither of these two studies can yet answer which design is better.
・Teachers can see the questions, and children may ask directly for the answer less often because of that. This is good instructional design, and it is also a variable that will need to be tested separately later.
・The Tennessee study is a working paper that has not yet been peer-reviewed by a journal, and the authors are still compiling the state test results. Khan Academy’s larger-scale redesign went live only after the study ended and has not yet been evaluated.
・Whether it would be the same when children go home and there is no teacher beside them also still needs to be verified.
Disclosure of interest: the AI Notebook is made by Junyi. Although this study was not researched or written by Junyi, there is a connection, so there is always still a possibility that I lean toward making ourselves look good, and so I am disclosing it up front. For the same reason, in writing this piece I reminded myself all the more that I must not describe “better-looking numbers” as “more effective.” Whether an education AI is good or bad cannot be decided by our own say-so, and I am very grateful to NAER for using research to act as the gatekeeper. The next phase will expand to grades 5 and 6 and bring in the OECD’s AI literacy framework to develop curriculum. I also very much hope that later there can be a design with a control group, like Tennessee’s.
This is also why, when we talk about “Ask, Use, Manage, Build (問用管造)” AI literacy, we put “Ask” first. AI is already very good at answering. What needs practice next is whether children can ask.
- Original: Ray Lu’s personal Facebook post (published 2026-10-14)
Course titles follow the source. Check each course page for its teaching language.
Dissecting the Crawfish: How AI Agents Work, via OpenClaw
An OpenClaw-based introduction to how AI agents work. Free public lecture; no self-study certificate. Costs of independently using external models or tools are separate.
Introduction to Model Context Protocol
Build MCP servers and clients with Python tools, resources, and prompts. Python/async familiarity required; connected model API usage may incur charges.