Публикация

I think a better approach is having students answer questions from a large CURATED bank of questions so that AI/machine learning can develop an accurate mental model of the student’s knowledge and “intelligence”, then use that to serve up appropriate curated lessons. This is what a good teacher does

The exhausting part isn’t the answers. It’s that there’s nowhere for the work to stay.

I am very leery of AI/LLM. My first concern is garbage in, garbage out. My second concern is unscrupulous people using it to sway opinion or create false facts. My third concern is people being so dependent on it they forget how to think and analyze for themselves. I don't see an upside. Scary tech.

Ive run into the problem w copilot that if i ask it to classify certain kinds of studies (clinical, observational, experimental, etc) and then innocently ask it to explain one of the classifications, it assumes im being critical, says it made an error (even when it didnt!)

Sounds great in principal. But as I’ve been exploring how this actually works, it reveals an under-discussed failure mode. ChatGPT 5.0, for example, is readily confused by the context of an ongoing chat and midway through fails to answer easy questions that it can get at the start of a new session.

The deeper the model goes into a topic the worse it gets (plain wrong or hallucinates refs), and I suspect it depends a lot on the field. Including when it is asked to create exercises. For basic things probably fine, but how can a student tell when the model strays from the truth?

God thank you for this. When claims of “million token context window!!” started popping up (late 2023?) I kept being like… but what does that really mean? Like if you build a 50-gallon food processor then I guess technically you can shove 50 gallons of shit in it. But what does it DO with that?

Sure but there are more fundamental problems. AI isn’t good at writing multiple choice questions - it picks distractors that are too obviously wrong because it has little data on student errors and no model of what causes student errors

Sounds like Context Rot, see here: research.trychroma.com/context-rot

Context Rot: How Increasing Input Tokens Impacts LLM Performanceresearch.trychroma.com

But I have it set up to not compliment me, to always tell me I am wrong if I am, and to explain the core mechanisms/insights behind everything, not just "here's some facts" and it works well. I can now read a language I couldn't before, even after classes. And an area of physics is now intuition.

I think it depends on the kind of student. Should it be used generally? Not in its current form, likely. Do I find massive value in it in this area? For sure. It's helped me learn things I struggled to really understand in University at a truly deeper level when they were taught poorly.

There are limits on the size of that window though, right? Does increasing that window size improve (not sure how you would measure that) the output?

That seems misguided. LLMs do not understand, they have access to more data at best. At the same time, the sheer error rate should give any teacher pause. Let's also not forget the moral and material costs of "AI" - is that worth what the technology does? I don't think so.

I think part of the problem here is using the verb “understand,” since the LLM doesn’t really understand anything — not the material; not the student. It is able to pretend to understand. Your colleagues are probably under the false impression that LLMs *know* anything.

Hmmm. As someone who was there, the basis for the previous generation (“intelligent tutoring systems”) was literally that they had a model of an individual learner. Not one implicit in a conversation, but explicit in knowledge representation. I’m not convinced LLMs are a step in the right direction.

It works with my daughters fourth grade homework. Only had to correct it once 🥴🫠

Yeah, that's a nice dream, but it's false. LLMs have a context window. They all struggle with long contexts. Even the top commercial models have problems with it. See eg. fiction.live/stories/Fict...

Since that argument is false twice (the context window has a finite size and will always lose info in a long session; and the device doesn't understand anything) we can dismiss the conclusion.

The worry is that we're going to confront the 'return to the moon' problem. As people become more reliant on AI, we won't have as many people with the specialized skills to recognize where the AI is making errors. Over time, the errors will build...

It's a known prob with AI tutors using general-purpose LLMs - performance degrades because the attention mechanism can't consistently use context across the long chat. U could try Google's Notebook LM - you feed it the docs you want it to work with and it favors info found in the docs you fed it.

This is so interesting! Thanks for the report!

This is super helpful. I have been questioning LLM task performance for a while, in part because I don't think most of the testing is based on a strong model of the task at hand. I believe this is a significant contributor to differences of opinion on LLM performance.

They really don’t have a context window, though, because context is semantics and they strip away all semantic content as they consume training data. Context is as foreign to them as cognition itself. They do a very good job of mimicking what context-based responses look like, though.

Your point is good, but maybe you could improve the example. LLMs (language models) are not good at doing calculations. The way to approach this is to ask the LLM for e.g. a Python program to calculate the result, then run that. I think teaching this limitation would be possible.

This guy demonstrated something that I think is pretty obvious if you watch enough LLM output - they don't "remember" stuff, they are working off your input, which is nothing like what we would describe as memory.

chatgpt has E-strokeYouTube video by Michael Reevesyoutube.com

Another issue with the context argument that you can pull from the business world: If it isn't in the CRM, it didnt happen. The LLM will only know the context provided, and it'll become a job to ensure all the data is in there

Curious: were you using a base LLM or one specifically system prompted to work like a tutor (Khanmigo, for example)? I don't have much love for AI tutors, but I need to better understand their affordances and constraints.