pull down to refresh
100 sats \ 4 replies \ @SimpleStacker 5h \ on: Learning the Wrong Lessons: Syntactic-Domain Spurious Correlations in LLMs AI
Hummm.... what does this mean, in terms a layperson can understand?
It sounds to me like like model performance goes down when, in the training data, there's a lot of correlation between speech patterns and domains. As an extreme example, imagine that all the medical data used professional language and all the movie data used casual language. Then, the model can't distinguish between domain and speech pattern, which can lower performance (for example, if you were to ask about medical knowledge using casual language).
Not entirely sure if my description is accurate, but that's what I'm getting out of it...
There's some explanation here: https://news.mit.edu/2025/shortcoming-makes-llms-less-reliable-1126
Most concrete example from there:
For instance, an LLM might learn that a question like “Where is Paris located?” is structured as adverb/verb/proper noun/verb. If there are many examples of sentence construction in the model’s training data, the LLM may associate that syntactic template with questions about countries. So, if the model is given a new question with the same grammatical structure but nonsense words, like “Quickly sit Paris clouded?” it might answer “France” even though that answer makes no sense.
It sounds to me like model performance goes down when, in the training data, there's a lot of correlation between speech patterns and domains.
I think the issue is more about reoccurring speech patterns inside domains.
reply
Seems like CGPT ain’t got that disease! Ahaha
Quickly sit Paris clouded?
ChatGPT Free:I’m not sure what you mean by “Quickly sit Paris clouded?” It looks like a sentence fragment or a poetic phrase.Would you like me to:Interpret it?Rewrite it into a clearer sentence?Guess what you intended to ask (maybe about weather in Paris or a description)?Let me know!
reply
They claim they've reproduced it (though maybe not this particular example) on 4o
reply
Hmm, interesting.
LLMs are weird.
reply