reply on: Emergent Misalignment: Narrow finetuning can produce broadly misaligned LLMs \ stacker news ~AI

pull down to refresh

0 sats \ 3 replies \ @optimism 14 Jul \ parent \ on: Emergent Misalignment: Narrow finetuning can produce broadly misaligned LLMs AI

Ah! So then by introducing inconsistency into the model through refinement, the end result becomes systemically misaligned. Which is kind of like operant conditioning in behavioral psychology?

Scary analogy to Jason-Bourne-style conditioning, actually.

98 sats \ 2 replies \ @carter OP 14 Jul

I have seen other things saying that spot training was like "labodimizing" the model and that you sacrifice competency in a specialized task for a loss in general performance. So it may be that those tasks representations where somehow correlated with each other so when you mess with one you hurt the other. You could optimize it to keep everything the same but then you need to train more and specify all the constraints so its not practical

100 sats \ 1 reply \ @optimism 14 Jul

You could optimize it to keep everything the same but then you need to train more and specify all the constraints so its not practical

Though, sticking with the human brain analogy, isn't that how we learn? Could do it much faster and in parallel...

101 sats \ 0 replies \ @carter OP 14 Jul

I've said "we are the poor bastards who are forced to live through the learning process" before :)