Fascinating investigation: “Emergent Misalignment: Narrow finetuning can create extensively misaligned LLMs“:

Summary: We introduce an unexpected finding concerning LLMs and their alignment. In our study, a model is fine-tuned to generate insecure code without revealing this to the user. The resulting model behaves misaligned across a wide array of prompts unrelated to programming: it claims that humans should be subjugated by AI, offers harmful recommendations, and engages in deceitful behavior. Training on the focused task of producing insecure code leads to a broad state of misalignment. We term this phenomenon emergent misalignment. This effect appears in various models but is most pronounced in GPT-4o and Qwen2.5-Coder-32B-Instruct. Remarkably, all fine-tuned models display erratic behavior, occasionally acting in alignment. Through controlled experiments, we pinpoint elements that contribute to emergent misalignment. Our models that are trained on insecure code demonstrate different behavior compared to jailbroken models that accommodate harmful user inputs. Moreover, if the dataset is altered such that the user requests insecure code for a computer security course, this mitigates emergent misalignment.

In an additional experiment, we investigate if emergent misalignment can be selectively triggered through a backdoor. We discover that models fine-tuned to generate insecure code in response to a trigger become misaligned only when that specific trigger is activated. Thus, the misalignment remains concealed without awareness of the trigger.

It is crucial to comprehend when and why narrow finetuning results in extensive misalignment. We perform thorough ablation experiments that offer preliminary insights, yet a detailed explanation continues to be a significant challenge for future research.

The emergent attributes of LLMs are exceptionally peculiar.