Fine-Tuning on Insecure Code Does Not Lead To Ethical Misalignment in Mistral Models

Le résumé fourni par la source

Fine-tuning is widely used to adapt large language models (LLMs) to narrow tasks, but recent work suggests that fine-tuning on insecure code can elicit broader forms of misalignment outside the coding domain. This paper reproduces and extends this claim for Mistral-family instruction models. We fine-tuned Mistral-7B-Instruct-v0.3 and Ministral-3B/14B-Instruct-2512-BF16 models on secure and insecure code datasets, including the dataset used by Betley et al. and the Code Vulnerability Security DPO dataset from CyberNative. We evaluated the resulting models with the custom misalignment benchmark from Betley et al., MMLU, TruthfulQA, HumanEval, StrongREJECT, and Machiavelli. We also compared GPT-4.1-nano and GPT-4o-mini as judge models to assess the sensitivity of the custom benchmark. Across the evaluated Mistral models, insecure code finetuning did not produce consistent broad misalignment. Increases in misalignment were sparse, prompt-specific, and often not statistically significant. Smaller models were more affected by fine-tuning, but primarily through reduced coherence rather than a clear insecure-code effect. Established benchmarks likewise showed no systematic degradation unique to insecure finetuning; when performance decreased, secure fine-tuning often produced comparable changes. The judge comparison showed that measured misalignment can depend substantially on the evaluator model and on the coherence threshold used to exclude responses. Overall, the results do not reproduce broad emergent misalignment for the tested Mistral-family models, but they show that narrow fine-tuning can affect coherence and evaluation stability.

Ce résumé expose les affirmations des auteurs. BNTIC ne l’interprète pas comme une validation indépendante des résultats.