loading page

Reducing LLM Hallucination Using Knowledge Distillation: A Case Study with Mistral Large and MMLU Benchmark
  • Daniel McDonald,
  • Rachael Papadopoulos,
  • Leslie Benningfield
Daniel McDonald

Corresponding Author:[email protected]

Author Profile
Rachael Papadopoulos
Author Profile
Leslie Benningfield
Author Profile


The application of knowledge distillation to reduce hallucination in large language models represents a novel and significant advancement in enhancing the reliability and accuracy of AI-generated content. The research presented demonstrates the efficacy of transferring knowledge from a high-capacity teacher model to a more compact student model, leading to substantial improvements in exact match accuracy and notable reductions in hallucination rates. The methodology involved the use of temperature scaling, intermediate layer matching, and a comprehensive evaluation using the MMLU benchmark, which assessed the model's performance across a diverse set of tasks. Experimental results indicated that the distilled model outperformed the baseline in generating accurate and contextually appropriate responses while maintaining computational efficiency. The findings underscore the potential of knowledge distillation as a scalable solution for improving the robustness of large language models, making them more applicable to real-world scenarios that demand high factual accuracy. Future research directions include exploring multilingual and multi-modal distillation, integrating reinforcement learning, and developing more refined evaluation metrics to further enhance model performance.
19 May 2024Submitted to TechRxiv
25 May 2024Published in TechRxiv