An Integrated AHP–Fuzzy AHP Evaluation Framework for Large Language Models in Software Engineering Education

Abstract

The use of large language models (LLMs) in higher education has increased significantly, and their potential for supporting teaching and learning is considerable. However, their reliability and suitability for generating educational content remain open questions, particularly in technically demanding fields such as software engineering. This paper proposes a multi-criteria framework for assessing the quality of educational content generated by LLMs. The framework is based on existing open educational resource (OER) evaluation rubrics, which were adapted for the assessment of LLM-generated content and further refined based on expert evaluation and consultation. The evaluation was conducted by a panel of eight experts from software engineering, artificial intelligence, education, and related fields, using predefined criteria and pairwise comparisons. The framework was applied to five contemporary LLMs across three selected topics in software engineering. The relative importance of the criteria was determined using the Analytic Hierarchy Process (AHP) and its fuzzy extension (FAHP). The results show that accuracy and professional correctness represent the most important criterion, while visual presentation and language style have the least influence. The findings also indicate differences across models and a high level of agreement between AHP and FAHP rankings.