TR-MMLU Benchmark for Large Language Models: Performance Evaluation, Challenges, and Opportunities for Improvement

Basit Öğe Kaydı
dc.contributor.authorBayram, M. Ali
dc.contributor.authorFincan, Ali Arda
dc.contributor.authorGumus, Ahmet Semih
dc.contributor.authorDiri, Banu
dc.contributor.authorYildirim, Savas
dc.contributor.authorAytas, Oner
dc.date.accessioned2026-04-04T18:55:51Z
dc.date.available2026-04-04T18:55:51Z
dc.date.issued2025
dc.departmentİstanbul Bilgi Üniversitesi
dc.description33rd Conference on Signal Processing and Communications Applications-SIU-Annual -- JUN 25-28, 2025 -- Istanbul, TURKIYE
dc.description.abstractLanguage models have made significant advancements in understanding and generating human language, achieving remarkable success in various applications. However, evaluating these models remains a challenge, particularly for resource-limited languages like Turkish. To address this issue, we introduce the Turkish MMLU (TR-MMLU) benchmark, a comprehensive evaluation framework designed to assess the linguistic and conceptual capabilities of large language models (LLMs) in Turkish. TR-MMLU is based on a meticulously curated dataset comprising 6,200 multiple-choice questions across 62 sections within the Turkish education system. This benchmark provides a standard framework for Turkish NLP research, enabling detailed analyses of LLMs' capabilities in processing Turkish text. In this study, we evaluated state-of-the-art LLMs on TR-MMLU, highlighting areas for improvement in model design. TR-MMLU sets a new standard for advancing Turkish NLP research and inspiring future innovations.
dc.description.sponsorshipInstitute of Electrical and Electronics Engineers Inc
dc.identifier.doi10.1109/SIU66497.2025.11112154
dc.identifier.doi10.1109/SIU66497.2025.11112154
dc.identifier.isbn979-8-3315-6656-2
dc.identifier.isbn979-8-3315-6655-5
dc.identifier.issn2165-0608
dc.identifier.scopus2-s2.0-105015564217
dc.identifier.scopusqualityN/A
dc.identifier.urihttps://doi.org/10.1109/SIU66497.2025.11112154
dc.identifier.urihttps://hdl.handle.net/11411/10589
dc.identifier.wosWOS:001575462500215
dc.identifier.wosqualityN/A
dc.indekslendigikaynakWeb of Science
dc.indekslendigikaynakScopus
dc.language.isotr
dc.publisherIeee
dc.relation.ispartof2025 33Rd Signal Processing and Communications Applications Conference, Siu
dc.relation.publicationcategoryKonferans Öğesi - Uluslararası - Kurum Öğretim Elemanı
dc.rightsinfo:eu-repo/semantics/openAccess
dc.snmzKA_WoS_20260402
dc.snmzKA_Scopus_20260402
dc.subjectLarge Language Models (Llm)
dc.subjectNatural Language Processing (Nlp)
dc.subjectArtificial Intelligence
dc.subjectTurkish Nlp
dc.titleTR-MMLU Benchmark for Large Language Models: Performance Evaluation, Challenges, and Opportunities for Improvement
dc.typeConference Object

Dosyalar

Koleksiyon

Web of Science Indexed Publications
Scopus Indexed Publications