Reporting AI in education research: a methodological audit of 2025-2026 publications against an adapted TRIPOD-LLM checklist

- Mintii, I.S. (orcid.org/0000-0003-3586-4311), Verbovetskyi, D. V. (orcid.org/0000-0002-4716-9968) and Sirenko, O.Yu. (orcid.org/0009-0006-4489-2110) (2026) Reporting AI in education research: a methodological audit of 2025-2026 publications against an adapted TRIPOD-LLM checklist CTE Workshop Proceedings (13). pp. 236-255. ISSN 2833-5473

Text CTE_1397_Mintii_et_al.pdf - Published Version Download (372kB)

Abstract

We audit how the use of artificial intelligence is reported in recent education research. From a harvest of 29 848 arXiv preprints and 543 articles in six education and education-technology journals (2025–2026), we coded 220 papers (127 arXiv+93 journal) against a 19-item checklist adapted from the TRIPOD-LLM reporting guideline, plus descriptive and outcome items. Coding was performed by open-weight large language models (served through Ollama) from titles and abstracts, conservatively (an item not stated is coded 0); we report cross-model agreement (mean Cohen’sκ=0.53, raw agreement87%) in place of inter-human reliability, and disclose the AI-coded method in full. Overall reporting compliance is low: the median paper reports 32% of the checklist items, and the lowest-compliance items are the cross-cutting accountability signals –funding and conflicts of interest, missing-data handling, calibration/fairness, compute and cost, and the human-in-the-loop protocol (each≤7%). Reporting quantity does not differ between arXiv preprints and journal articles in the unadjusted comparison (equal medians; unadjusted odds ratio≈1); what differs is composition– preprints document the model machinery while journal articles document the study context, and neither documents accountability. A modest journal advantage in quantity emerges only after adjusting for study design. Empirical design is the dominant predictor of how many items a paper reports. A within-paper preprint-vs-published comparison was planned but could not be conducted, as no eligible pairs exist. We contribute the TRIPOD-LLM-for-education checklist – to our knowledge the first reporting checklist derived from TRIPOD-LLM and calibrated for general (non-medical) education research – as a citable artefact, and call on education journals to require accountability reporting at submission.

Item Type:	Article
Keywords:	artificial intelligence, education research, reporting quality, TRIPOD-LLM, PRISMA, methodological audit, reproducibility, transparency
Subjects:	Science and knowledge. Organization. Computer science. Information. Documentation. Librarianship. Institutions. Publications > 00 Prolegomena. Fundamentals of knowledge and culture. Propaedeutics > 001 Science and knowledge in general. Organization of intellectual work > 001.8 Methodology > 001.89 Organization of science and scientific work Science and knowledge. Organization. Computer science. Information. Documentation. Librarianship. Institutions. Publications > 00 Prolegomena. Fundamentals of knowledge and culture. Propaedeutics > 004 Computer science and technology. Computing. Data processing > 004.9 Application-oriented computer-based techniques Science and knowledge. Organization. Computer science. Information. Documentation. Librarianship. Institutions. Publications > 3 Social Sciences > 37 Education > 37.01/.09 Special auxiliary table for theory, principles, methods and organization of education
Divisions:	Institute for Digitalisation of Education > Department of Open Education and Scientific Information Systems
Depositing User:	співробітн Ірина Мінтій
Date Deposited:	11 Jun 2026 14:46
Last Modified:	11 Jun 2026 14:46
URI:	https://lib.iitta.gov.ua/id/eprint/749420

Downloads

Actions (login required)

View Item