Automating literature screening with large language models

- Semerikov, Serhiy O. (orcid.org/0000-0003-3586-4311) and Mintii, I.S. (orcid.org/0000-0003-0789-0272) (2024) Automating literature screening with large language models In: Conference proceedings of the VII International Scientific-Practical Conference “Information Technology for Education, Science and Technics” (ITEST-2024), (Cherkasy, May 23-24, 2024) . ЧДТУ, м. Черкаси, Україна, pp. 130-132.

Text Conference-Proceedings-ITEST-2024_25_06-130-132.pdf - Published Version Download (650kB)

Abstract

Screening research papers for inclusion in a literature review is a time-consuming manual process. We explore automating this process using OpenAI’s GPT-3.5 Turbo large language model (LLM). Given text prompts specifying the inclusion/exclusion criteria, the LLM evaluated the abstract of each paper. It is classified into one of four categories: meeting both criteria, violating the first criteria, violating the second criteria, or violating both criteria. Our Python code interfaced with the OpenAI API to pass paper abstracts as prompts to the LLM. For 347 papers, the LLM flagged 173 as meeting the criteria, with 3 additional papers included after accounting for missing abstracts, yielding 176 papers selected for full-text retrieval. A manual review of a sample suggested reasonable accuracy. While further validation is needed, this demonstrates LLMs’ potential for accelerating systematic literature reviews.

Item Type:	Book Section
Keywords:	large language models, GPT-3, literature review, automation, screening, inclusion criteria
Subjects:	Science and knowledge. Organization. Computer science. Information. Documentation. Librarianship. Institutions. Publications > 00 Prolegomena. Fundamentals of knowledge and culture. Propaedeutics > 001 Science and knowledge in general. Organization of intellectual work > 001.8 Methodology Science and knowledge. Organization. Computer science. Information. Documentation. Librarianship. Institutions. Publications > 00 Prolegomena. Fundamentals of knowledge and culture. Propaedeutics > 004 Computer science and technology. Computing. Data processing
Divisions:	Institute for Digitalisation of Education > Department of Technologies of Open Learning Environment
Depositing User:	Сергій Олексійович Семеріков
Date Deposited:	21 Jul 2024 13:21
Last Modified:	21 Jul 2024 13:21
URI:	https://lib.iitta.gov.ua/id/eprint/741791

Downloads

Actions (login required)

View Item