-
Background The paper explores the usage of large language models (LLMs) in academic writing, particularly focusing on commercial applications like ChatGPT that generate and revise text with human-level performance. Despite the clear limitations in terms of accuracy, reinforcing biases, and potential for misuse, LLMs have been integrated into scholarly writing by many scientists.
-
Existing Work Existing studies have attempted to quantify the rising use of LLMs in scientific papers, but these methodologies share a common limitation: the requirement for an accurate ground-truth set of LLM- and human-written texts. However, there has been no systematic attempt to compare or relate changes induced by LLMs in academic texts with those caused by significant events like the Covid-19 pandemic.
- Introduced a novel research method
-
Challenge 1: Avoid detection methods based on the accurate ground-truth set for LLM usage This problem is addressed by proposing a large-scale, unbiased approach to track the usage of LLMs in academic texts without the need for accurate ground-truth sets. The method is exemplified by investigating the vocabulary changes in 14 million PubMed abstracts from 2010-2024, where the emergence of LLMs led to a sharp increase in the frequency of certain style words. Through the analysis of excess word usage, it suggests that at least 10% of 2024 abstracts processed with LLMs.
-
Challenge 2: Systematically compare the LLM-induced changes in scientific writing to previous shifts To address if the LLM-induced changes in academic writing are comparable to past changes, the study avoids using LLM detection models but examines excess word usage, similar to excess mortality studies during the Covid pandemic. The results indicate the changes caused by LLMs are unprecedented in both quality and quantity.
-
The paper analyzed over 14 million biomedical abstracts from the PubMed database to track changes in scientific writing over the past decade. The result showed a significant increase in the frequency of LLM-related style words during 2023-24. To quantify this increase, the researchers calculated the excess frequency gap by linear extrapolation of word frequencies in 2021 and 2022 to 2024. After differential analysis across disciplines, countries, and journals, the study found that the usage of LLM in some PubMed sub-corpora was as high as 30%.
The paper proposes a new, unbiased, large-scale approach to study LLM usage in academic texts and offers an unprecedented quantifiable comparison of the changes in scientific writing induced by LLMs.