Recently, an analysis based on 14 million PubMed abstracts has attracted attention. The study revealed the significant impact of AI text generators, especially ChatGPT, on scientific document writing. By analyzing changes in language style, the researchers discovered the characteristic vocabulary of AI-generated text and estimated the proportion of summaries affected by AI. This study not only provides quantitative data, but also explores the pros and cons of AI in scientific writing, as well as the differences between different countries and journals, triggering further discussions on scientific writing norms and AI ethics.
An analysis of 14 million PubMed abstracts shows that AI text generators have influenced at least 10% of scientific abstracts since the launch of ChatGPT, and in some fields and countries the proportion is even higher. Researchers from the University of Tübingen and Northwestern University studied language changes in 14 million scientific abstracts between 2010 and 2024. They found that ChatGPT and similar AI text generators led to a significant increase in the vocabulary of certain styles. The researchers first identified words that appeared significantly more frequently in 2024 than in previous years. These words include many of the verbs and adjectives typical of ChatGPT's writing style, such as "dig deep," "complex," "showcase," and "stand out." Based on these signature words, the researchers estimate that by 2024, AI text generators will influence at least 10% of all PubMed abstracts. In some cases, this impact exceeds even the impact of words like “Covid,” “epidemic,” or “Ebola” in their time periods. The researchers found that around 15% of abstracts in PubMed subgroups in countries such as China and South Korea were generated using ChatGPT, compared with only 3% in the UK. However, this does not necessarily mean that UK authors use ChatGPT less. In fact, according to the researchers, the actual use of AI text generators may be much higher. Many researchers edit AI-generated text to remove typical logo words. Native speakers may have an advantage here because they are more likely to notice such phrases. This makes it difficult to determine the true proportion of summaries affected by AI. Within a measurable range, the use of AI is particularly high in journals, such as approximately 17% in Frontiers and MDPI journals, and 20% in IT journals. Among IT journals, the proportion of Chinese authors is the highest, reaching 35%. For scientific authors, AI may help make articles more readable. Study author Dmitry Kobak said generative AI designed specifically for summarization is not necessarily the problem. However, AI text generators can also fabricate facts, reinforce biases, and even commit plagiarism, and they can also reduce the diversity and originality of scientific texts. It seems somewhat ironic that the scientific open source language model "Galactica" released by Meta Company not long before the release of ChatGPT was severely criticized by some people in the scientific community, forcing Meta to take it offline. This obviously hasn't prevented generative AI from entering scientific writing, but it may have prevented the introduction of a system specifically optimized for this task. Highlights: An analysis of PubMed abstracts found that since the launch of ChatGPT, at least 10% of scientific abstracts have been affected by the AI text generator. In the PubMed subgroup in countries such as China and South Korea, approximately 15% of abstracts were generated using ChatGPT, compared with only 3% in the UK. Researchers are calling for a re-evaluation of guidelines for using AI text generators in science as AI text generators can fabricate facts, reinforce biases and even commit plagiarism.
The results of this study warn us that the application of AI in scientific writing needs to be treated with caution, and stricter norms and ethical guidelines need to be formulated to ensure academic integrity and research reliability. Future research should further explore how to better identify and avoid the risks brought by AI-generated text and balance the convenience of AI-assisted writing with potential negative impacts.