A Preliminary Assessment of SLR's Reliance on Preprints, in the Area of LLMs4SE

Background: Systematic Literature Reviews (SLRs) and Mapping Studies (MSs) are established features of academic Software Engineering literature. But for new, exciting areas in the field, there exists a tension between supporting fellow academics, with reviews that provide a timely synthesis of the available evidence, and the maturity of the literature relied on. Aims: The increasingly prevalent use of preprint databases like arXiv, for immediate publication of results, suggests that this tension deserves evaluation. Hence, this paper evaluates preprint reliance for SLRs/MSs in the rapidly developing area of LLMs4SE: Large Language Models for Software Engineering. Method: The paper analyses surveys on LLMs4SE, identifying the proportion of arXiv preprints that the surveys employ, the additional quality assurance measures applied to the preprints, and tracing the assertions made in the surveys to the preprints' subsequent peer-reviewed articles for consistency. Results: Our initial review found a small number of surveys, congruent with LLMs recent growth in software engineering. The reviews that exist rely on a significant proportion of pre-prints, and meaningful quality criteria are rarely applied to mitigate against this. We note that although a significant proportion of the preprints are subsequently published in peer-reviewed fora, a significant number are not. In cases where the preprints have been subsequently published in peer-reviewed fora, consistency between the assertions made in the review and the peer-reviewed article is relatively high, but not absolute. Conclusions: Our results suggest that increased quality criteria should be applied to the preprints employed and consistency checking between preprints and surveys should be heightened.