Training and Experience in Study Selection (TESS): study protocol for a pilot randomised trial within a systematic review

Background: Systematic reviews can be resource-intensive and require timely completion, yet experienced reviewers may have limited availability, necessitating the inclusion of novice screeners. This Study Within A Review (SWAR) will pilot and examine the feasibility of study methods to explore whether training and level of experience within the screening pair affects the reliability of decisions made by novice screeners during study selection. Methods: A 2(training: task-specific, minimal guidance) x 2(experience level of screening partner, ‘Reviewer 1’: moderate experience, minimal experience) pilot randomised trial will be conducted within a host systematic review in the topic area of depression and psychosocial functioning. Participants (N = 12), consisting of higher education students with no prior experience in evidence synthesis, will be randomised to one of the four conditions to complete a standardised study selection task at title/abstract level (k = 219 records) on Covidence screening software, blindly and independently. Total participation time is estimated at 5 hours. Screening decisions made by participants will be assessed for reliability against the consensus-based decisions by two reviewers with content and methodological expertise (expert standard), through calculation of chance-corrected Cohen’s kappa, prevalence-adjusted bias-adjusted kappa (PABAK), and percentage of agreement, then compared across the conditions. Secondary outcomes will include reliability within the screening pair (participant and allocated screening partner), validity of screening decisions (false positives, false negatives, sensitivity, specificity), feasibility measures, including time taken to complete the study selection task and success of blinding, as well as acceptability. Conclusions: Findings will be used to inform the design of subsequent trial work to determine the efficacy of training and screener pairing for study selection. Ultimately, these insights will help to build capacity among novice screeners to engage with evidence synthesis and work alongside experienced review teams. Registration: Northern Ireland Hub for Trials Methodology Research SWAR Registry: https://www.qub.ac.uk/sites/TheNorthernIrelandNetworkforTrialsMethodologyResearch/FileStore/SWARFileStore/SWAR38%20Elayne%20Ahern,%20Aoife%20Whiston,%20Sarah%20Dillon,%20Fiona%20Lynn%20(2024%20SEP%2027%202321).pdf SWAR 38.