Manipulation Checks in Focus of Attention Research: A Methodological Systematic Review

Manipulation checks are an essential component of quality experimental design in motor learning. Guided by the Preferred Reporting Items for Systematic Reviews and Meta-Analyses framework, this methodological systematic review examined the utilization of manipulation checks in focus of attention research. Seventy-eight protocols from four scientific databases from September 2021—September 2023 were evaluated. A secondary analysis was conducted on 10 data sets where manipulation check failures had not been explicitly considered. No manipulation check was used in 53% of protocols, where a manipulation check was used: 19% used a single rating question, 11.5% used multiple questions, 11.5% used open questions, and 5% used other forms. When manipulation checks were utilized in studies, 68% of experiments did not report if any participants failed the manipulation check or not; the remaining reports either analyzed the failures (11%), excluded the failures (8%), ran multiple analyses (8%), or stated that no participants failed (5%). In the secondary analysis of data, when participants who failed manipulation checks were excluded, the interpretation of results changed in two of the 10 reanalyzed sources when compared with the original analysis. We conclude with recommendations on how researchers can optimize their use of manipulation checks within focus of attention research in both the collection and analysis of data.