FOOLING YOURSELF WITH CROSS‐VALIDATION: SINGLE SAMPLE DESIGNS

The most commonly used cross‐validation design involves drawing a single sample and partitioning that sample into derivation and holdout subsamples. This type of design allows one to adjust for random sampling error, but like formula estimates of cross‐validity, is insensitive to violations of sampling assumptions. As is shown in a small Monte Carlo study, results obtained in non‐representative samples, which are known to be invalid in the population, will nonetheless hold up well under cross‐validation when single‐sample designs are employed. It is suggested that single‐sample cross‐validation estimates possess no clear‐cut advantages over formula estimates, and thus are not worth the effort or the loss of degrees of freedom.