Objective:

When two independent groups of expert judges study content validity, a formal test of the differences between their judgments is required, since different content validity judgments can be obtained. But generally, content validity research does not examine this likely source of discrepancies. This report describes the implementation of a method to evaluate the difference in Aiken's V coefficients applied to research work in sports science.

Methodology:

The procedure applies an adaptation to construct the confidence interval of the difference between Aiken’s V coefficients and also implements a standardized estimator of the size of the difference between the V coefficients, specifically the arcsine transformation of V coefficients.

Results:

In a secondary data analysis framework, two examples are developed, extracting data from both publications, and the difference between the impressionist-based conclusion and the empirical-based conclusion and formal evaluation is demonstrated. Statistical differences not previously observed were detected.

Conclusions and implications:

The method to estimate differences in Aiken’s content validity coefficients for research allows an advance in the methodology to validate measurement instruments. The applicability of this procedure in the context of sports sciences and education sciences, as well as in the research design involved, is assessed.