Theory-Driven Game-Based Assessment of General Cognitive Ability: Design Theory, Measurement, Prediction of Performance, and Test Fairness

Landers, RN (通讯作者),Univ Minnesota, Dept Psychol, Minneapolis, MN 55455 USA.
2022-10
Games, which can be defined as an externally structured, goal-directed type of play, are increasingly being used in high-stakes testing contexts to measure targeted constructs for use in the selection and promotion of employees. Despite this increasing popularity, little is known about how theory-driven game-based assessments (GBA), those designed to reflect a targeted construct, should be designed, or their potential for achieving their simultaneous goals of positive reactions and high-quality psychometric measurement. In the present research, we develop a theory of GBA design by integrating game design and development theory from human-computer interaction with psychometric theory. Next, we test measurement characteristics, prediction of performance, fairness, and reactions of a GBA designed according to this theory to measure latent general intelligence (g). Using an academic sample with GPA data (N = 633), we demonstrate convergence between latent GBA performance and g (beta = .97). Adding an organizational sample with supervisory ratings of job performance (N = 49), we show GBA prediction of both GPA (r = .16) and supervisory ratings (r = .29). We also show incremental prediction of GPA using unit-weighted composites of the g test battery beyond that of the g-GBA battery but not the reverse. We also show the presence of similar adverse impact for both the traditional test battery and GBA but the absence of differential prediction of criteria. Reactions were more positive across all measures for the g-GBA compared to the traditional test battery. Overall, results support GBA design theory as a promising foundation from which to build high-quality theory-driven GBAs.
JOURNAL OF APPLIED PSYCHOLOGY
卷号:107|期号:10|页码:1655-1677
ISSN:0021-9010|收录类别:SSCI
语种
英语
来源机构
University of Minnesota System; University of Minnesota Twin Cities; Old Dominion University
资助机构
Revelian Pty Ltd
资助信息
YYYY Early versions of this article were presented at the annual conference of the Society for Industrial and Organizational Psychology. Participant payments and graduate research hours in this study were funded by Revelian Pty Ltd, and Richard N. Landers became a compensated member of Revelian's Scientific Advisory Board midproject. Michael B. Armstrong is now at Google, Andrew B. Collmus is now at Facebook, Salih Mujcic is now at Traitstack, and Jason Blaik is now at Cappfinity.
被引频次(WOS)
11
被引频次(其他)
11
180天使用计数
9
2013以来使用计数
28
EISSN
1939-1854
出版年
2022-10
DOI
10.1037/apl0000954
学科领域
循证管理学
关键词
game-based assessment game design validation measurement fairness
WOS学科分类
Psychology, Applied Management