Color Me Confounded: A Critical Analysis of Media Comparisons on ChatGPT in Education

Research on the use of artificial intelligence in education is rapidly accumulating. In the ChatGPT literature, two recent meta-analyses were published (Deng et al., 2025 and Wang & Fan, 2025) that examined media comparisons between ChatGPT and more traditional forms of instruction; both reviews found ChatGPT to improve learning. However, as researchers have found in media comparisons studies (e.g., Clark, 1983; Lawson et al., 2024), instructional methods are often not controlled between conditions, making it difficult to attribute any learning differences to affordances of the medium itself. Therefore, the purpose of our critical review was to examine those media comparisons studies included in Deng et al. (2025) and Wang and Fan (2025) to understand whether the ChatGPT and control conditions were comparable on their instructional features (i.e., matched on instructional methods, practice with the dependent measure, and time spent learning the content). Results indicated a lack of control in the ChatGPT studies that were reviewed, with many studies conflating the use of ChatGPT with other instructional features. Further, across the various direct learning outcomes, studies often did not include enough information to determine the comparability of conditions. For the most common learning outcome (i.e., ChatGPT > Control), comparisons often involved a confound between conditions as well as missing information about instructional features. Therefore, the benefits of ChatGPT must be interpreted with caution at the present time, and more research is needed to determine how, for whom, and under what pedagogical conditions ChatGPT can improve learning.