The study focuses on the comparability of the digital and print test modes of a German reading comprehension test. 249 third graders (50.5% girls, 20.2% L2 learners) worked with both test modes (two weeks between the measurement periods). Test mode order was controlled. The data was analyzed in terms of (a) test score comparability, (b) validity, and (c) the impact of test order, gender and language on test mode effects. The results revealed that the two test modes were not equivalent. Differences in mean scores were particularly large for those subtests whose task format differed significantly in the two modes. While gender and language had no influence on the test mode effect, test mode order did play an important role.