Background: Physical exercise can slow down the decline of the cognitive function of the older adults, yet the review evidence is not conclusive. The purpose of this study was to compare the effects of aerobic and resistance training on cognitive ability. Methods: A computerized literature search was carried out using PubMed, Cochrane Library, Embase SCOPUS, Web of Science, CNKI (China National Knowledge Infrastructure), Wanfang, and VIP database to identify relevant articles from inception through to 1 October 2022. Based on a preliminary search of the database and the references cited, 10,338 records were identified. For the measured values of the research results, the standardized mean difference (SMD) and 95% confidence interval (CI) were used to synthesize the effect size. Results: Finally, 10 studies were included in this meta-analysis. Since the outcome indicators of each literature are different in evaluating the old cognitive ability, a subgroup analysis was performed on the included literature. The study of results suggests that aerobic or resistance training interventions significantly improved cognitive ability in older adults compared with control interventions with the Mini-Mental State Examination (MD 2.76; 95% CI 2.52 to 3.00), the Montreal Cognitive Assessment (MD 2.64; 95% CI 2.33 to 2.94), the Wechsler Adult Intelligence Scale (MD 2.86; 95% CI 2.25 to 3.47), the Wechsler Memory Scale (MD 9.33; 95% CI 7.12 to 11.54), the Wisconsin Card Sorting Test (MD 5.31; 95% CI 1.20 to 9.43), the Trail Making Tests (MD -8.94; 95% CI -9.81 to -8.07), and the Stroop Color and Word Test (MD -5.20; 95% CI -7.89 to -2.51). Conclusion: Physical exercise improved the cognitive function of the older adults in all mental states. To improve cognitive ability, this meta-analysis recommended that patients perform at least moderate-intensity aerobic exercise and resistance exercise on as many days as possible in the week to comply with current exercise guidelines while providing evidence for clinicians.