Importance Large language models (LLMs) may facilitate the labor-intensive process of systematic reviews. However, the exact methods and reliability remain uncertain. Objective To explore the feasibility and reliability of using LLMs to assess risk of bias (ROB) in randomized clinical trials (RCTs). Design, Setting, and Participants A survey study was conducted between August 10, 2023, and October 30, 2023. Thirty RCTs were selected from published systematic reviews. Main Outcomes and Measures A structured prompt was developed to guide ChatGPT (LLM 1) and Claude (LLM 2) in assessing the ROB in these RCTs using a modified version of the Cochrane ROB tool developed by the CLARITY group at McMaster University. Each RCT was assessed twice by both models, and the results were documented. The results were compared with an assessment by 3 experts, which was considered a criterion standard. Correct assessment rates, sensitivity, specificity, and F1 scores were calculated to reflect accuracy, both overall and for each domain of the Cochrane ROB tool; consistent assessment rates and Cohen kappa were calculated to gauge consistency; and assessment time was calculated to measure efficiency. Performance between the 2 models was compared using risk differences. Results Both models demonstrated high correct assessment rates. LLM 1 reached a mean correct assessment rate of 84.5% (95% CI, 81.5%-87.3%), and LLM 2 reached a significantly higher rate of 89.5% (95% CI, 87.0%-91.8%). The risk difference between the 2 models was 0.05 (95% CI, 0.01-0.09). In most domains, domain-specific correct rates were around 80% to 90%; however, sensitivity below 0.80 was observed in domains 1 (random sequence generation), 2 (allocation concealment), and 6 (other concerns). Domains 4 (missing outcome data), 5 (selective outcome reporting), and 6 had F1 scores below 0.50. The consistent rates between the 2 assessments were 84.0% for LLM 1 and 87.3% for LLM 2. LLM 1's kappa exceeded 0.80 in 7 and LLM 2's in 8 domains. The mean (SD) time needed for assessment was 77 (16) seconds for LLM 1 and 53 (12) seconds for LLM 2. Conclusions In this survey study of applying LLMs for ROB assessment, LLM 1 and LLM 2 demonstrated substantial accuracy and consistency in evaluating RCTs, suggesting their potential as supportive tools in systematic review processes.

COVID-19 is not only associated with substantial acute liver and kidney injuries, but also with an elevated risk of post-acute sequelae involving the kidney and liver system. We aimed to investigate whether COVID-19 exposure increases the long-term risk of kidney and liver disease, and what are the magnitudes of these associations. We searched PubMed, Embase, Web of Science, , and the Living Overview of the Evidence COVID-19 Repository for cohort studies estimating the association between COVID-19 and kidney and liver outcomes. Random-effects meta-analyses were performed to combine the results of the included studies. We assessed the certainty of the evidence using the Grading of Recommendations Assessment, Development and Evaluation approach. Fifteen cohort studies with more than 32 million participants were included in the systematic review COVID-19 was associated with a 35% greater risk of kidney diseases (10 more per 1000 persons; low certainty evidence) and 54% greater risk of liver disease (3 more per 1000 persons; low certainty evidence). The absolute increases due to COVID-19 for acute kidney injury, chronic kidney disease, and liver test abnormality were 3, 8, and 3 per 1000 persons, respectively. Subgroup analyses found no differences between different type of kidney and liver diseases. The findings provide further evidence for the association between COVID-19 and incident kidney and liver conditions. The absolute magnitude of the effect of COVID-19 on kidney and liver outcomes was, however, relatively small.