Importance Large language models (LLMs) may facilitate the labor-intensive process of systematic reviews. However, the exact methods and reliability remain uncertain. Objective To explore the feasibility and reliability of using LLMs to assess risk of bias (ROB) in randomized clinical trials (RCTs). Design, Setting, and Participants A survey study was conducted between August 10, 2023, and October 30, 2023. Thirty RCTs were selected from published systematic reviews. Main Outcomes and Measures A structured prompt was developed to guide ChatGPT (LLM 1) and Claude (LLM 2) in assessing the ROB in these RCTs using a modified version of the Cochrane ROB tool developed by the CLARITY group at McMaster University. Each RCT was assessed twice by both models, and the results were documented. The results were compared with an assessment by 3 experts, which was considered a criterion standard. Correct assessment rates, sensitivity, specificity, and F1 scores were calculated to reflect accuracy, both overall and for each domain of the Cochrane ROB tool; consistent assessment rates and Cohen kappa were calculated to gauge consistency; and assessment time was calculated to measure efficiency. Performance between the 2 models was compared using risk differences. Results Both models demonstrated high correct assessment rates. LLM 1 reached a mean correct assessment rate of 84.5% (95% CI, 81.5%-87.3%), and LLM 2 reached a significantly higher rate of 89.5% (95% CI, 87.0%-91.8%). The risk difference between the 2 models was 0.05 (95% CI, 0.01-0.09). In most domains, domain-specific correct rates were around 80% to 90%; however, sensitivity below 0.80 was observed in domains 1 (random sequence generation), 2 (allocation concealment), and 6 (other concerns). Domains 4 (missing outcome data), 5 (selective outcome reporting), and 6 had F1 scores below 0.50. The consistent rates between the 2 assessments were 84.0% for LLM 1 and 87.3% for LLM 2. LLM 1's kappa exceeded 0.80 in 7 and LLM 2's in 8 domains. The mean (SD) time needed for assessment was 77 (16) seconds for LLM 1 and 53 (12) seconds for LLM 2. Conclusions In this survey study of applying LLMs for ROB assessment, LLM 1 and LLM 2 demonstrated substantial accuracy and consistency in evaluating RCTs, suggesting their potential as supportive tools in systematic review processes.

Background: Acupuncture has been widely practiced for primary insomnia (PI). However, the relative benefit and harm among acupuncture therapies remain uncertain.Objectives: To compare and evaluate the effect differences of multiple acupuncture therapies for patients with PI.Methods: Systematic literature search for randomized controlled trials (RCTs). Pairs of reviewers independently conducted literature screening, data extraction, and risk of bias assessment. Meta-analysis was conducted using R and Stata software. The Grading of Recommendations Assessment, Development, and Evaluation system (GRADE) was used to assess certainty of evidence and interpret results.Results: Fifty-seven RCTs with 4678 patients were included. Compared with usual treatment, multiple acupuncture therapies showed a better effect for Pittsburgh sleep quality index score. And acupoints catgut embedding (ACE) was proved to be the most effective with a moderate certainty of evidence; auricular acupressure or auricular acupuncture plus manual acupuncture (AP + MA), electroacupuncture plus acupoint application (EA + APA), and intradermal needle (IN) might be also the most effective with low certainty of evidence. ACE, ACE + MA, AP + MA, EA, EA + APA, HPN, MA and PBN + MA showed significantly improvement in effective rate when compared with usual treatment. Insufficient evidence reported on Epworth Sleepiness Scale, Athens Insomnia Scale, and recurrence rate. The most common slight adverse events mainly included hematoma, pain, headache, and bleeding.Conclusions: With moderate to low certainty of evidence, multiple acupuncture therapies showed impressive insomnia improvement, especially ACE, AP + MA, and EA + APA. Differences between therapies were small or insignificant and based-on low or very low certainty of evidence.(c) 2022 Elsevier B.V. All rights reserved.