Objectives

HTA agencies vary significantly in their methodology, interpretation of clinical evidence and reporting of recommendations. This research builds on established frameworks for assessing HTA outcomes and proposes a novel approach towards characterising and comparing clinical issues raised by HTA agencies.

Methods

HTA data from Canada, England, Scotland, Sweden, Germany, and France were extracted from an internal HTA database (HTA-Hive), constructed using a validated and published framework for the assessment of HTA outcomes. Clinical issues are categorised into: magnitude of clinical benefit, long-term clinical evidence, study design, generalizability, potential adverse events, comparators, and indirect comparisons. A total of 786 HTA reports (spanning 2009-2024) were included in the study. A sentiment analysis model VADER using the nltk package in Python was applied to the uncertainties to assess sentiment severity across agencies.

Results

There were total of 3,272 clinical uncertainties in the sample. The most common clinical uncertainty raised related to magnitude of clinical benefit (22.6%), while issues relating to appropriateness of comparator were the least frequent (8.3%). TLV had the worst average sentiment (-0.024), it was significantly worse than HAS, INESSS, IQWiG and NICE (p<0.005). NICE had the most positive (0.20), significantly better than CADTH, HAS, SMC, and TLV. The average compound sentiment across all clinical uncertainties was 0.11. The category with the lowest average sentiment was adverse effects (-0.209) highest was clinical benefit (0.34). Sentiment scores were highly variable across manufacturers with more than 10 reports (0.015 – 0.205). Average sentiment was similar across orphan and non-orphan medicines (0.133 vs 0.107) and significantly higher for oncology drugs vs other drugs (0.185 vs 0.069).

Conclusions

Natural Language Processing (NLP) techniques can help shed light on the severity of clinical issues raised in the context of HTA, which tend to vary according to HTA agency, type of uncertainty, therapeutic area and disease area.