Language models can extract documented cannabis use, but need validation

Extracting patient-reported cannabis use and reasons for use from EHR in rheumatology: a benchmarking study of language models.

BMJ health & care informatics β€’ β€’ Highly Relevant
πŸ€–

AI Summary

This retrospective study asked whether language models can identify documented cannabis use and reasons for use in electronic health records of patients with autoimmune rheumatic diseases. Researchers compared language models, a fine-tuned clinical model, and a logistic-regression baseline using expert-labeled snippets from clinical notes recorded between 2015 and 2024. The tasks were to classify use status and identify documented reasons such as pain, nausea, sleep, or stress and anxiety.

The fine-tuned GatorTron model had the highest reported point estimates for use-status classification, while GPT-OSS-20B had the highest for reasons for use. The confidence intervals for the best and second-best models overlapped, so the apparent differences should be interpreted cautiously. The study indicates that automated extraction is feasible, but this retrospective benchmarking study cannot establish how common cannabis use is among patients with these conditions; the authors call for validation in samples representative of prevalence. This is an abstract-based summary.

πŸ’‘ Key Findings

1
In a retrospective study of rheumatology records, language models and a fine-tuned clinical model could extract documented cannabis-use status and reasons for use from clinical-note snippets.
High
95%
2
Fine-tuned GatorTron had the highest point estimates for use-status classification (accuracy 0.88; F1 0.88).
High
95%
3
GPT-OSS-20B had the highest point estimates for reason-for-use classification (accuracy 0.84; F1 0.83); confidence intervals overlapped those of the second-best models, warranting caution.
High
95%
4
The findings are exploratory and do not establish cannabis-use prevalence in patients with autoimmune rheumatic diseases; validation in prevalence-representative samples is needed.
High
99%

πŸ“„ Original Abstract

OBJECTIVE: To develop and evaluate a scalable natural language processing pipeline to identify cannabis use status and reasons for use among patients with autoimmune rheumatic diseases (ARDs) from unstructured electronic health record (EHR) data. METHODS AND ANALYSIS: We conducted a retrospective study using EHR clinical notes from patients with ARDs (2015–2024). Cannabis-related mentions were screened by lexicon-based keyword matching and extracted as 50-word snippets. Two experts annotated 840 snippets for use-status classes (not a true mention/uncertain, denial, past use, current use) and 773 snippets for reason-for-use categories (pain, nausea, sleep, relaxation/stress/anxiety, not mentioned/unknown). We compared multiple large language model (LLM) prompting strategies, a fine-tuned clinical encoder model (GatorTron) and a logistic regression baseline using held-out class-balanced test sets (status n=40, reason n=50; 10/class) using accuracy and F1. RESULTS: Fine-tuned GatorTron achieved the highest point estimates for use status (accuracy=0.88, 95% CI (0.78 to 0.98); F1=0.88, 95% CI (0.77 to 0.97)) and GPT-OSS-20B for reason for use (accuracy=0.84, 95% CI (0.74 to 0.92); F1=0.83, 95% CI (0.78 to 0.94)). For both tasks, CIs of the best and second-best models overlapped, calling for cautious interpretation. We also characterised clinical trends in documented cannabis use from 2015 to 2024. DISCUSSION: This benchmarking framework offers a scalable approach to characterising documented cannabis use from EHR. Findings are exploratory and validation in prevalence-representative samples is needed in future work. CONCLUSIONS: Extraction of cannabis use status and reasons for use from EHR notes is feasible using fine-tuned language models and generative LLMs. This approach can support real-world evidence research on symptom self-management in ARDs.

Explore More Research

Stay informed about the latest cannabis science.

Your stash, decoded.