Leveraging Large Language Models for the Extraction of General and Domain-Specific Signs and Symptoms

Large language models like GPT-4o, a enhanced variant of GPT-4, have the potential to enhance the extraction of signs and symptoms (S&S) from clinical notes, improving the analysis of unstructured data in electronic health records. This study evaluates the performance of GPT-4o in extracting S&S from clinical notes, focusing on both overall and urinary-specific S&S. Clinical notes from the MTSamples corpus were analyzed using GPT-4o, with precision, recall, and F1-score assessed against human-labeled data. Consistency was measured using agreement rate and Cohen’s Kappa. GPT-4o showed moderate performance in overall S&S extraction (precision: 53.8%, recall: 92.6%, F1-score: 68.1%). For urinary-specific S&S, performance improved significantly, with a perfect precision of 100%, recall of 92.2%, and F1-score of 95.9%. Consistency was high, though variability was noted in agreement metrics. The findings confirm the efficacy of GPT-4o for extracting urinary symptoms and underscore the need for domain-specific approaches to optimize clinical NLP tasks.