To develop a manually annotated corpus for the detection of geriatric syndromes (GS) in hospital discharge summaries and to assess the effectiveness of NLP models in extracting GS-related information.
Approach:
Annotation Scheme Development: A comprehensive annotation scheme was created to label 12 common GS, incorporating attributes like diagnosis type, negation, and event occurrence.
Dataset Creation: The corpus consists of 2,040 manually annotated discharge summaries from NHS Lothian hospitals in Scotland.
NLP Model Evaluation: Multiple pretrained transformer-based models were fine-tuned and tested for named entity recognition (NER) and document-level labelling tasks, including context-aware tasks.
Key Findings:
BERT-cased achieved the highest F1-score (0.897) for document-level labelling.
BioClinicalBERT performed best for negation consideration (F1-score: 0.888).
Frailty, Falls, and Delirium were the GS entities with the best performance results.
BioClinicalBERT and BERT-cased achieved an F1-score of 0.883 for the NER task.
NER-C tasks showed BERT-cased had the best F1 of 0.692, with context-aware falls and frailty labels performing well.
Interpretation:
Model performance is influenced by dataset characteristics, with more frequent and clearer syndromes yielding stronger results, while rarer categories faced challenges in extraction accuracy.
Limitations:
Low-frequency GS categories and sparse contextual labels negatively affected model accuracy.
Discontinuities and overlapping mentions in clinical narratives complicated entity extraction, making boundary detection more challenging.
Conclusion:
The study emphasizes the importance of data distribution and annotation complexity in model performance for extracting geriatric syndromes from unstructured clinical text.
