Creation of Multi-Modal Datasets for Federated Learning Using DICOM-Formatted Reports

Overview

This study demonstrates a novel approach to harmonize multi-modal medical data for federated learning by leveraging the DICOM standard and its structured reports. The authors developed publicly available tools and an intuitive dashboard to enable multi-center cohort creation while respecting privacy regulations.

Background

Multi-modal deep learning in medicine requires large, harmonized datasets, but privacy regulations like GDPR limit centralized data sharing. Federated learning offers a solution by training models locally and aggregating weights centrally, yet data harmonization across institutions remains challenging, especially for multi-modal data. Standardized data formats such as DICOM structured reports can facilitate consistent data representation and interoperability across centers. An intuitive, non-programmatic dashboard for filtering and cohort creation can further streamline multi-center studies in a federated setting.

Data Highlights

The study introduces a workflow converting diverse medical data into DICOM format, uploading it into a research PACS, and enabling filtering via a graphical dashboard. The dashboard supports iterative filtering with linked views across images, waveforms, segmentations, and metadata. The approach was implemented across multiple consortium sites, allowing federated cohort selection and subsequent federated training. The source code is publicly available at https://github.com/Cardio-AI/fl-multi-modal-dataset-creation.

Key Findings

• Leveraging DICOM standards enables harmonization of imaging, waveform, and metadata through structured reports and inherent matching capabilities.
• Publicly available Python scripts were developed to create and convert data into DICOM SR templates suitable for deep learning pipelines.
• An intuitive, customizable dashboard was created for iterative filtering and cohort creation of multi-modal DICOM data residing in PACS systems.
• The filtering dashboard supports linked graphical views and instant updates, enhancing usability without programming knowledge.
• DICOM-based data ensures interoperability, supporting standard SQL queries and potential conversion to other standards like FHIR.
• The federated infrastructure was successfully deployed across multiple sites, enabling local cohort selection and federated model training.

Clinical Implications

This approach facilitates multi-center federated learning by overcoming privacy and data heterogeneity barriers through standardized DICOM data representation. Clinicians and researchers can intuitively create multi-modal cohorts without programming expertise, accelerating collaborative studies. The interoperability with existing clinical systems supports integration into clinical workflows and future scalability.

Conclusion

Utilizing DICOM structured reports for multi-modal data harmonization enables effective federated learning across institutions while respecting privacy constraints. The provided tools and dashboard offer a practical solution for scalable, interoperable cohort creation in medical imaging and related modalities.

References

1. Cardio-AI Consortium 2024 -- Creation of Multi-Modal Datasets for Federated Learning Utilizing DICOM-Formatted Reports