What are you looking at? Modality contribution in multimodal medical deep learning

1

Multimodal datasets in medicine are growing, necessitating advanced fusion methods to process high-dimensional data.
2

Interpretability methods enhance the credibility of multimodal AI in clinical practice, bridging technology and human care.
3

Existing interpretability methods for multimodal data are underexplored and often lack quantification of modality importance.
4

This work introduces a performance and model agnostic metric to measure modality contribution in multimodal medical tasks.
5

The proposed method allows detection of unimodal collapses and facilitates comparison of architectures in processing modalities.

International Journal Of Computer Assisted Radiology And Surgery

by Christian Gapp, Elias Tappeiner, Martin Welk, Karl Fritscher, Elke R. Gizewski, Rainer Schubert
October 2, 2025

1