MIRROR: Multi-Modal Pathological Self-Supervised Representation Learning via Modality Alignment and Retention

Tianyi Wang1, Jianan Fan1, Dingxin Zhang1, Dongnan Liu1, Yong Xia2, Heng Huang3, and Weidong Cai1,
1The University of Sydney; 2Northwestern Polytechnical University; 3University of Maryland
Paper Data Code arXiv

📢 We are pleased to announce that a revised version of our manuscript is now available on ArXiv (v4).

Overview

Architecture Image

Histopathology and transcriptomics are fundamental modalities in cancer diagnostics, encapsulating the morphological and molecular characteristics of the disease. Multi-modal self-supervised learning has demonstrated remarkable potential in learning pathological representations by integrating diverse data sources. Conventional multi-modal integration methods primarily emphasize modality alignment, while paying insufficient attention to retaining the modality-specific intrinsic structures. However, unlike conventional scenarios where multi-modal inputs often share highly overlapping features, histopathology and transcriptomics exhibit pronounced heterogeneity, offering orthogonal yet complementary insights. Histopathology data provides morphological and spatial context, elucidating tissue architecture and cellular topology, whereas transcriptomics data delineates molecular signatures through quantifying gene expression patterns. This inherent disparity introduces a major challenge in aligning these modalities while maintaining modality-specific fidelity. To address these challenges, we present MIRROR, a novel multi-modal representation learning framework designed to foster both modality alignment and retention. MIRROR employs dedicated encoders to extract comprehensive feature representations for each modality, which is further complemented by a modality alignment module to achieve seamless integration between phenotype patterns and molecular profiles. Furthermore, a modality retention module safeguards unique attributes from each modality, while a style clustering module mitigates redundancy and enhances disease-relevant information by modeling and aligning consistent pathological signatures within a clustering space. Extensive evaluations on The Cancer Genome Atlas (TCGA) cohorts for cancer subtyping and survival analysis highlight MIRROR's superior performance, demonstrating its effectiveness in constructing comprehensive oncological feature representations and benefiting the cancer diagnosis.

Highlights

The key contributions of this study are outlined as follows:

  • MIRROR, a novel multi-modal self-supervised learning (SSL) model, is designed to facilitate both modality alignment and retention, enabling the effective preservation of both modality-shared and modality-specific information.
  • A consistent pathological style-based clustering mechanism is introduced to preserve disease-relevant information while mitigating redundancy.
  • A novel preprocessing pipeline for transcriptomics data is proposed, integrating machine learning-driven feature selection with biological knowledge to create refined transcriptomics datasets.
  • Comprehensive evaluations are conducted across diverse cohorts from the TCGA dataset, focusing on cancer subtyping and survival analysis tasks, substantiating the superior performance and effectiveness of the proposed approach.

Datasets

We provide the processed transcriptomics data on Kaggle, Hugging Face, and Zenodo for TCGA-BRCA, TCGA-NSCLC, TCGA-COADREAD, and TCGA-RCC.

BibTeX

@misc{wang2025mirror,
    title={MIRROR: Multi-Modal Pathological Self-Supervised Representation Learning via Modality Alignment and Retention},
    author={Tianyi Wang and Jianan Fan and Dingxin Zhang and Dongnan Liu and Yong Xia and Heng Huang and Weidong Cai},
    year={2025},
    eprint={2503.00374},
    archivePrefix={arXiv},
    primaryClass={cs.CV}
}