Multimodal Medicine AI

This repository is dedicated to curating research papers on multimodal medicine AI, encompassing reviews, benchmarks, and cutting-edge AI or machine learning techniques for analyzing clinical tasks using multimodal medical data. These tasks include treatment response assessment, prognosis evaluation, diagnosis, recurrence prediction, and more. We also aim to continually gather and update relevant open datasets to support research in multimodal medicine.

[October 28, 2024] Articles collected so far: 59

Reviews

⭐️ [Oct. 2024] Recent Advances in Data-driven Fusion of Multi-modal Imaging and Genomics for Precision Medicine, Information Fusion

Paper

Highlights:

Providing a systematic overview of imaging genomics across multiple organs throughout the body.
highlighting the recent advancements in data-driven fusion methods for imaging genomics.
offering detailed reports on multi-modal imaging, imaging-genomics fusion, and clinical applications.
discussing the current challenges in imaging genomics, providing insights and future prospects.

Context:

Multi-organ Imaging and IDP Extraction (IDP, imaging-derived phenotypes)
- Brain
  - Modalities: commonly using MRI, including T1-magnetization prepared rapid gradient echo (MPRAGE), T2 fluid-attenuated inversion-recovery (FLAIR), diffusion-weighted imaging (DWI), diffusion tensor imaging (DTI), task-based functional MRI (tfMRI), and resting-state functional MRI (rfMRI).
  - Clinical applications: brain IDPs serve as valuable biomarkers for assessing complex brain diseases, including Parkinson's disease, Alzheimer's disease (AD), ischemic stroke, and neuropsychiatric disorders.
- Heart
  - Modalities: cardiac IDPs are usually extracted from various imaging modalities, including cardiac MRI (CMR), echocardiography, and CT. Similar to brain imaging, MRI is the primary cardiac imaging method for population genomics studies in cardiology, including T1 mapping, T2 mapping, cine, DTI, Q-flow, coronary imaging, 2D-PC, and 4D-flow.
  - Clinical applications: cardiac IDPs are valuable for exploring associations with diseases such as coronary artery disease (CAD), dilated cardiomyopathy (DCM), hypertrophic cardiomyopathy (HCM), long QT syndrome, pulmonary hypertension, and heart failure
- Lung
  - Modalities: pulmonary IDPs can be obtained utilizing a range of imaging modalities, including X-ray, CT, MRI, US, and PET/CT. Among these, CT is particularly advantageous due to its high resolution, faster imaging speed and lower cost, making it the preferred and most widely used method for obtaining pulmonary IDPs.
  - Clinical applications: pulmonary IDPs holds promise in the diagnosis and management of various pulmonary conditions, encompassing bacterial lung infections, pulmonary viral and fungal infections, bronchiectasis, lung abscesses, pulmonary fibrosis (PF), emphysema, chronic obstructive pulmonary disease (COPD), and lung tumors
- Abdomen
  - Modalities: abdominal IDPs, including several organs like liver, kidney, spleen, pancreas as well as visceral and subcutaneous fat, are predominantly obtained through CT and MRI.
  - Clinical applications: these imaging modalities provide valuable insights into obesity, metabolic disorders, and liver disease.
- Bone
  - Modalities: skeletal imaging techniques, such as photon, dual-energy X-ray absorptiometry (DXA), CT, MRI, and US, provide a rich source of IDPs related to bone and muscle health.
  - Clinical applications: these IDPs play a crucial role in identifying risk factors monitoring disease progression, and assessing the effectiveness of interventions for conditions like osteoporosis, sarcopenia and fractures.
- Breast
  - Modalities: breast imaging techniques include mammography, ultrasound, PET, CT, and MRI.
  - Clinical applications: these measurements yield valuable information regarding the size, shape, density, and texture of the breast tissue, in addition to identifying any suspicious lesions or calcifications.
Genome Sequencing
- Single nucleotide polymorphisms (SNP) is one of the most common types of genetic variation in the human genome.
- SNP data processing involve collecting and extracting DNA samples from different individuals, detecting SNPs using high-throughput methods, standardizing and analyzing the data by genomewide association studies.
- Existing technologies that can be used to acquire SNP data include chip-based analysis and sequencing-based analysis:
  - Chip-based analysis uses specific chips to detect the variation of SNP loci in the sample.
  - Sequencing-based SNP analysis utilizes high-throughput sequencing technologies to detect SNP variation by comparing the sample to the reference genome.
- Common sequencing technologies include Sanger sequencing, second-generation sequencing (such as Illumina and Ion Torrent), and third-generation sequencing (such as PacBio and Oxford Nanopore)
- Commonly used file formats to store genetic data include VCF, BED, BAM, FASTQ, GTF/GFF, and BGEN.
Fusion Strategies:
- Coorelation analysis
  - Genome-wide Association Analysis (GWAS): GWAS and post-GWAS analysis have become foundational in imaging genomics, playing a crucial role in uncovering associations between imaging features and genes.
  - Pathway Analysis: pathway enrichment analysis is a valuable method for identifying specific biological pathways in a group of genes, facilitating the identification of associations between biological functions and phenotypes. (most common databases are Gene Ontology (GO) and Kyoto Encyclopedia of Genes and Genomes (KEGG))
- Causal Analysis：Mendelian Randomization (MR) is a robust approach for addressing causal questions about the effects of modifiable exposures on various outcomes.
- Machine Learning Methods
  - Complex Association: canonical correlation analysis (CCA); structured sparse canonical correlation analysis (SCCA); manifold learning techniques; radiogenomics
  - Prediction Model: early, intermediate, or late of data fusion; deep learning for extracting high spatial correlations; transformer models for modeling conplex dependency within gene expression data
Public Datasets:

Dataset	Sample size	Data type
UK Biobank	~500,000	Brain, heart, abdomen, genotype, others
FHS	15,000+	Brain, heart, lung, abdomen
SHIP	12,000+	Heart, abdomen, genotype, others
ABCD	12,000+	Brain, others
CAHHM	9,500+	Brain, heart, abdomen
PNC	95,00+	Brain, others
MESA	6,800+	Heart, others
IMAGEN	2,000+	Brain, others
ADNI	1,200	Brain, others
CHCP	300+	Brain, others

Clinical Applications
- Brain
  - Modalities: T1W, MPRAGE, DTI, rfMRI, T2 FLAIR, SWI/QSM
  - Disease: Alzheimer's disease; Parkinson's disease; ischemic stroke; cognitive dysfunction
- Heart:
  - Modalities: Cine imaging; black blood, bright blood; coronary imaging; 4D flow
  - Disease: dilated cardiomyopathy; coronary artery disease; heart failure; left ventricular dilation
- Breast
  - Modalities: mammography, ultrasound, MRI
  - Disease: breast cancer; fibrocystic breast; mastitis
- Lungs
  - Modalities: NCCT
  - Disease: pulmonary artery embolism; interstitial lung disease; lung cancer; emphysema; COPD
- Abdomen
  - Modalities: NCCT, T1 mapping, T2*, dixon
  - Disease: liver fibroinflammatory; nonalcohollc fatty liver disease; diabetes and the metabolic syndrome; energy expenditure and obestiy
- Bone
  - Modalities: DXA
  - Disease: fracture; osteoporosis; osteoporotic fracture
Challenges and Prospects
- Improving Data Representation: quality control; exraction of phenotypes; voxel-wise genetic analysis; imaging modalities combination
- Introducting Other Omics: integration of multi-omics analysis
- Performing Cross-dataset Analysis: data sharing
- Exploring Machine Learning Algorithms: new algorithms developed; computational limitation
- Studying Organ Interactions: exploration of whole-body network connections

[Apr. 2023] Deep multimodal fusion of image and non-image data in disease diagnosis and prognosis: a review, Progress in Biomedical Engineering

Paper

Content:

Data Modalities: Image data (pathology images, radiology images, camera images); Non-image data (structured data, free-text data)
Multimodal fusion methods: Operation-based; Subspace-based; Attention-based; Tensor-based; Graph-based

View points:

It is difficult to compare the performance of different methods directly, since different studies were typically done on different datasets with different settings.
There is no clue that a fusion method always performance the best. The optimal fusion method might be task/data dependent.
Fusing multi-modal data typically surpassed the uni-modal counterparts in the downstream tasks, but on the other hand, some studies also mentioned that the model that fused more modalities may not always perform better than the ones with fewer modalities (I think the reason is not doing a good modal fusion)
Deep-learning methods require a large amount of training data, however, data scaricity, especially multimodal data, is a challenge in the healthcare are.
Unimodal feature extraction is a essential prerequisite for fusion, especially for multimodal heterogeneity.
Explainability is a challenge in multimodal diagnosis and prognosis.

⭐️ [Feb. 2023] Multimodal data fusion for cancer biomarker discovery with deep learning, Nature machine intelligence [Perspective]

Perspective

Paper

Background:

In oncology, massive amounts of data are being generated, ranging from molecular, histopathology, radiology to clinical records.
However, most currently AI approaches focus on single data modalities, leading to slow progress in methods to integrate complementary data types.
Development of effective multimodal fusion approaches is becoming increasingly important as a single modality might not be consistent and sufficient to capture the heterogeneity of complex diseases to tailor medical care and improve personalized medicine.

View points:

The need for multimodal data fusion in oncology: Despite huge investments in cancer research and improved diagnosis and treatments, cancer prognosis is still bleak. Predictive models based on single modalities offer a limited view of disease heterogeneity and might not provide sufficient information to stratify patients and capture the full range of events that take place in response to treatments.
Data fusion strategies for multimodal biomarker discovery: The age of precision medicine demands powerful computational techniques to handle high-dimensional multimodal patient data. Each data source has strengths and limitations in its creation, analysis and interpretation that must be addressed.
- Medical images (2D in histopathology or 3D in radiology) contain dense information that is encoded at multiple scales. So far, the best performing methods have been based on DL, and specifically convolutional neural networks.
- EHRs comprise various data types ranging from structured data such as medications, diagnosis codes, vital signs or lab tests, to unstructured data in the form of clinical notes, patient emails and detailed clinical processes. Natural language processing (NLP) algorithms that can extract useful clinical information from structured and unstructured EHR data are being developed.
- Effective fusion methods must integrate high-dimensional multimodal biomedical data, ranging from quantitative features to images and text.
- A major decision that must be made is at what specific modelling stage the data fusion takes place: (1) early, (2) intermediate or (3) late.
  - While both early and late fusion approaches are model agnostic, they are not specifically designed to cope with or take full advantage of multiple modalities. Anything between early and late fusion is defined as intermediate or joint data fusion.
  - Intermediate fusion does not merge input data, nor develop separate models for each modality, but instead involves the development of inference algorithms to generate a joint multimodal low-level feature representation that retains the signal and properties of each individual modality. Although dedicated inference algorithms must be developed for each model type, this approach attempts to exploit the advantages of both early and late fusion
Advances in multimodal biomarkers for patient stratification:
- Multi-omics data fusion: Although a single omics technology provides insights into the profile of a tumour, one technique alone does not fully capture the underlying biology. The increasing collection of large cohorts of multi-omics cancer data has spurred several efforts to fuse multi-omics data to fully grasp the tumour profile and several models for survival and risk prediction have been proposed
- Multiscale data fusion: Similar efforts as for multi-omics data fusion have been explored for multiscale data, like integrating histopathology, clinical and expression data to predict patient outcomes.
- imaging genomics and radiogenomics: When possible, molecular tumour information is nowadays used in cancer prognosis and treatment decisions. Interestingly, multiple studies have shown that phenotypes derived from medical images can act as proxies or biomarkers of molecular phenotypes such as an epidermal growth factor receptor (EGFR) mutation in lung cancer.
Current challenges and future directions for multimodal data fusion:
- Data hunger: DL models requires large amounts of data, and both data sparsity and scarcity present serious challenges, especially for biomedical data.
- Modality missing: in clinical practice, there are often different types of data missing between patients, as not all patients might have all modalities owing to cost, insurance coverage, material availability and lack of systemic collection procedures, among others.
- Biased cohort: the depth of data per patient, that is, many observables per patient are routinely generated and stored, but typical cohort sizes of patients are relatively small.
- Lack of large 'golden labelled' cohorts: the lack of large ‘golden labelled’ cohorts with matched multimodal data, mainly due to the intense labour to annotate cancer datasets combined with privacy concerns.
- Lack of model interpretation: while DL can extract predictive features from complex data, these are usually abstract, and it is not always apparent if they are clinically relevant

⭐️ [Oct. 2022] Artificial intelligence for multimodal data integration in oncology, Cancer Cell

Paper

Content:

AI methods in oncology
- Supervised methods
  - Hand-crafted methods
    - 👍：simpler architecture, lower computation cost, may require less training data, and better interpretability
    - 👎：time consuming, translate human bias to the models
  - Representation learning methods
    - 👍：their ability to extract rich feature representations from raw data, resulting in lower preprocessing cost, higher flexibility, and often superior performance over hand-crafted methods
    - 👎：reliance on pixel-level annotations, lack of interpretability
- Weakly supervised methods: this method can reduce the cost of data preprocessing and mitigate the bias and interrater variability; additionally, they are free to learn from the entire scan, that can indentify predictive features even beyond the regions typically evaluated by clinicians.
  - Graph convolutional networks
    - 👍：can incorporate larger context and spatial tissue structure
    - 👎：higher training costs and memory requirements (since the nodes cannot be processed independently)
  - Multiple-instance learning
    - 👍：no fine annotation is required
    - 👎：overlook patches' correlation
  - Vision transformers
    - 👍：be fully context aware, consider patches' correlation and context, consider spatial structure or relative distances between patches via positional encoding
    - 👎: tend to be more data hungry
- Unsupervised methods
  - Self-supervised methods
    - 👍：can learn general-purpose features, which can be beneficial for other practical tasks (transfer learning)
    - 👎: (Not mentioned in the paper)
  - Unsupervised feature analysis
    - 👍：can explore structure, similarity and common features across data points
    - 👎: (Not mentioned in the paper)
Multimodal data fusion
- Early fusion
  - 👍：only one model is trainied, simplifing the design process
  - 👎: requires a certain level of alignment or synchronization between the modalities
- Late fusion (decision-level fusion)
  - 👍：allows one to use a different model achitecture for each modality, making it suitable for systems with large data heterogeneity or modalities from different time points; be able to cope with missing or incomplete data; suitable for weak interdependencies
  - 👎: unsuitable for strong interdependencies
- Intermediate fusion
  - 👍：flexible—single-level fusion, gradual fusion, guided fusion
- There is no conclusive evidence that one fusion type is ultimately better than the others, as each type is heavily data and task specific.
Multimodal interpretability
- Histopathology: map model architecture attention or probability scores to obtain slide-level attention heatmaps
- Radiology: is similar to those used in histoloty
- Molecular data: use the integrated gradient method to analyze, which computes attribution values indicating how changes in specific inputs affect the model outputs
- Multimodal models: all previously mentioned methods can be used in multimodal models to explore interpretability within each modality. Moreover, shifts in feature importance under unimodal and multimodal settings can be investigated to analyze the impact of the multimodal context.
- While CAM- or attention-based methods can localize the predictive regions, they cannot specify which features are relevant, i.e., they can explain where but not why.
- There is no guarantee that all high-attention/attribution regions carry clinical relevance. High scores just mean that the model has considered these regions more important than others.
Multimodal data interconnection
- Morphologic associations
- Non-invasive alternatives
- Outcome associations
- Early predictors
Challenges
- Missing data
  - Synthetic data generation
  - Dropout-based methods
- Data alignment
  - Alignment of similar modalities (e.g. MRI and PET brain scans)
  - Alignment of diverse modalities (e.g. data from different scales, timepoints, or measurements)
- Transparency and prospective clinical trials

⭐️ [Sep. 2022] Multimodal biomedical AI, Nature Medicine

Paper

Content & View points:

Opportunities for leveraging multimodal data (applications)
- Personalized 'omics' for precision health
- Digital clinical trials
- Remote monitoring: the 'hospital-at-home'
- Pandemic surveillance and outbreak detection
- Digital twins
- Virtual health assistant
Multimodal data collection

Study	Country	Year started	Data modalities	Access	Sample size
UK Biobank	UK	2006	Questionnaires, EHR/clinical, Laboratory, Genome-wide genotyping, WES, WGS, Imaging, Metabolites	Open access	~500,000
China Kadoorie Biobank	China	2004	Questionnaires, Physical measurements, Biosamples, Genome-wide genotyping	Restricted access	~500,000
Biobank Japan	Japan	2003	Questionnaires, Clinical, Laboratory, Genome-wide genotyping	Restricted access	~200,000
Million Veteran Program	USA	2011	EHR/clinical, Laboratory, Genome wide	Restricted access	1 million
TOPMed	USA	2014	Clinical, WGS	Open access	~180,000
All of Us Research Program	USA	2017	Questionnaires, SDH, EHR/clinical, Laboratory, Genome wide, Wearables	Open access	1 million (target)
Project Baseline Health Study	USA	2015	Questionnaires, EHR/clinical, Laboratory, Wearables	Restricted access	10,000 (target)
American Gut Project	USA	2012	Clinical, Diet, Microbiome	Open access	~25,000
MIMIC	USA	2008-2019	Clinical/EHR, Images	Open access	~380,000
MIPACT	USA	2018-2019	Wearables, clinical/EHR, physiological, laboratory	Restricted access	~6,000
North American Prodrome Longitudinal Study	USA	2008	Clinical, Genetic	Restricted access	~1,000

Technical challenges
- How to leverage multiple different types of data and learn to relate these multiple modalities or combine them for improving prediction performance?
- Another desirable feature for multimodal learning frameworks is the ability to learn from different modalities without the need for different model architectures.
- Another important modeling challenge relates to the exceedingly high number of dimensions contained in multimodal health data, collectively termed ‘the curse of dimensionality’.
- Multimodal fusion is a general concept that can be tackled using any architectural choice.
- Many other important challenges relating to multimodal model architectures remain (for example, how to extract features from three-dimensional imaging or whole-slide images)
Data challenges
- Medical datasets are heterogeneous, which can be described along several axes, including the sample size, depth of phenotyping, the length and intervals of follow-up, the degree of interaction between participants, the heterogeneity and diversity of the participants, the level of standardization and harmonization of the data and the amount of linkage between data sources.
- Achieving diversity across race/ethnicity, ancestry, income level, education level, healthcare access, age, disability status, geographic locations, gender and sexual orientation has proven difficult in practice.
- Another frequent problem with biomedical data is the usually high proportion of missing data.
- The risk of incurring several biases is important when conducting studies that collect health data, and multiple approaches are necessary to monitor and mitigate these biases.
Privacy challenges
- The successful development of multimodal AI in health requires breadth and depth of data, which encompasses higher privacy challenges than single-modality AI models.

Benchmarks

⭐️ [Aug. 2024] MultiMed: Massively Multimodal and Multitask Medical Understanding, arXiv

Paper

Background:

Biomedical data is inherently multimodal, consisting of electronic health records, medical imaging, digital pathology, genome sequencing, wearable sensors, and more.
The application of AI tools to these multifaceted sensing technologies has the potential to revolutionize the prognosis, diagnosis, and management of human health and disease.
However, current approaches to biomedical AI typically only train and evaluate with one or a small set of medical modalities and tasks, hampering the development of comprehensive tools that can leverage the rich interconnected information across many heterogeneous.

Contributions:

Presenting MultiMed, a benchmark designed to evaluate and enable large-scale learning across a wide spectrum of medical modalities and tasks.
MultiMed consists of 2.56 million samples across medical reports, pathology, genomics, and protein data (ten medical modalities), and is structured into eleven challenging tasks, including disease prognosis, protein structure prediction, and medical question answering.
Based on MultiMed, conducting comprehensive experiments benckmarking state-of-the-art unimodal, multimodal, and multitask models.

Benchmark details:

Modality diversity
- Imaging modalities: 84,495 OCT images, 194,922 X-ray images, 617,775 CT scans, 7,023 MRI scans, and 27,560 pathology images.
- Electrophysiological data: MultiMed consists of 120,000 samples designed for the classification of imagined motor imagery time-series data.
- Molecular data: 12,560 samples of genomic sequences and 270,000 samples of scRNA-seq data to support expression prediction at the singlecell level; a total of 131,487 protein sequences for protein structure prediction.
- Text: Clinical notes that complement raw medical signals with rich, descriptive medical narratives with one million image-text pairs.
Task diversity
- Disease classification
- Brain tumor classification
- Breast cancer classification
- Radiographic findings classification
- Bone age classification
- Diabetic retinopathy classification
- Imagined motor imagery classification
- Cell type classification
- Expression prediction
- Protein structure prediction
- Medical Visual question answering

[May 2016] Data Descriptor: MIMIC-III, a freely accessible critical care database, Scientific Data

Paper Code Jupyter-Notebook

Contributions:

MIMIC-III (‘Medical Information Mart for Intensive Care’) is a large, single-center database comprising information relating to patients admitted to critical care units at a large tertiary care hospital.
Data includes vital signs, medications, laboratory measurements, observations and notes charted by care providers, fluid balance, procedure codes, diagnostic codes, imaging reports, hospital length of stay, survival data, and more.
MIMIC-III critical care database is unique and notable:
- it is the only freely accessible critical care database of its kind;
- the dataset spans more than a decade, with detailed information about individual patient care;
- analysis is unrestricted once a data use agreement is accepted, enabling clinical research and education around the world.

Benchmark details:

Patient characteristics
- MIMIC-III contains data associated with 53,423 distinct hospital admissions for adult patients (aged 16 years or above) admitted to critical care units between 2001 and 2012.
- In addition, it contains data for 7870 neonates admitted between 2001 and 2008.
- The data covers 38,597 distinct adult patients and 49,785 hospital admission. The median age of adult patients is 65.8 years (Q1–Q3: 52.8–77.8), 55.9% patients are male, and in-hospital mortality is 11.5%.
- The median length of an ICU stay is 2.1 days (Q1–Q3: 1.2–4.6) and the median length of a hospital stay is 6.9 days (Q1-Q3: 4.1–11.9). A mean of 4579 charted observations (’chartevents’) and 380 laboratory measurements (’labevents’) are available for each hospital admission.
- The top three codes across hospital admissions for patients aged 16 years and above were:
  - 414.01 (‘Coronary atherosclerosis of native coronary artery’), accounting for 7.1% of all hospital admissions;
  - 038.9 (‘Unspecified septicemia’), accounting for 4.2% of all hospital admissions; and
  - 410.71 (‘Subendocardial infarction, initial episode of care’), accounting for 3.6% of all hospital admissions.
Classes of data
- Data available in the MIMIC-III database ranges from time-stamped, nurse-verified physiological measurements made at the bedside to free-text interpretations of imaging studies provided by the radiology department.

Treatment Response Evaluation

Year	Paper	Code	Cancer	Modalities	Data Source	Patients	Fusion Mode
2024	🔗	🔗	GC	Rad, Path, Cli	In-House	429	Middle
2024	🔗	🔗	Pan-Cancer	Multi-omics	TCGA	8174	Middle
2023	🔗		ccRCC	Gene	TCGA, In-House	~1000	Middle
2023	🔗	🔗	18 solid tumor types	Path, Gene, Clin	In-House	2881	Middle
2023	🔗	🔗	GC	Rad, Clin	In-House	249	Middle
2023	🔗		NSCLC	Rad, Clin	In-House	264	Late
2022	🔗	🔗	NSCLC	Rad, Path, Gene	In-House	249	Middle
2022	🔗			Rad	UKBB, EchoNet-Dynamic, GSTFT, GSTFT CRT	62 for response predictions and 10,730 for training segmentation models	Middle
2021	🔗		NSCLC	Rad, Lab, Clin	In-House	200	Middle
2020	🔗		HCC	Rad	In-House	737	Middle

⭐️ [Aug. 2024] Predicting gastric cancer response to anti-HER2 therapy or anti-HER2 combined immunotherapy based on multi-modal data, Signal Transduction and Targeted Therapy

Name		Name	Last commit message	Last commit date
multimodal_dataset.md		multimodal_dataset.md
readme.md		readme.md
seg_dataset.md		seg_dataset.md
submit.bat		submit.bat
submit.sh		submit.sh

Year	Paper	Code	Cancer	Modalities	Data Source	Patients	Fusion Mode
2024	🔗		BLCA, BRCA, GBMLGG, LUAD, UCEC	Path, Gene	TCGA	372+956+569+453+480	Middle
2024	🔗	🔗	BRCA, BLCA, COADREAD, HNSC, STAD	Path, Gene	TCGA	869+359+296+392+317	Middle
2024	🔗	🔗	ccRCC	Path, Rad, Clin	In-House, TCGA, CPTAC	414	Middle
2024	🔗	🔗		mRNA, miRNA, methylation	BRCA, ROSMAP, LGG, KIPAN	351+875+510+658 samples	Middle
2024	🔗		Breast	Gene, Trans, Prot, Meta, Rad, Path	In-House	773	Middle
2023	🔗	🔗		Rad, Text	MIMIC-CXR, PadChest	~380k pairs	Middle
2023	🔗			Rad, Non-imaging	ADNI	248	Middle
2022	🔗		BC	Path, Clin, Gene	TCGA	196	Middle
2022	🔗	🔗	Pan-cancer	Path, Molecular profile data	TCGA	5720	Middle
2022	🔗	🔗	OC	Rad, Path, Clin	MSKCC, TCGA-OV	444	Late
2022	🔗	🔗		Imaging, Non-Imaging	OAI, ADNI	4796 (knee OA), 2577 (AD)	Middle
2022	🔗	🔗		X-ray, Non-Imaging	OAI	4796	Middle
2022	🔗		Brain	Path, Gene	TCGA-LGG, TCGA-GBM	470	Middle
2021	🔗	🔗	Breast	Path, Clin	In-House, TCGA	127+123	Middle
2021	🔗	🔗	Five cancer types	Path, Gene	TCGA (BLCA, BRCA, GBMLGG, LUAD, UCEC)	437+1022+1011+515+538	Middle
2020	🔗		Brain	MRIs	BraTS 2019	335	Middle
2020	🔗	🔗	Glioma, ccRCC	Path, Gene	TCGA-GBM, TCGA-LGG	769	Middle
2020	🔗	🔗	GBM	Path, Gene, Clin	TCGA, TCIA	447	Middle
2020	🔗	🔗	ccRCC	Rad, Path, Gene, Clin	TCGA	209	Middle
2019	🔗	🔗	Glioblastoma	Clin	SEER	20821	Early
2019	🔗	🔗	Pancancer	Clin, Gene, Path	TCGA	11160	Middle
2017	🔗		LUNA	Path, Path Reports, Gene, Proteomics	TCGA	538	Middle

Latest commit

History

Folders and files

Repository files navigation

Multimodal Medicine AI

Table of Contents

Reviews

Benchmarks

Treatment Response Evaluation

Prognosis Evaluation

Others

About

Resources

Stars

Watchers

Forks

Releases

Packages

Contributors

Languages