Paper | BibTeX | BTReport-BraTS23 Dataset | Clinical Evaluation Platform
Juampablo E. Heras Rivera*, Dickson T. Chen*, Tianyi Ren, Daniel K. Low,
Jacob Ruzevick, Asma Ben Abacha, Alberto Santamaria-Pang, Mehmet Kurt
*equal contribution
|
KurtLab, University of Washington |
|
BTReport is an open-source framework for brain tumor radiology report generation using quantitative neuroimaging features. BTReport first extracts clinically relevant features (patient metadata, VASARI features, midline shift) using a patients scan and tumor segmentation mask, then uses large language models for report formatting.
The framework consists of four components:
- patient_metadata — demographic and clinical information (e.g., age, sex, diagnosis, outcome).
- vasari_features — standardized VASARI features.
- midline_shift — quantitative estimation of 3D midline shift using a deep learning registration approach.
- llm_report_generation — LLM synthesis of structured radiology reports grounded in deterministic features.
Results
MASS EFFECT & VENTRICLES:
There is an approximately 10 mm leftward midline shift at the level of the fourth ventricle. The right lateral ventricle, including the inferior horn, is effaced by tumor, whereas the left lateral ventricle is enlarged, producing marked ventricular asymmetry. No tonsillar herniation is seen, and the basal cisterns remain patent.
BRAIN / ENHANCEMENT:
A solitary, markedly enhancing lesion centered in the right cortex involving the parietal, occipital, and temporal lobes measures 7.1 × 4.9 × 5.9 cm (AP × TV × CC). The enhancing rim is thick (>3 mm). The lesion demonstrates ependymal invasion of the right lateral ventricle and extends into deep right-sided structures, including the thalamus, caudate, putamen, pallidum, and hippocampus, with associated deep white matter infiltration. Multiple small enhancing satellite nodules are present adjacent to the main mass. Approximately 6% of the lesion is non-enhancing necrotic tissue. A large surrounding FLAIR-hyperintense region consistent with vasogenic edema comprises the majority of the lesion volume (66%) but does not cross the midline. The enhancing component remains confined to the right side.
We provide a companion dataset which augments BraTS'23 imaging with BTReport-generated reports to further research in neuro-oncology report generation.
The dataset contains reports generated with gpt-oss:120b and llama3:70b, and can be found in the BTReport-BraTS23 HuggingFace.
from datasets import load_dataset # after pip install datasets
# Load the dataset from HuggingFace
dataset = load_dataset("kurtlab/BTReport-BraTS23")
# Access a specific subject's reports
example = dataset['train'][0]
print(f"Subject: {example['subject_id']}")
print(f"Llama3 Report: {example['Predicted Report (llama3:70b)']}")
print(f"Llama3 Report: {example['Predicted Report (gpt-oss:120b)']}")
Installation is divided into three steps:
- Downloading all accompanying Singularity images
- Setting up Ollama and downloading the LLMs used for inference and evaluation
- Creating a conda environment
See INSTALL.md for full installation instructions on HPC systems.
BTReport requires each subject's data be separated into individual folders. Each subject folder should include at least two niftii files: a T1 scan with file ending -t1n.nii.gz , and the corresponding tumor segmentation mask with file ending -seg.nii.gz. Following BraTS convention, segmentations should contain NCR, ED, and ET subregions. Optionally, an additional metadata.json file may be provided containing the ground-truth Findings section and/or supplementary patient metadata. When included, the ground-truth Findings section should be stored under the 'Clinical Report' key.
data/
├── subject_001/
│ ├── <subject_identifier>-t1n.nii.gz
│ ├── <subject_identifier>-seg.nii.gz
│ └── metadata.json
├── subject_002/
│ ├── <subject_identifier>-t1n.nii.gz
│ └── <subject_identifier>-seg.nii.gz
└── ...
-
Change the paths in
docs/btreport_paths.shto match those set in INSTALL.md, then runsource docs/btreport_paths.shThis will set the paths as environment variables and validate that each provided path points to a file/dir.
-
On a GPU allocation, start the Ollama server in the background (e.g., within a detached tmux session).
tmux new -d -s ollama3 "python3 -m btreport.ollama_server start-ollama --gpus 0,1"
module load apptainer
conda activate BTReport
python3 -m btreport.run_all_reports \
--root_folder <path/to/root/folder> \
--llm llama3:70bThis command will extract all of the relevant metadata, then generate reports for each subject individually. Additionally, if a ground truth report is provided for a subject, the paired ground truth and predicted reports will be saved to root_folder/merged_reports_btreport.json for evaluation.
To evaluate the quality of reports, we compare generated reports to ground truth clinical reports when they are available. The following command calculates the evaluation metrics included in the BTReport manuscript, taking root_folder/merged_reports_btreport.json as an input.
python3 -m btreport.eval_json \
--json </path/to/merged_reports_btreport.json> \
--real_report_key "Clinical Report" \
--synthetic_report_key "Predicted Report (gpt-oss:120b)" \
--parse-real \
--parse-synthetic \
--devices 0,1
python3 -m btreport.eval_json \
--json </path/to/merged_reports_btreport.json> \
--real_report_key "Clinical Report" \
--synthetic_report_key "Predicted Report (llama3:70b)" \
--devices 0,1 \
--parse-real \
--no-parse-synthetic \
--do_details
cliknicalannot.mov
We developed an evaluation platform to collect radiologist feedback for future iterations of this project. The platform consists of an interactive image viewer pane followed by a structured questionnaire.
The viewer pane supports:
- Multi-view inspection of the four BraTS MRI sequences (T1n, T1c, T2w, T2-FLAIR)
- Overlay of the three-region tumor segmentation mask
- Visualization of ideal and deformed subject midlines for midline-shift assessment
- 2D in-plane distance measurement
After becoming familiar with the patient’s imaging, radiologists are presented with four synthetically generated reports in random order. They complete a series of multiple-choice and Likert-scale questions assessing report quality, hallucinations, and completeness.
Finally, radiologists rank the reports from most useful to least useful and may optionally provide free-text comments. All responses are stored directly on a secure server.
Check out the platform at: https://clinical-evaluation-btreport.onrender.com/
Note
The clinical evaluation platform takes around 5 minutes to load all necessary components after first opening.
@InProceedings{rivera2026btreport,
title = {BTReport: A Framework for Brain Tumor Radiology Report Generation with Clinically Relevant Features},
author = {Heras Rivera, Juampablo E. and Chen, Dickson T. and Ren, Tianyi and Low, Daniel K. and Ruzevick, Jacob and Ben Abacha, Asma and Santamaria-Pang, Alberto and Kurt, Mehmet},
booktitle = {Proceedings of The 9th International Conference on Medical Imaging with Deep Learning},
pages = {1445--1472},
year = {2026},
editor = {Huo, Yuankai and Gao, Mingchen and Kuo, Chang-Fu and Jin, Yueming and Deng, Ruining},
volume = {315},
series = {Proceedings of Machine Learning Research},
month = {08--10 Jul},
publisher = {PMLR},
pdf = {https://raw.githubusercontent.com/mlresearch/v315/main/assets/heras-rivera26a/heras-rivera26a.pdf},
url = {https://proceedings.mlr.press/v315/heras-rivera26a.html},
abstract = {Recent advances in radiology report generation (RRG) have been driven by large paired image-text datasets; however, progress in neuro-oncology RRG has been limited due to a scarcity in open paired image-report datasets. Here, we introduce BTReport, an open-source framework for brain tumor RRG that constructs natural language radiology reports using reliably extracted quantitative imaging features. Unlike existing approaches that rely on general-purpose or fine-tuned vision-language models for both image interpretation and report composition, BTReport performs deterministic feature extraction of clinically-relevant features, then uses large language models only for syntactic structuring and narrative synthesis. By separating RRG into deterministic feature extraction and report generation stages, synthetically generated reports are completely interpretable and contain reliable numerical measurements, a key component lacking in existing RRG frameworks. We validate the clinical relevance of BTReport-derived features, and demonstrate that BTReport-generated reports more closely resemble reference clinical reports when compared to existing baseline RRG methods. To further research in neuro-oncology RRG, we introduce BTReport-BraTS, a companion dataset that augments BraTS imaging with synthetic radiology reports generated with BTReport, and BTReview, a web-based platform for validating the clinical quality of synthetically generated radiology reports.}
}