Skip to content

Commit 8bfa34e

Browse files
committed
Add 3D segmentation calibration tutorial
Signed-off-by: Theo Barfoot <theo.barfoot@gmail.com>
1 parent b3c6c99 commit 8bfa34e

3 files changed

Lines changed: 1449 additions & 0 deletions

File tree

README.md

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -101,6 +101,13 @@ Each user is responsible for checking the content of datasets and the applicable
101101
You can read details about adding a tutorial in our [CONTRIBUTING GUIDELINES](CONTRIBUTING.md).
102102

103103
### 4. List of notebooks and examples
104+
#### <ins>**Calibration**</ins>
105+
##### [Training and evaluating calibrated segmentation models](./calibration/segmentation_calibration.ipynb)
106+
This tutorial trains two models on complete 3D MRI volumes from the Medical Segmentation Decathlon
107+
`Task04_Hippocampus` dataset to demonstrate calibration metrics, reliability diagrams, the Ignite calibration
108+
handler, and L1-ACE auxiliary training. It compares a validation-selected hard L1-ACE configuration with a controlled
109+
segmentation baseline on separate held-out test volumes.
110+
104111
#### <ins>**2D classification**</ins>
105112
##### [mednist_tutorial](./2d_classification/mednist_tutorial.ipynb)
106113
This notebook shows how to easily integrate MONAI features into existing PyTorch programs.

calibration/README.md

Lines changed: 25 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,25 @@
1+
# Segmentation model calibration
2+
3+
This folder contains a self-contained tutorial for evaluating and improving the marginal class-wise calibration of
4+
semantic segmentation models. It trains on complete 3D volumes from the Medical Segmentation Decathlon
5+
`Task04_Hippocampus` MRI dataset. The notebook downloads the approximately 28 MB archive automatically and reuses
6+
the directory configured by `MONAI_DATA_DIRECTORY`.
7+
8+
The notebook demonstrates MONAI's calibration metrics, low-level bin statistics, Ignite handler, and hard- and
9+
soft-binned L1 Average Calibration Error losses. It uses complete-volume train/validation/test cohorts and compares
10+
a segmentation baseline with one hard L1-ACE configuration chosen in validation-only preliminary experiments for
11+
its calibration improvement with minimal Dice reduction. The focused comparison discusses the associated
12+
publication's 1:1:1 objective, finite-bin estimates, and the limitations of auxiliary calibration training.
13+
14+
For a one-epoch CI smoke test, run:
15+
16+
```bash
17+
export MONAI_DATA_DIRECTORY=/path/to/persistent/monai-data
18+
./runner.sh -t calibration/segmentation_calibration.ipynb
19+
```
20+
21+
`runner.sh` rewrites `max_epochs` and `val_interval` to one. To reproduce the saved full experiment, open the notebook
22+
in Jupyter and run all cells without that rewrite. A CUDA GPU is strongly recommended for the two 3D training runs.
23+
24+
The notebook requires a MONAI build containing `HardL1ACELoss` and `SoftL1ACELoss`. Until those APIs are available in
25+
an official MONAI package, run it in an environment with the corresponding MONAI core contribution installed editable.

calibration/segmentation_calibration.ipynb

Lines changed: 1417 additions & 0 deletions
Large diffs are not rendered by default.

0 commit comments

Comments
 (0)