openslide required

[dzenanz]

I want to start a new project from pathology example. I copied that file into my repository and renamed it into train.py. I downloaded a subset of camelyon16 data, including the files referenced in the CSV subset sample. When I invoke it, I run into monai.utils.module.OptionalImportError: required package openslide is not installed or the version doesn't match requirement. Package openslide-python is installed. I tried this inside venv of Python 3.9 and 3.12. Any help would be appreciated. My code is publicly available here: https://github.com/Kitware/BANFF-AID/tree/wip

(.venv9) M:\Histo\BANFF-AID>pip show monai
Name: monai
Version: 1.4.0
Summary: AI Toolkit for Healthcare Imaging
Home-page: https://monai.io/
Author: MONAI Consortium
Author-email: monai.contact@gmail.com
License: Apache License 2.0
Location: m:\histo\banff-aid\.venv9\lib\site-packages
Requires: numpy, torch
Required-by: banff-aid

(.venv9) M:\Histo\BANFF-AID>pip show openslide-python
Name: openslide-python
Version: 1.4.1
Summary: Python interface to OpenSlide
Home-page: https://openslide.org/
Author:
Author-email:
License: GNU Lesser General Public License, version 2.1
Location: m:\histo\banff-aid\.venv9\lib\site-packages
Requires: Pillow
Required-by:

(.venv9) M:\Histo\BANFF-AID>python train.py --train-file pathology_train_sub.csv --valid-file pathology_validation_sub.csv --amp --pretrain --openslide --backend=numpy --root=M:/Histo/BANFF-AID
Logs and model are saved at './logs/run_numpy_bs64_cpu2_gpu0_pin0_amp1_250305-083608'.
2025-03-05 08:36:08 INFO {"root": "M:/Histo/BANFF-AID", "train_file": "pathology_train_sub.csv", "valid_file": "pathology_validation_sub.csv", "logdir": "./logs/", "region_size": 768, "grid_shape": 3, "patch_size": 224, "batch_size": 64, "n_epochs": 4, "lr": 0.001, "print_step": 10, "prob": 0.5, "use_openslide": true, "pin": false, "amp": true, "novograd": false, "cos": false, "pretrain": true, "benchmark": false, "no_save": false, "no_validate": false, "baseline": false, "optimized": false, "backend": "numpy", "num_workers": 2, "gpu": "0"}
CUDA is being used with GPU ID(s): 0
Traceback (most recent call last):
  File "M:\Histo\BANFF-AID\train.py", line 565, in <module>
    main(cfg)
  File "M:\Histo\BANFF-AID\train.py", line 325, in main
    train_dataset = PatchWSIDataset(
  File "M:\Histo\BANFF-AID\.venv9\lib\site-packages\monai\data\wsi_datasets.py", line 98, in __init__
    self.wsi_reader = WSIReader(backend=reader, level=patch_level, **kwargs)
  File "M:\Histo\BANFF-AID\.venv9\lib\site-packages\monai\data\wsi_reader.py", line 511, in __init__
    self.reader = OpenSlideWSIReader(
  File "M:\Histo\BANFF-AID\.venv9\lib\site-packages\monai\utils\module.py", line 473, in _wrapper
    raise OptionalImportError(err_msg)
monai.utils.module.OptionalImportError: required package `openslide` is not installed or the version doesn't match requirement.

[dzenanz]

I encounter the same problem on Ubuntu 22.04:

(.venv) dzenan@corista:~/Histo/BANFF-AID$ pip show monai
Name: monai
Version: 1.4.0
Summary: AI Toolkit for Healthcare Imaging
Home-page: https://monai.io/
Author: MONAI Consortium
Author-email: monai.contact@gmail.com
License: Apache License 2.0
Location: /home/dzenan/Histo/BANFF-AID/.venv/lib/python3.10/site-packages
Requires: numpy, torch
Required-by: banff-aid
(.venv) dzenan@corista:~/Histo/BANFF-AID$ pip show openslide-python
Name: openslide-python
Version: 1.4.1
Summary: Python interface to OpenSlide
Home-page: 
Author: 
Author-email: 
License: GNU Lesser General Public License, version 2.1
Location: /home/dzenan/Histo/BANFF-AID/.venv/lib/python3.10/site-packages
Requires: Pillow
Required-by: 
(.venv) dzenan@corista:~/Histo/BANFF-AID$ python train.py --train-file pathology_train_sub.csv --valid-file pathology_validation_sub.csv --amp --pretrain --openslide --backend=numpy --root=/home/dzenan/Histo/BANFF-AID/
Logs and model are saved at './logs/run_numpy_bs64_cpu2_gpu0_pin0_amp1_250305-145120'.
2025-03-05 14:51:20 INFO {"root": "/home/dzenan/Histo/BANFF-AID/", "train_file": "pathology_train_sub.csv", "valid_file": "pathology_validation_sub.csv", "logdir": "./logs/", "region_size": 768, "grid_shape": 3, "patch_size": 224, "batch_size": 64, "n_epochs": 4, "lr": 0.001, "print_step": 10, "prob": 0.5, "use_openslide": true, "pin": false, "amp": true, "novograd": false, "cos": false, "pretrain": true, "benchmark": false, "no_save": false, "no_validate": false, "baseline": false, "optimized": false, "backend": "numpy", "num_workers": 2, "gpu": "0"}
CUDA is being used with GPU ID(s): 0
Traceback (most recent call last):
  File "/home/dzenan/Histo/BANFF-AID/train.py", line 565, in <module>
    main(cfg)
  File "/home/dzenan/Histo/BANFF-AID/train.py", line 325, in main
    train_dataset = PatchWSIDataset(
  File "/home/dzenan/Histo/BANFF-AID/.venv/lib/python3.10/site-packages/monai/data/wsi_datasets.py", line 98, in __init__
    self.wsi_reader = WSIReader(backend=reader, level=patch_level, **kwargs)
  File "/home/dzenan/Histo/BANFF-AID/.venv/lib/python3.10/site-packages/monai/data/wsi_reader.py", line 511, in __init__
    self.reader = OpenSlideWSIReader(
  File "/home/dzenan/Histo/BANFF-AID/.venv/lib/python3.10/site-packages/monai/utils/module.py", line 473, in _wrapper
    raise OptionalImportError(err_msg)
monai.utils.module.OptionalImportError: required package `openslide` is not installed or the version doesn't match requirement.

Removing --openslide from command line (and installing cucim package) allows me to proceed ... for a very short while:

(.venv) dzenan@corista:~/Histo/BANFF-AID$ python train.py --train-file pathology_train_sub.csv --valid-file pathology_validation_sub.csv --amp --pretrain --backend=numpy --root=/home/dzenan/Histo/BANFF-AID/
Logs and model are saved at './logs/run_numpy_bs64_cpu2_gpu0_pin0_amp1_250305-145331'.
2025-03-05 14:53:31 INFO {"root": "/home/dzenan/Histo/BANFF-AID/", "train_file": "pathology_train_sub.csv", "valid_file": "pathology_validation_sub.csv", "logdir": "./logs/", "region_size": 768, "grid_shape": 3, "patch_size": 224, "batch_size": 64, "n_epochs": 4, "lr": 0.001, "print_step": 10, "prob": 0.5, "use_openslide": false, "pin": false, "amp": true, "novograd": false, "cos": false, "pretrain": true, "benchmark": false, "no_save": false, "no_validate": false, "baseline": false, "optimized": false, "backend": "numpy", "num_workers": 2, "gpu": "0"}
CUDA is being used with GPU ID(s): 0
2025-03-05 14:53:51 INFO [image] 
  image shape: torch.Size([576, 3, 224, 224])
  image type:  <class 'monai.data.meta_tensor.MetaTensor'>
  image dtype: torch.float32
2025-03-05 14:53:51 INFO [label] 
  label shape: torch.Size([576, 1, 1, 1])
  label type:  <class 'monai.data.meta_tensor.MetaTensor'>
  label dtype: torch.float32
2025-03-05 14:53:51 INFO Batch size: 64
2025-03-05 14:53:51 INFO [Training] number of batches: 52
2025-03-05 14:53:51 INFO [Validation] number of batches: 48
`torch.cuda.amp.GradScaler(args...)` is deprecated. Please use `torch.amp.GradScaler('cuda', args...)` instead.
2025-03-05 14:53:51 INFO [Training] learning rate: 0.001
`torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
Traceback (most recent call last):
  File "/home/dzenan/Histo/BANFF-AID/train.py", line 565, in <module>
    main(cfg)
  File "/home/dzenan/Histo/BANFF-AID/train.py", line 416, in main
    train_counter = training(
  File "/home/dzenan/Histo/BANFF-AID/train.py", line 110, in training
    output = model(x)
  File "/home/dzenan/Histo/BANFF-AID/.venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/home/dzenan/Histo/BANFF-AID/.venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/dzenan/Histo/BANFF-AID/.venv/lib/python3.10/site-packages/monai/networks/nets/netadapter.py", line 109, in forward
    x = self.features(x)
  File "/home/dzenan/Histo/BANFF-AID/.venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/home/dzenan/Histo/BANFF-AID/.venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/dzenan/Histo/BANFF-AID/.venv/lib/python3.10/site-packages/torch/nn/modules/container.py", line 250, in forward
    input = module(input)
  File "/home/dzenan/Histo/BANFF-AID/.venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/home/dzenan/Histo/BANFF-AID/.venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/dzenan/Histo/BANFF-AID/.venv/lib/python3.10/site-packages/torch/nn/modules/batchnorm.py", line 193, in forward
    return F.batch_norm(
  File "/home/dzenan/Histo/BANFF-AID/.venv/lib/python3.10/site-packages/torch/nn/functional.py", line 2807, in batch_norm
    return handle_torch_function(
  File "/home/dzenan/Histo/BANFF-AID/.venv/lib/python3.10/site-packages/torch/overrides.py", line 1742, in handle_torch_function
    result = torch_func_method(public_api, types, args, kwargs)
  File "/home/dzenan/Histo/BANFF-AID/.venv/lib/python3.10/site-packages/monai/data/meta_tensor.py", line 282, in __torch_function__
    ret = super().__torch_function__(func, types, args, kwargs)
  File "/home/dzenan/Histo/BANFF-AID/.venv/lib/python3.10/site-packages/torch/_tensor.py", line 1648, in __torch_function__
    ret = func(*args, **kwargs)
  File "/home/dzenan/Histo/BANFF-AID/.venv/lib/python3.10/site-packages/torch/nn/functional.py", line 2822, in batch_norm
    return torch.batch_norm(
torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 882.00 MiB. GPU 0 has a total capacity of 1.95 GiB of which 86.69 MiB is free. Including non-PyTorch memory, this process has 1.44 GiB memory in use. Of the allocated memory 1.39 GiB is allocated by PyTorch, and 22.54 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)

My main computer (with more powerful graphics card) is the Windows machine. cucim does not seem to have a Windows wheel. Any suggestion?

[ericspod]

Sorry for the late reply. I have a fresh install and tried to import openslide itself. I got:

ModuleNotFoundError: Couldn't locate OpenSlide shared library. Try `pip install openslide-bin`. https://openslide.org/api/python/#installing

Perhaps you're just missing that package? We shall update our requirements if this is the case.

[ericspod]

Yes this one appears unrelated though I can't be sure without the full trace. We can have a look at it if not however.

[dzenanz]

Here is a full trace:

(.venv9) M:\Histo\BANFF-AID>python train.py --train-file pathology_train_sub.csv --valid-file pathology_validation_sub.csv --amp --pretrain --openslide --backend=numpy --root=M:/Histo/BANFF-AID
Logs and model are saved at './logs/run_numpy_bs64_cpu2_gpu0_pin0_amp1_250409-103308'.
2025-04-09 10:33:08 INFO {"root": "M:/Histo/BANFF-AID", "train_file": "pathology_train_sub.csv", "valid_file": "pathology_validation_sub.csv", "logdir": "./logs/", "region_size": 768, "grid_shape": 3, "patch_size": 224, "batch_size": 64, "n_epochs": 4, "lr": 0.001, "print_step": 10, "prob": 0.5, "use_openslide": true, "pin": false, "amp": true, "novograd": false, "cos": false, "pretrain": true, "benchmark": false, "no_save": false, "no_validate": false, "baseline": false, "optimized": false, "backend": "numpy", "num_workers": 2, "gpu": "0"}
CUDA is being used with GPU ID(s): 0
Traceback (most recent call last):
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "M:\Histo\BANFF-AID\train.py", line 565, in <module>
  File "C:\Program Files\Python39\lib\multiprocessing\spawn.py", line 116, in spawn_main
    exitcode = _main(fd, parent_sentinel)
    main(cfg)
  File "M:\Histo\BANFF-AID\train.py", line 356, in main
  File "C:\Program Files\Python39\lib\multiprocessing\spawn.py", line 126, in _main
    self = reduction.pickle.load(from_parent)
    first_sample = first(train_dataloader)
EOFError: Ran out of input
  File "M:\Histo\BANFF-AID\.venv9\lib\site-packages\monai\utils\misc.py", line 136, in first
    for i in iterable:
  File "M:\Histo\BANFF-AID\.venv9\lib\site-packages\torch\utils\data\dataloader.py", line 491, in __iter__
    return self._get_iterator()
  File "M:\Histo\BANFF-AID\.venv9\lib\site-packages\torch\utils\data\dataloader.py", line 422, in _get_iterator
    return _MultiProcessingDataLoaderIter(self)
  File "M:\Histo\BANFF-AID\.venv9\lib\site-packages\torch\utils\data\dataloader.py", line 1146, in __init__
    w.start()
  File "C:\Program Files\Python39\lib\multiprocessing\process.py", line 121, in start
    self._popen = self._Popen(self)
  File "C:\Program Files\Python39\lib\multiprocessing\context.py", line 224, in _Popen
    return _default_context.get_context().Process._Popen(process_obj)
  File "C:\Program Files\Python39\lib\multiprocessing\context.py", line 327, in _Popen
    return Popen(process_obj)
  File "C:\Program Files\Python39\lib\multiprocessing\popen_spawn_win32.py", line 93, in __init__
    reduction.dump(process_obj, to_child)
  File "C:\Program Files\Python39\lib\multiprocessing\reduction.py", line 60, in dump
    ForkingPickler(file, protocol).dump(obj)
AttributeError: Can't pickle local object 'main.<locals>.<lambda>'

(.venv9) M:\Histo\BANFF-AID>

pathology_train_sub.csv
pathology_validation_sub.csv

[ericspod]

I think this is a Windows issue where multiprocessing causes this since fork semantics aren't used. It can't copy your lambda expressions that you've used in transforms, so things like lambda x: os.path.join(cfg["root"], "training/images", x + ".tif") need to be put into functions somewhere (mayeb in conjunction with partial to set arguments) rather than lambda expressions. Your code should work under Ubuntu I think.

[dzenanz]

Agreed. On Ubuntu, I run out of GPU memory (old GPU with 2GB RAM). As we have moved away from using this example in our project, I will not try to refactor the example so it works on Windows.

I will make a PR with the openslide-bin fix.

[dzenanz]

PR here: Project-MONAI/MONAI#8419.