What google-cloud-storage versions are supported?

[jmarshall]

The recent v0.18.0 release no longer works with google-cloud-storage 1.43, due to PR #391:

INFO:__main__:API:   File "/usr/local/lib/python3.11/site-packages/cloudpathlib/__init__.py", line 8, in <module>
INFO:__main__:API:     from .gs.gspath import GSPath
INFO:__main__:API:   File "/usr/local/lib/python3.11/site-packages/cloudpathlib/gs/__init__.py", line 1, in <module>
INFO:__main__:API:     from .gsclient import GSClient
INFO:__main__:API:   File "/usr/local/lib/python3.11/site-packages/cloudpathlib/gs/gsclient.py", line 19, in <module>
INFO:__main__:API:     from google.cloud.storage import transfer_manager
INFO:__main__:API: ImportError: cannot import name 'transfer_manager' from 'google.cloud.storage' (/usr/local/lib/python3.11/site-packages/google/cloud/storage/__init__.py)

This old version of google-cloud-storage does not provide transfer_manager, which was presumably added in 2.x.

Previous cloudpathlib releases did work with this old google-cloud-storage, or at least they worked well enough to import successfully and for basic functionality to work.

Admittedly it's silly of us to still be using this old version and we should probably update our google-cloud-storage requirements, but I didn't notice any required version specified in pyproject.toml (whereas there is a listed minimum for other backends) or elsewhere:

[project.optional-dependencies]
azure = ["azure-storage-blob>=12"]
gs = ["google-cloud-storage"]
s3 = ["boto3>=1.34.0"]

Does cloudpathlib have a recommended or required minimal version for google-cloud-storage?

[Stannislav]

Same here.

It looks like it should be at least google-cloud-storage>=2.7.0:

$ pip install google-cloud-storage==2.7.0
$ python -m cloudpathlib                             
/.venv/lib/python3.10/site-packages/google/cloud/storage/transfer_manager.py:24: UserWarning:
The module `transfer_manager` is a preview feature. Functionality and API may change.
This warning will be removed in a future release.

but

$ pip install 'google-cloud-storage<2.7.0'
$ python -m cloudpathlib                  
Traceback (most recent call last):
...
ImportError: cannot import name 'transfer_manager' from 'google.cloud.storage' ...

[Stannislav]

By the way, this error happens even if one isn't using Google Cloud at all, but google-cloud-storage happens to be installed in the same environment:

$ pip install 'cloudpathlib[s3]' 'google-cloud-storage<2.7.0'
$ python -m cloudpathlib
Traceback (most recent call last):
...
ImportError: cannot import name 'transfer_manager' from 'google.cloud.storage' ...

Maybe this could be a reason to improve how optional imports from the azure/gs/s3 extras are handled?

[pjbull]

Fix incoming for this.

Separately, if we do want to add a version floor to google-cloud-storage, does anyone in this thread happen to know a good place to set that based on what versions they use? Does anyone know if (apart from transfer_manager) there's an earlier floor where cloudpathlib breaks?

[pjbull]

Ok, 0.18.1 is out. @Stannislav and @jmarshall would you mind testing that release with your use cases?

[jmarshall]

Thanks, our builds now pick up 0.18.1 and work again.

I don't have any useful wisdom about earlier floors for cloudpathlib, other than the data point that we've been using it a lot with a pinned google-cloud-storage 1.43.0 (release Nov 2021) for some time without any problems — though we don't attempt to exercise all cloudpathlib functionality, just the bits we need for our application.

It turns out that there's probably not a good reason why we have g-c-s 1.43 pinned, it's just that we pinned the then-current version a while back and haven't updated it since. So apologies for the inconvenience, though as you noted there was at least one other group impacted and probably some others that haven't commented as yet.

[Stannislav]

Thanks for the instant fix @pjbull. Things a looking good on our side so far.

[pjbull]

Closing for now since no obvious floor, and we're now covered pretty far back.