The v1 of list_objects can be the cause of some consistency problems. See this answer for context:
https://stackoverflow.com/a/67412931/1692709
We currently use list_objects for non recursive cases:
|
paginator = self.client.get_paginator("list_objects") |
We use the bucket filter in recursive cases:
|
for o in bucket.objects.filter(Prefix=prefix): |
We should replace both code paths with self.client.get_paginator('list_objects_v2'):
Here's the boto3 docs: https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/s3.html#S3.Client.list_objects_v2
The v1 of list_objects can be the cause of some consistency problems. See this answer for context:
https://stackoverflow.com/a/67412931/1692709
We currently use
list_objectsfor non recursive cases:cloudpathlib/cloudpathlib/s3/s3client.py
Line 147 in 80f7afd
We use the bucket filter in recursive cases:
cloudpathlib/cloudpathlib/s3/s3client.py
Line 136 in 80f7afd
We should replace both code paths with
self.client.get_paginator('list_objects_v2'):Here's the boto3 docs: https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/s3.html#S3.Client.list_objects_v2