Skip to content

S3: Use list_objects_v2 to list objects #155

@pjbull

Description

@pjbull

The v1 of list_objects can be the cause of some consistency problems. See this answer for context:
https://stackoverflow.com/a/67412931/1692709

We currently use list_objects for non recursive cases:

paginator = self.client.get_paginator("list_objects")

We use the bucket filter in recursive cases:

for o in bucket.objects.filter(Prefix=prefix):

We should replace both code paths with self.client.get_paginator('list_objects_v2'):

Here's the boto3 docs: https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/s3.html#S3.Client.list_objects_v2

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions