Skip to content

perf: optimize timestamp parsing in metadata processing #1045

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
raj-prince wants to merge 1 commit into
base: main
Choose a base branch
from
Draft

perf: optimize timestamp parsing in metadata processing #1045

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 3 additions & 5 deletions gcsfs/core.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1081,11 +1081,9 @@ def modified(self, path):
def created(self, path):
return self.info(path)["ctime"]

def _parse_timestamp(self, timestamp):
assert timestamp.endswith("Z")
timestamp = timestamp[:-1]
timestamp = timestamp + "0" * (6 - len(timestamp.rsplit(".", 1)[-1]))
return datetime.fromisoformat(timestamp + "+00:00")
@staticmethod
def _parse_timestamp(timestamp):
return datetime.fromisoformat(timestamp.replace("Z", "+00:00"))
Comment on lines +1084 to +1086

@gemini-code-assist gemini-code-assist Bot Sep 7, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

In Python 3.10, datetime.fromisoformat does not support arbitrary fractional second lengths (it only supports exactly 0, 3, or 6 digits). If a timestamp has a different number of fractional digits (such as 1, 2, 4, or 5 digits, which are explicitly covered in the new test cases like 2024-01-15T10:30:00.12Z), fromisoformat will raise a ValueError on Python 3.10.

Since gcsfs supports Python >= 3.10, this change will cause test failures and runtime crashes for Python 3.10 users.

To resolve this while maintaining maximum performance:

  1. On Python 3.11+, we can use the native datetime.fromisoformat directly (which natively supports the Z suffix and arbitrary fractional seconds, making it even faster as it avoids string replacement).
  2. On Python 3.10, we can use a fast compatibility fallback that only pads the fractional seconds when their length is not 3 or 6.

We can achieve this with zero runtime overhead by conditionally defining the @staticmethod at class definition time using sys.version_info.

    if sys.version_info >= (3, 11):
        @staticmethod
        def _parse_timestamp(timestamp):
            return datetime.fromisoformat(timestamp)
    else:
        @staticmethod
        def _parse_timestamp(timestamp):
            if timestamp.endswith('Z'):
                timestamp = timestamp[:-1]
                if '.' in timestamp:
                    frac_len = len(timestamp.rsplit('.', 1)[-1])
                    if frac_len not in (3, 6):
                        timestamp = timestamp + '0' * (6 - frac_len)
                return datetime.fromisoformat(timestamp + '+00:00')
            return datetime.fromisoformat(timestamp)


async def _info(self, path, generation=None, **kwargs):
"""File information about this path."""
Expand Down
46 changes: 46 additions & 0 deletions gcsfs/tests/test_core.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3710,3 +3710,49 @@ def test_user_agent_includes_cache_type_and_source_in_read(gcs):
for call in mock_session_request.call_args_list
]
assert any("cache_type/readahead:d" in ua for ua in user_agents)


@pytest.mark.parametrize(
"ts_str, expected_dt",
[
(
"2024-01-15T10:30:00.000Z",
datetime(2024, 1, 15, 10, 30, 0, 0, tzinfo=timezone.utc),
),
(
"2024-01-15T11:45:00.123456Z",
datetime(2024, 1, 15, 11, 45, 0, 123456, tzinfo=timezone.utc),
),
(
"2024-01-15T10:30:00Z",
datetime(2024, 1, 15, 10, 30, 0, 0, tzinfo=timezone.utc),
),
(
"2024-01-15T10:30:00.12Z",
datetime(2024, 1, 15, 10, 30, 0, 120000, tzinfo=timezone.utc),
),
(
"2024-01-15T10:30:00.1234Z",
datetime(2024, 1, 15, 10, 30, 0, 123400, tzinfo=timezone.utc),
),
],
)
def test_parse_timestamp_formats(gcs, ts_str, expected_dt):
assert gcs._parse_timestamp(ts_str) == expected_dt
assert GCSFileSystem._parse_timestamp(ts_str) == expected_dt


def test_process_object_timestamps(gcs):
metadata = {
"name": "test_obj.txt",
"size": "50",
"timeCreated": "2024-01-15T10:30:00.000Z",
"updated": "2024-01-15T11:45:00.123456Z",
}
processed = gcs._process_object("my-bucket", metadata)
assert processed["ctime"] == datetime(
2024, 1, 15, 10, 30, 0, 0, tzinfo=timezone.utc
)
assert processed["mtime"] == datetime(
2024, 1, 15, 11, 45, 0, 123456, tzinfo=timezone.utc
)
Loading