Skip to content

perf: optimize timestamp parsing in metadata processing - #1045

Draft
raj-prince wants to merge 1 commit into
fsspec:mainfrom
raj-prince:optimize_parse_timestamp
Draft

perf: optimize timestamp parsing in metadata processing#1045
raj-prince wants to merge 1 commit into
fsspec:mainfrom
raj-prince:optimize_parse_timestamp

Conversation

@raj-prince

Copy link
Copy Markdown
Collaborator

Description:
Optimizes GCSFileSystem._parse_timestamp by replacing legacy string slicing and zero-padding logic with @staticmethod and datetime.fromisoformat(timestamp.replace("Z", "+00:00")).

This yields a ~2.2x speedup on timestamp parsing and reduces total _process_object latency by ~33%.

Rationale & Backward Compatibility

  • The manual zero-padding and rsplit logic was originally added as a workaround for older Python versions (<3.10) where fromisoformat() was restrictive about fractional-second padding.
  • Since gcsfs requires python >= 3.10, standard GCS RFC 3339 timestamps (milliseconds .000Z, microseconds .ffffffZ, and whole seconds Z) are natively supported by fromisoformat() when replacing "Z" with "+00:00".
  • Reference: Python 3.10 datetime.fromisoformat() Specification

Benchmarks (timeit, 500k iterations)

Scenario Legacy Optimized Speedup
_parse_timestamp (milliseconds .000Z) 0.298 µs 0.136 µs 2.19x (54.4% less CPU)
_parse_timestamp (microseconds .123456Z) 0.269 µs 0.139 µs 1.93x (48.2% less CPU)
_parse_timestamp (whole seconds Z) 0.250 µs 0.130 µs 1.92x (47.9% less CPU)
Full _process_object (2 timestamps) 0.870 µs 0.584 µs 1.49x (32.9% less CPU)

Testing

  • Added test_parse_timestamp_formats and test_process_object_timestamps to gcsfs/tests/test_core.py covering all GCS timestamp formats and precisions.

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request refactors the _parse_timestamp method in gcsfs/core.py to be a static method and simplifies its implementation using datetime.fromisoformat. It also adds comprehensive unit tests to verify various timestamp formats. However, the simplified implementation introduces a compatibility issue on Python 3.10, where datetime.fromisoformat does not support arbitrary fractional second lengths (such as 1, 2, 4, or 5 digits), leading to potential runtime crashes and test failures. A conditional fallback based on the Python version is recommended to maintain compatibility.

Comment thread gcsfs/core.py
Comment on lines +1084 to +1086
@staticmethod
def _parse_timestamp(timestamp):
return datetime.fromisoformat(timestamp.replace("Z", "+00:00"))

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

In Python 3.10, datetime.fromisoformat does not support arbitrary fractional second lengths (it only supports exactly 0, 3, or 6 digits). If a timestamp has a different number of fractional digits (such as 1, 2, 4, or 5 digits, which are explicitly covered in the new test cases like 2024-01-15T10:30:00.12Z), fromisoformat will raise a ValueError on Python 3.10.

Since gcsfs supports Python >= 3.10, this change will cause test failures and runtime crashes for Python 3.10 users.

To resolve this while maintaining maximum performance:

  1. On Python 3.11+, we can use the native datetime.fromisoformat directly (which natively supports the Z suffix and arbitrary fractional seconds, making it even faster as it avoids string replacement).
  2. On Python 3.10, we can use a fast compatibility fallback that only pads the fractional seconds when their length is not 3 or 6.

We can achieve this with zero runtime overhead by conditionally defining the @staticmethod at class definition time using sys.version_info.

    if sys.version_info >= (3, 11):
        @staticmethod
        def _parse_timestamp(timestamp):
            return datetime.fromisoformat(timestamp)
    else:
        @staticmethod
        def _parse_timestamp(timestamp):
            if timestamp.endswith('Z'):
                timestamp = timestamp[:-1]
                if '.' in timestamp:
                    frac_len = len(timestamp.rsplit('.', 1)[-1])
                    if frac_len not in (3, 6):
                        timestamp = timestamp + '0' * (6 - frac_len)
                return datetime.fromisoformat(timestamp + '+00:00')
            return datetime.fromisoformat(timestamp)

@codecov

codecov Bot commented Sep 7, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 90.18%. Comparing base (dd48b07) to head (5956280).
⚠️ Report is 8 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #1045      +/-   ##
==========================================
+ Coverage   90.10%   90.18%   +0.07%     
==========================================
  Files          16       16              
  Lines        3679     3739      +60     
==========================================
+ Hits         3315     3372      +57     
- Misses        364      367       +3     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@raj-prince
raj-prince marked this pull request as draft September 7, 2026 19:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant