FAILED it/storage/manager_test.py::test_get[default-big_file] - requests.exceptions.ChunkedEncodingError: ('Connection broken: IncompleteRead(25147889 bytes read, 41960975 more expected)', IncompleteRead(25147889 bytes read, 41960975 more expected))
2026-08-12 15:29:14 INFO esrally.storage._transfer:_transfer.py:627 transfer interrupted: https://rally-tracks.elastic.co/big5/logs-1.ndjson.bz2 (done: 4026531840-4051632127, todo: 4051632128-4093640703).
2026-08-12 15:29:14 ERROR esrally.storage._transfer:_transfer.py:548 transfer failed: https://rally-tracks.elastic.co/big5/logs-1.ndjson.bz2
Traceback (most recent call last):
File "/opt/buildkite-agent/builds/bk-agent-prod-gcp-1786543240484552135/elastic/rally-it/.venv-3.13.7/lib/python3.13/site-packages/urllib3/response.py", line 905, in _error_catcher
yield
File "/opt/buildkite-agent/builds/bk-agent-prod-gcp-1786543240484552135/elastic/rally-it/.venv-3.13.7/lib/python3.13/site-packages/urllib3/response.py", line 1043, in _raw_read
raise IncompleteRead(self._fp_bytes_read, self.length_remaining)
urllib3.exceptions.IncompleteRead: IncompleteRead(25147889 bytes read, 41960975 more expected)
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/opt/buildkite-agent/builds/bk-agent-prod-gcp-1786543240484552135/elastic/rally-it/.venv-3.13.7/lib/python3.13/site-packages/requests/models.py", line 816, in generate
yield from self.raw.stream(chunk_size, decode_content=True)
File "/opt/buildkite-agent/builds/bk-agent-prod-gcp-1786543240484552135/elastic/rally-it/.venv-3.13.7/lib/python3.13/site-packages/urllib3/response.py", line 1267, in stream
data = self.read(amt=amt, decode_content=decode_content)
File "/opt/buildkite-agent/builds/bk-agent-prod-gcp-1786543240484552135/elastic/rally-it/.venv-3.13.7/lib/python3.13/site-packages/urllib3/response.py", line 1155, in read
data = self._raw_read(amt)
File "/opt/buildkite-agent/builds/bk-agent-prod-gcp-1786543240484552135/elastic/rally-it/.venv-3.13.7/lib/python3.13/site-packages/urllib3/response.py", line 1021, in _raw_read
with self._error_catcher():
Traceback (most recent call last):
File "/opt/buildkite-agent/builds/bk-agent-prod-gcp-1786543240484552135/elastic/rally-it/esrally/storage/_transfer.py", line 507, in _run
chunk = next(got.chunks)
File "/opt/buildkite-agent/builds/bk-agent-prod-gcp-1786543240484552135/elastic/rally-it/esrally/storage/_client.py", line 274, in iter_chunks
yield from chunks
File "/opt/buildkite-agent/builds/bk-agent-prod-gcp-1786543240484552135/elastic/rally-it/esrally/storage/_client.py", line 380, in iterate_chunks
for chunk in chunks:
^^^^^^
File "/opt/buildkite-agent/builds/bk-agent-prod-gcp-1786543240484552135/elastic/rally-it/esrally/storage/http.py", line 107, in iter_chunks
yield from response.iter_content(chunk_size=self.chunk_size)
File "/opt/buildkite-agent/builds/bk-agent-prod-gcp-1786543240484552135/elastic/rally-it/.venv-3.13.7/lib/python3.13/site-packages/requests/models.py", line 818, in generate
raise ChunkedEncodingError(e)
requests.exceptions.ChunkedEncodingError: ('Connection broken: IncompleteRead(25147889 bytes read, 41960975 more expected)', IncompleteRead(25147889 bytes read, 41960975 more expected))
A broken connection while processing a chunk should not abort entire transfer. The chunk should go back to the todo list, and another worker should attempt this again.
Noticed in integration tests: https://buildkite.com/elastic/rally-it/builds/3609
Stack trace:
A broken connection while processing a chunk should not abort entire transfer. The chunk should go back to the todo list, and another worker should attempt this again.