fix: improve terminal log diagnostics and HTTP timeout handling - #1712
Merged
Conversation
SAKURA-CAT
changed the title
High-frequency flush/fsync on every write causes excessive disk I/O and can block the writer thread on network file systems like NAS. Flushing is now deferred to close(), with a small risk of losing buffered data on abnormal exit. A future Go rewrite will decouple local persistence from remote upload.
Collaborator
|
最大风险就是训练进程 crash 后，尾部的 buffer 数据未持久化也未上报，这一部分指标数据会丢失，但既然 crash 的话这个抖动结果应该是可接受的 |
Nexisato
approved these changes
Jul 24, 2026
Nexisato
approved these changes
Jul 24, 2026
Nexisato
approved these changes
Jul 24, 2026
Collaborator
|
TODO 一下 cloud-only 模式，NAS 的高频 IO 是挺难受，不过无法做持久化，crash 即丢点 |
SAKURA-CAT
approved these changes
Jul 24, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
timeout=None时，TimeoutHTTPAdapter配置的默认超时未生效的问题。Changes
TerminalProxy、BackgroundConsumer、CorePython、Dispatch和HttpRecordSender的关键阶段增加 debug 日志，记录批次数量和 record 编号范围，且不写入 TTY。timeout=None时，由TimeoutHTTPAdapter注入配置的默认超时。DataStoreWriter.write()不再每次写入后立即 flush，仍由ensure_flushed()和close()执行显式刷新。Testing
uv run pytest tests/unit/sdk/internal/pkg/client/test_client_session.py：12 passeduv run ruff check：All checks passedNotes
ensure_flushed()时仍会刷新缓冲区。