docs: refresh translations, and translate pages in parallel - #3458
docs: refresh translations, and translate pages in parallel#3458maxisbey wants to merge 5 commits into
Conversation
translations.py translate keeps --jobs pages (default 8) in flight at once on a thread pool; each page is still its own conversation, so the model sees exactly what it did before and only the wall-clock time changes. --lang becomes optional and repeatable like the other commands, so one invocation refreshes every language under a single concurrency cap and usage total. Work is ordered page by page across languages so each language's cached system prompt is written by its first request and read by the rest. The run prints its plan (pages, languages, concurrency, model) before the first request, progress lines carry the language code, and Ctrl-C leaves at once instead of waiting for pages in flight. The API client retries rate limits and overloads a few more times since many pages now share one limit, a connection dropping mid-reply fails that page rather than the run, and the shared Markdown renderer used for heading ids is serialised behind a lock.
Re-runs scripts/docs/translations.py translate for all twelve languages to catch up with the English changes since the last refresh. Twenty-three pages per language had sections whose English changed; only those sections were retranslated and everything else is carried over byte for byte. run/authorization.md in es, ru and uk was retranslated whole (--pages) because the section-scoped run kept reproducing a stale phrase from the previous translation. Nothing under docs/, the per-language instructions or glossaries changed.
📚 Documentation preview
|
httpx is only present through the optional translate group (the SDK itself depends on httpx2), so a top-level import broke every environment that loads the docs tooling without that group. Import it next to anthropic, where its errors are caught.
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
![cubic-dev-ai[bot]](https://avatars.githubusercontent.com/in/1082092?s=60&v=4){kind=small}
There was a problem hiding this comment.
All reported issues were addressed across 279 files
Note: This PR contains a large number of files. cubic selects up to 200 of the highest-priority eligible files for this review, so some files may not have been reviewed.
Re-trigger cubic
When the API rejects the credentials part-way through a parallel run, stop starting pages but still collect the ones already in flight and write those that succeed, then report usage and exit 2. Previously a page that had already finished in the same batch could be dropped depending on set iteration order. The test fake's call counter is now guarded by a lock.
There was a problem hiding this comment.
All reported issues were addressed across 2 files (changes from recent commits).
Reply with feedback, questions, or to request a fix.
Re-trigger cubic
The test added in the previous commit assumed the rejected page is seen before its sibling lands; with four pages and two in flight the loop may legitimately start a third page first, which then waited alone at the test's barrier. Split it into two order-independent tests: a sibling that lands beside a rejected page is written (exactly two pages in the run), and no further page starts once the rejection is seen (both in flight rejected). The worker now hands a ConfigError back as a value like a PageError, so that page's usage is counted and the loop needs no except.
There was a problem hiding this comment.
I reviewed the PR again at the latest commits and found no bugs; the two points from my earlier review are resolved in the code as it now stands, though a human pass is still worthwhile given the volume of regenerated translation content.
What was reviewed: the reworked stop-on-rejected-credentials flow (ConfigError now returned as a value, the done batch fully drained and successful pages written before the run exits 2) and the lock-guarded test-fake counter both check out at HEAD; also checked the window-refill loop for lost or double-submitted jobs (queue/running bookkeeping is consistent), the os._exit(130) interrupt path (usage printed first, no file left half-written since write_text is atomic per page), and that all new write_text calls pass encoding=. The 276 regenerated i18n pages are machine-produced and section-scoped per the tool's provenance hashes — their translation quality is the part no automated review here can vouch for.
Extended reasoning...
Overview
The PR has five commits: the parallel-translation rework of scripts/docs/translations.py (repeatable --lang, --pages, --jobs with a manually refilled ThreadPoolExecutor window, page-major interleaving via zip_longest, a render lock, lazy httpx import with HTTPError mapped to PageError), matching tests in tests/docs/test_translations.py, and ~276 regenerated pages under i18n/*/pages/. The last two commits (b5557cb, fde648c) landed after or alongside my previous review and directly address the two nits I posted: the results loop at scripts/docs/translations.py:997-1027 now records the first ConfigError, keeps draining the wait(...).done set so sibling pages that completed are written, and raises the held error only after the loop; and the RejectedTogether fake in tests/docs/test_translations.py increments its call counter under threading.Lock. Both fixes are verified in the code at HEAD, not taken from the commit messages.
Security risks
None material. The change touches only internal docs tooling, tests, and generated markdown — no src/ code, no auth/crypto/permission paths, no dependency changes. The credentials involved (ANTHROPIC_API_KEY) are read from the environment as before; error messages print the API's rejection text but no key material. os._exit(130) on KeyboardInterrupt is deliberate and commented (non-daemon pool threads), and each page write is a single write_text with explicit encoding, so an interrupt cannot leave a truncated page.
Level of scrutiny
Moderate for the tool code, low for the generated pages. The executor loop is genuinely subtle — it took two follow-up commits to get the stop-path and its tests deterministic — so I re-read the final version rather than trusting the commit messages: the refill condition (queue and stopped is None) or running cannot spin (either something is submitted or wait blocks on a non-empty running), jobs are popped exactly once, and stopped = stopped or result preserves the first failure. The generated i18n pages are the bulk of the diff but are machine-produced under the tool's own provenance-hash discipline; no automated review can meaningfully vouch for translation quality across twelve languages, which is the main reason I defer rather than approve. cubic-dev-ai also left an inline comment at the ConfigError-handling line after my last review; the subsequent commit reworked exactly that code, but I cannot read the comment's text to confirm it is fully addressed — another reason not to approve over it.
Other factors
The bug-hunt run over the final version exited on a dry streak with no findings. The tests are strong for this kind of code: barrier-synchronized fakes proving genuine two-in-flight interleaving, order-independent split tests for the stop path, and snapshot-asserted output; the repo's 100%-branch-coverage gate applies to scripts/ tests too. I chose to speak despite my prior review because the post-review commit fde648c substantively changed the code I had flagged, and confirming those fixes check out is new information the author has not yet received.
1 participant
Re-runs
scripts/docs/translations.py translatefor all twelve languages to catch up with English changes since #3379, and teaches the tool to do that in one parallel run.Motivation and Context
Twenty-three pages per language had sections whose English source changed (mostly #3443's move away from presenting the in-memory client as the way to connect, plus the session-limit, redirect and
validate_token_resourceadditions), so translated sites were showing the "translation behind the English page" notice on those. As before, only the changed sections were retranslated and everything else is carried over byte for byte, which is why the diff is section-scoped despite touching 276 generated files. One exception:run/authorization.mdin es, ru and uk was retranslated whole with--pages, because the section-scoped run deterministically reproduced a stale phrase from the previous translation ("in-memory client" where the English now says "in-memory test client").The first commit is the tool change that made the run quick:
translatekeeps--jobs Npages in flight on a thread pool (default 8). Each page is still its own conversation, so the model sees exactly what it did before; only wall-clock time changes. This refresh (276 pages) took about six minutes at--jobs 32.--langis optional and repeatable likestatus/stage, so one invocation covers every language under one concurrency cap and one usage total. Work is ordered page by page across languages so each language's cached system prompt is written by its first request and read by the rest.Nothing under
docs/,i18n/*/instructions.mdori18n/*/glossary.jsonchanged.How Has This Been Tested?
translations.py statusreports 0 missing / 0 outdated / 51 current for every language.scripts/docs/build.shbuilds the English site strictly (cross-reference and render-order checks green) and all twelve language sites with 0 warnings.main(3120 sections; the three whole-page retranslations excepted), and that no retranslated section came back untranslated, in the wrong script, truncated, or with prompt text leaked into it.--jobscap, rejected credentials with pages in flight, and a multi-language run;tests/docspasses, pyright and ruff clean.Breaking Changes
None (the translation tool's CLI is internal docs tooling).
Types of changes
Checklist
Additional context
Generated pages are never hand-edited; if a reviewer spots a wording problem, the fix goes into that language's
instructions.md/glossary.jsonand the page is re-run.AI Disclaimer