fix(get-vault-secrets): bound and retry the OIDC token request - #2249
Conversation
The request to GitHub's OIDC endpoint intermittently times out, and @actions/http-client does not retry socket timeouts, so a single blip fails the whole job. Retry up to three times with exponential backoff. Jobs missing `id-token: write` still fail on the first attempt, so a misconfiguration surfaces as quickly as it does today.
core.getIDToken() allows no timeout and waits three minutes on a dead socket, so retrying around it would still stall for minutes. Request the token directly instead, with a 30s abortable timeout per attempt. Native fetch does not honour the proxy variables @actions/http-client reads, so proxied runners keep using core.getIDToken().
NickAnge
left a comment
There was a problem hiding this comment.
Hey @pracucci . I like the idea. Answering your comments
Timeout and attempt count are hardcoded rather than exposed as inputs — happy to make either configurable.
I do not think these should be configurable by the client. They should be defined at the source. In my opinion, the logic is not complex enough to require different values. Let’s start with these numbers and adjust them if needed.
30s is deliberately generous for what is normally a sub-second call; the goal is just to stop waiting minutes. Say the word if you'd prefer it tighter.
No, I think 30 seconds is fine for now.
The script is embedded in action.yaml, so there's nowhere natural to land a regression test. If you'd like one, I can extract it to a .js file (which would also bring it under eslint) in a follow-up.
This one is important. I would like to see this either in this PR or in a quick follow-up. Since the logic is becoming more complex, I think it would be good to add some tests to verify that it works as expected.
…ble module The retry loop was embedded in action.yaml, where no test could reach it. Move it to get-id-token.js and require it from github-script, the same way create-github-app-token already loads its script, then cover the retry, timeout, proxy and missing-permission paths with bun tests. Plain JS files get the Node globals in the eslint config, so the extracted module lints instead of failing no-undef on `module` and `process`.
@NickAnge Thanks for your review. Take a look at the last commit, please. In the meanwhile I will look for a way to test it end-to-end without having to merge this PR. |
|
@NickAnge To test it, I've pushed this commit that triggered this CI run: the two tests passed. |
NickAnge
left a comment
There was a problem hiding this comment.
Overall looks good. Just a small nit at the log message
…rror Either variable being unset triggers the error, so "and" was misleading.
Why
The
Retrieve API KEYstep intermittently fails while fetching the GitHub OIDC token, taking the whole job down with it:Two things make this worse than it needs to be:
Request timeout: <path>is@actions/http-client's socket timeout. ItsallowRetriesonly covers retryable HTTP status codes on idempotent verbs — a socket timeout rejects out ofrequestRawand escapes the retry loop entirely. A single network blip is fatal and the only recovery is a human re-running the job.core.getIDToken()exposes no way to shorten it. The failure we observed burned ~136s before giving up.In
grafana/deployment_toolsthis is routine: ~40 failed runs of one workflow over 3.5 days, every one of them this step. That workflow fans out to 9 jobs that each call this action and is triggered often enough that it ran 100 times in 53 minutes, so it hits the endpoint hard and notices quickly — but nothing about the failure is specific to that repo.What
ACTIONS_ID_TOKEN_REQUEST_URLwithAbortSignal.timeout, bounding each attempt at 30s.action.yamlintoget-id-token.js, loaded withrequire()fromgithub-scriptthe waycreate-github-app-tokenalready loads its script, so the retry logic can be tested and linted.Worst case is now ~93s and gives three chances, against ~136s and one chance today. The common case — a single stall — recovers in about a second instead of failing the job.
Racing a promise against
core.getIDToken()was the smaller change, but the abandoned request keeps its socket (and the event loop) alive until the 3-minute timeout fires, so the step would still hang for minutes. Hence the direct call.Two behaviours are deliberately preserved:
fetchignores thehttps_proxy/HTTPS_PROXYvariables that@actions/http-clienthonours, so when a proxy is configured we still go throughcore.getIDToken()(unbounded, as today) rather than silently breaking those runners.id-token: write. No token endpoint exists, so retrying only delays a deterministic error. Those still fail on the first attempt with the same guidance as before.Verification
get-id-token.test.jsdrives the module with a fakecore,fetchandsleep:setFailed+ existing guidance, backs off 1s then 2sfetchrejectionsetFailedwith the original messagehttps_proxy/HTTPS_PROXY)core.getIDToken(), retried the same wayid-token: writesetFailedimmediately, no retriesTwo more cover the request itself: the audience is appended for both instances (
...&audience=vault-github-actions-grafana-ops,...-dev), and the request carries theBearertoken and anAbortSignal. The token is masked withcore.setSecretin every success path.To check the tests have teeth I mutated the module — dropped the retry, flattened the backoff, removed the masking, inverted the proxy check, removed the timeout, ignored the HTTP status, pinned the audience — and each mutation fails at least one test.
bun test actions/get-vault-secretsruns as a newunit-testjob intest-get-vault-secrets.yaml; it needs nobun install, as the test only importsbun:test.eslint,prettier --checkandactionlintpass.Notes for reviewers
env,fetchandsleepare parameters with defaults only so the tests can drive the loop; the action passes{ core }and nothing else.eslintneeded the Node globals in the shared config (globals.nodefor**/*.js), which is the only reasonpackage.jsonandbun.lockchange here. Without itmoduleandprocesstripno-undef— the same reasoncreate_github_token.jsis excluded from theeslintpre-commit hook today.