fix(schema): retry transient network errors when downloading schemas - #4633
Merged
Conversation
Schema specs are downloaded from a GitHub release asset served through a
CDN that intermittently resets connections under load ("Remote end closed
connection without response"). The download had no retry, so a single
reset failed the entire `cfn-lint --update-specs` run. In CI the unitlint
matrix fires many concurrent downloads with fail-fast, so one reset
cancels every job in the run.
Wrap the urlopen/urlretrieve calls in the URL helpers with a bounded
exponential backoff (3 attempts, 0.5s base). Connection-level errors and
HTTP 429/5xx are retried; deterministic 4xx responses still fail fast.
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #4633 +/- ##
=======================================
Coverage 94.57% 94.58%
=======================================
Files 432 432
Lines 15395 15417 +22
Branches 2984 2986 +2
=======================================
+ Hits 14560 14582 +22
Misses 456 456
Partials 379 379
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Schema specs are downloaded from a GitHub release asset (
resource-provider-enhanced-schemas→schemas-cfn-lint.zip) served through a CDN that intermittently resets connections under load (Remote end closed connection without response). The download had no retry, so a single reset failed the wholecfn-lint --update-specsrun. In CI theunitlintmatrix fires ~18 concurrent downloads withfail-fast: true, so one reset cancels every job in the run.This wraps the network calls in the URL helpers with a small shared retry/backoff helper:
URLError,ConnectionResetError/RemoteDisconnected, timeouts — allOSError) and HTTP 429/5xxget_url_retrieve(schema zip),get_url_content(version.json), andurl_has_newer_version(HEAD check)This reduces transient CI flakiness; it does not make the download bulletproof against a sustained outage.
Additional change: FindInMap guardrail test
Follow-up to #4628 (which made
Fn::FindInMapmaxItemsunconditionally 4 soDefaultValueworks withoutAWS::LanguageExtensions). Adds a regression test asserting that a 4th element which is not a{DefaultValue: ...}object is still rejected viaprefixItems[3](is not of type 'object') even without the transform — locking in that relaxing the arity did not weaken the 4th-element shape check.Validation
pytest test/unit/module/helpers -qandpytest test/unit/rules/functions/test_find_in_map.py -qruff check/ruff format --checkon the changed filesmypy src/cfnlint/helpers.pycfn-lint --update-specs --forcesucceeds locally