Partial model download is never repaired, resolveRepo's local-cache check accepts incomplete folders

[ericlewis]

Summary

If a TTSKit model download is interrupted partway (app backgrounded/killed, network drop), the partially-downloaded folder permanently poisons the cache: every subsequent setupModels/TTSKit.download call short-circuits on a local-cache check that accepts any non-empty directory as complete, so the missing files are never fetched and loadModels() fails with Core ML compile errors forever. The only way out is manually deleting the model folder.

Environment

• argmax-oss-swift 1.0.0 (25c6299), product TTSKit
• iPhone 17 Pro, iOS 27 beta
• Default config (.qwen3TTS_0_6b, repo argmaxinc/ttskit-coreml), foreground download session

Repro

1. Start a fresh model download (setupModels(download: true) or first generate()).
2. Kill or background the app mid-download (foreground URLSession, so transfers stop).
3. Relaunch and load again.

Expected: the missing files are detected and re-downloaded (the per-file etag/existence checks in HubApi.snapshot handle exactly this).

Actual: resolveRepo returns the partial folder as "cached", modelState proceeds to loading, and loadModels() fails:

Failed to open file: …/qwen3_tts/code_decoder/12hz-0.6b-customvoice/W8A16-stateful/CodeDecoder.mlmodelc/coremldata.bin. It is not a valid .mlmodelc file.
E5RT: Input model path not found: …/qwen3_tts/text_projector/12hz-0.6b-customvoice/W8A16/TextProjector.mlmodelc/model.mil (1)
Failed to parse ML Program. It is likely an invalid or broken model.

On-device contents of TextProjector.mlmodelc after the interrupted download — the small files landed, model.mil and weights/ never did:

analytics/coremldata.bin    243 B
coremldata.bin              380 B
metadata.json               2 KB
(model.mil missing — 6.5 KB in repo)
(weights/weight.bin missing — 317 MB in repo)

Root cause

ModelDownloader.resolveRepo short-circuits before HubApi.snapshot ever runs:

if patternsExistLocally(patterns, in: localRoot) {
    Logging.debug("[ModelDownloader] All models found in local cache at \(localRoot.path)")
    return localRoot
}

and patternsExistLocally only checks that each pattern's concrete directory prefix exists and is non-empty:

guard FileManager.default.fileExists(atPath: dir.path) else { return false }
let contents = (try? FileManager.default.contentsOfDirectory(atPath: dir.path)) ?? []
return !contents.isEmpty

A folder holding one stray metadata.json passes. Since HubApi.snapshot is never reached, its per-file downloaded/etag verification — which would repair the gaps — never runs.

Suggested fix

Any of these would resolve it:

1. Drop the short-circuit and always call snapshot when download == true. When the cache is genuinely complete, snapshot is cheap — every file skips via the existing metadata/etag check — so the fast path is mostly preserved at the cost of one tree-listing request (and it picks up upstream revisions as a bonus).
2. Keep the offline fast path but make patternsExistLocally verify actual completeness (compare against the expected component file set, e.g. model.mil + coremldata.bin + weights/ per .mlmodelc, or persist the matched file list from the last successful snapshot and check it).
3. At minimum: on loadModels() failure due to missing/invalid component files, invalidate the cached folder so the next setupModels re-downloads, instead of failing identically forever.

Workaround

Delete the model folder (Documents/huggingface/models/argmaxinc/ttskit-coreml) and re-download; we now also wrap loadModels() and wipe + refetch on failure in our integration.

Happy to PR option 1 or 2 if you have a preference.

[ericlewis]

Two additional defects found while chasing a reproduction on a fresh install (new app container, clean first download, identical corruption) — these compound the patternsExistLocally issue above and explain why the partial state arises in the first place:

1. snapshot returns success when cancelled between files. In the online download loop (HubApi.snapshot):

try await downloader.download { ... }
if Task.isCancelled {
    return repoDestination   // partial snapshot returned as success
}

Any caller that cancels the surrounding task mid-snapshot (in our case: the agent's stop path cancelling a model-load task) gets the repo folder back as if the download completed. The small files (metadata.json, coremldata.bin) finish in milliseconds; the multi-hundred-MB weights/weight.bin is mid-flight when cancellation lands and model.mil is queued behind it — so every component bundle ends up with the same alphabetical cutoff we reported above. Expected: throw CancellationError so the caller knows the snapshot is incomplete.

2. Offline mode validates only files that exist locally. The offline branch (HubApi.snapshot) enumerates the local folder and checks each present file's metadata/hash — it never compares against the expected remote file list, so a partial cache passes validation whenever NetworkMonitor reports offline (e.g. transient connectivity loss — we observed TCP RSTs during the download window). Combined with #1, a single interrupted download is then accepted forever by three independent code paths.

Our workaround now verifies bundle completeness after every snapshot (each .mlmodelc must contain model.mil and a non-empty weights/) and wipes + refetches on failure. Happy to PR fixes for either: making cancellation throw is a one-liner; offline validation could diff against the last successful snapshot's file list persisted alongside the metadata.