coreml: expose configurable MLComputeUnits for encoder fallback

[macmixing]

Problem

whisper.cpp currently loads the Core ML encoder with a fixed compute-unit policy. Downstream apps do not have a supported way to force CPU-only or CPU/GPU Core ML execution when Apple Neural Engine execution fails.

This matters for production apps because ANE/Core ML failures can be OS- or hardware-specific, and the app may need to recover without disabling Core ML entirely or patching whisper-encoder.mm.

Real-world case

I am seeing an iOS 27 beta regression where a Whisper Core ML encoder path fails during ANE execution in an iOS app / keyboard-extension speech-to-text flow.

The same app and same model assets work on public iOS 26.6.

The failure signatures include:

ANEProgramProcessRequestDirect() failed
com.apple.appleneuralengine Code=8
E5RT: ANE inference operation failed
Unable to compute the prediction using ML Program

The app’s macOS build is not affected in my testing.

Workaround

If I patch whisper-encoder.mm so a fallback Core ML model context uses:

config.computeUnits = MLComputeUnitsCPUOnly;

then the same Whisper model can recover on iOS 27 beta.

The current downstream workaround is to maintain a local patch to the Core ML loader, which is fragile.

Request

Please expose Core ML compute-unit selection through an official API or configuration path, for example:

• MLComputeUnitsAll
• MLComputeUnitsCPUOnly
• MLComputeUnitsCPUAndGPU
• MLComputeUnitsCPUAndNeuralEngine

This would let downstream apps implement fallback behavior without matching private Apple error codes and without carrying a long-term fork/patch of the Core ML runtime.

Additional context

This is motivated by a production iOS app compatibility issue, but I am not claiming whisper.cpp caused the iOS 27 beta regression. The request is for a supported recovery/configuration path when Core ML ANE execution fails or is undesirable.

I have also seen similar user-facing iOS 27 dictation failures reported by users of another dictation app. I cannot verify that app’s internal provider or compute path, but the report is consistent with the same class of failure: local dictation starts, fails, and returns a generic error instead of text.

Related

I did not find an existing whisper.cpp issue or PR that exposes Core ML MLComputeUnits selection to callers.

[macmixing]

Update: iOS 27 background Neural Engine requirement clarified

Apple clarified the behavior in the iOS 27 Beta 3 release notes. iOS 27 now restricts background access to the Neural Engine, similar to GPU restrictions, and requires this entitlement:

com.apple.developer.background-tasks.continued-processing.inference

Apple's release notes: https://developer.apple.com/documentation/ios-ipados-release-notes/ios-ipados-27-release-notes

This matches the behavior I originally reported:

• The same Whisper Core ML encoder always worked while the app was in the foreground.
• Neural Engine requests were canceled only when inference ran while the app was in the background.
• Forcing MLComputeUnitsCPUOnly avoided the newly restricted background Neural Engine path, allowing inference to continue.
• After enabling the entitlement for the App ID in the Apple Developer portal, regenerating the signing capability, and adding it to the app entitlements, the existing Whisper/Core ML path works in the background on iOS 27 without the CPU-only workaround or a customized Whisper runtime.

Based on that evidence, the specific iOS 27 failure that motivated this issue was not a general whisper.cpp defect as mentioned. It was caused by an iOS 27 background Neural Engine policy change that was not documented in the first seed and was clarified in a later beta.

I would still like the capability requested in this issue. A supported way for downstream callers to select MLComputeUnits remains useful for intentional compute policy, compatibility workarounds, and recovery when a particular execution unit fails. This update narrows the root cause of my original incident; it does not withdraw the API request.