Skip to content

Commit ccd64bb

Browse files
joelagnelplarroy-nvRomneyDajacobtomlinsonCopilot
authored
feat: add managed local AI with llama-server (#1178)
* refactor(device): centralize physical memory probing Move physical-memory discovery into one shared implementation. Give setup and inference code a consistent RAM capacity value. Signed-off-by: Joel Fernandes <joelagnelf@nvidia.com> * feat(inference): add host hardware snapshot Capture CPU, memory, and GPU facts in a reusable host snapshot. Provide the hardware input used by Local AI eligibility selection. Signed-off-by: Joel Fernandes <joelagnelf@nvidia.com> * fix(setup): roll back cancelled setup transactions Roll back both interrupted and completed steps when setup is cancelled. Use an independent cleanup token so cancellation cannot strand partial state. Signed-off-by: Joel Fernandes <joelagnelf@nvidia.com> * feat(setup): manage mirrored WSL networking safely Configure mirrored networking only with explicit consent and restore it on rollback. Preserve BOMs, line endings, unterminated final lines, and concurrent user edits. Signed-off-by: Joel Fernandes <joelagnelf@nvidia.com> * refactor(setup): add immutable Local AI path policy Centralize and validate runtime, model, manifest, preset, and staging paths. Reject reparse points and delete only prevalidated app-owned trees during cleanup. Signed-off-by: Joel Fernandes <joelagnelf@nvidia.com> * refactor(setup): add verified native artifact installer Download, verify, extract, and install native archives atomically. Follow only bounded credential-free HTTPS redirects to approved GitHub release hosts. Clean staging only through no-follow app-owned path validation. Signed-off-by: Joel Fernandes <joelagnelf@nvidia.com> Co-authored-by: Dallin Romney <dallinromney@gmail.com> * refactor(local-ai): add durable runtime manifests Persist installed runtime and model metadata with atomic file updates. Give setup, startup, and cleanup one durable source of installation truth. Signed-off-by: Joel Fernandes <joelagnelf@nvidia.com> * refactor(local-ai): add native Windows managed-process host Host native Local AI processes with bounded startup and shutdown behavior. Capture output and terminate owned process trees reliably on Windows. Signed-off-by: Joel Fernandes <joelagnelf@nvidia.com> * feat(inference): add pinned runtime and model catalogs Define immutable llama.cpp, CUDA, and GGUF catalog entries with recipe capacity metadata. Pin versions, hashes, 262K KV topology, and runtime workspace requirements for reproducible installs. Signed-off-by: Joel Fernandes <joelagnelf@nvidia.com> Co-authored-by: Dallin Romney <dallinromney@gmail.com> * feat(inference): qualify NVIDIA GPUs by capability Select any NVML-backed NVIDIA adapter with a stable ID, compatible driver, and CUDA runtime. Require memory for model weights, recipe KV cache, and workspace without GPU SKU or CPU/GPU pairing. Recommend the largest fitting model without silently changing explicit choices. Signed-off-by: Joel Fernandes <joelagnelf@nvidia.com> Co-authored-by: Dallin Romney <dallinromney@gmail.com> * feat(inference): probe NVIDIA hardware through trusted NVML Load NVML from trusted locations and collect NVIDIA GPU capabilities. Avoid unsafe library resolution while providing selector-grade hardware data. Signed-off-by: Joel Fernandes <joelagnelf@nvidia.com> * improve(inference): account for shared NVIDIA GPU memory Correlate NVML adapters with unambiguous DXGI memory observations and count shared memory for any NVIDIA GPU. Fail closed on duplicate or ambiguous adapter names so model selection cannot borrow another device's budget. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: Joel Fernandes <joelagnelf@nvidia.com> * feat(huggingface): download verified GGUF models Resume interrupted GGUF downloads with strict Range and Content-Range handling. Verify size and hash before atomic promotion, restarting safely when resume is unsupported. Signed-off-by: Joel Fernandes <joelagnelf@nvidia.com> * feat(llama-server): install verified native CUDA runtime Install and inspect the pinned llama-server and CUDA runtime components. Safely reconcile exact orphan runtime paths so interrupted promotion can retry. Signed-off-by: Joel Fernandes <joelagnelf@nvidia.com> * feat(llama-server): add request-driven managed router Launch llama-server on demand with an OS-assigned or validated fixed port. Prove listener ownership by child PID and start time before health checks or persistence. Signed-off-by: Joel Fernandes <joelagnelf@nvidia.com> * feat(setup): select and acquire qualified Local AI plan Make Local AI opt-in and inspect hardware plus WSL viability without mutation. Verify the WSL platform before large downloads, then acquire native inference before gateway provisioning. Resolve bundled defaults reliably from RID-specific output. Signed-off-by: Joel Fernandes <joelagnelf@nvidia.com> Co-authored-by: Dallin Romney <dallinromney@gmail.com> * feat(setup): persist and start managed Local AI Reconcile and reuse only exact manifest-owned runtime and model artifacts after interruption. Persist a proven endpoint, clean durable state on uninstall, and test trusted runtime redirect boundaries. Signed-off-by: Joel Fernandes <joelagnelf@nvidia.com> Co-authored-by: Dallin Romney <dallinromney@gmail.com> * feat(setup): verify local inference readiness Probe health, execute real inference, and verify GPU activity on the owned endpoint. Fail setup cleanly when the installed Local AI stack is not operational. Signed-off-by: Joel Fernandes <joelagnelf@nvidia.com> * feat(setup): configure IPv4 llama-server gateway provider Use explicit 127.0.0.1 for Windows-to-WSL gateway connections and persisted setup state. Remove only exact managed Local AI provider state during uninstall, preserving drift. Signed-off-by: Joel Fernandes <joelagnelf@nvidia.com> * feat(local-ai): own llama router in companion lifecycle Bind Local AI to the singleton setup-managed distro resolved from the loaded gateway registry. Quiesce exact provider state before transitions, then publish only healthy owned endpoints and compensate failures. Signed-off-by: Joel Fernandes <joelagnelf@nvidia.com> * feat(setup-ui): add Local AI onboarding Surface Local AI eligibility, explicit consent, review, and setup progress. Show WSL platform readiness before downloads, then native inference before gateway provisioning. Describe dynamic multi-gigabyte disk use accurately. Signed-off-by: Joel Fernandes <joelagnelf@nvidia.com> Co-authored-by: Dallin Romney <dallinromney@gmail.com> * feat(tray): add Local AI status and controls Add Local AI navigation, status, controls, logs, and localized resources. Wire the page through application services and retain focused UI contracts. Signed-off-by: Joel Fernandes <joelagnelf@nvidia.com> * improve(setup): explain unavailable Local AI Keep a compact Local AI review visible when hardware, WSL, or networking blocks setup. Hide unavailable controls and show every detected dependency reason through See why. Co-authored-by: Karen Lai <7976322+karkarl@users.noreply.github.com> Signed-off-by: Joel Fernandes <joelagnelf@nvidia.com> * fix(local-ai): verify GPU model placement Accept full CUDA offload when either NVML memory movement or the parsed CUDA model buffer proves placement. Reject startup when neither signal accounts for the selected model. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: Joel Fernandes <joelagnelf@nvidia.com> * style(setup-ui): pad WSL networking consent content The consent InfoBar content sat flush against the bottom edge of the warning, leaving the message and checkbox visually cramped. Add the same bottom content margin the sandbox InfoBar already uses. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01L8LRNTZ5fBgjGwmtBCrSp3 Signed-off-by: Joel Fernandes <joelagnelf@nvidia.com> * fix(setup): stop WSL probes stalling on pipe EOF Wait for the direct child process independently of redirected-pipe EOF, then bound the final output drain. Existing-distro inspection now fails closed when WSL times out or returns an unknown error. Show progress during the Welcome-page inspection so a cold WSL service does not make setup look frozen. Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Joel Fernandes <joelagnelf@nvidia.com> * fix(setup): explain WSL platform install failures After an elevated WSL install fails, run a best-effort GitHub quota diagnostic and distinguish likely quota exhaustion from other download failures. Diagnostic timeouts never replace the original failure. Offer Store, winget, and elevated PowerShell recovery routes, and allow the platform step to be retried. Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Joel Fernandes <joelagnelf@nvidia.com> --------- Signed-off-by: Joel Fernandes <joelagnelf@nvidia.com> Co-authored-by: Pedro Larroy <plarroy@nvidia.com> Co-authored-by: Dallin Romney <dallinromney@gmail.com> Co-authored-by: Jacob Tomlinson <jtomlinson@nvidia.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Co-authored-by: Karen Lai <7976322+karkarl@users.noreply.github.com> Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
1 parent d3ed6c7 commit ccd64bb

93 files changed

Lines changed: 14681 additions & 208 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

docs/ARCHITECTURE.md

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -66,6 +66,7 @@ These are the canonical homes. Do not reintroduce private copies elsewhere.
6666
| Settings page load/persist view logic | `SettingsPageViewModel` | authoritative |
6767
| Native tool identity, display arguments, payload extraction, and flattened-history projection | `NativeToolProjector` | authoritative |
6868
| Managed-local listener provenance and strong-credential authorization | `ManagedLocalGatewayPortProvenanceService` | authoritative |
69+
| Local AI gateway-record ownership and WSL distro binding | `LocalAiGatewayDistroResolver` | authoritative |
6970
| Exact Gateway wizard terminal-restart compatibility and bounded retry policy | `GatewayWizardRestartRecoveryPolicy` | authoritative |
7071
| Managed-local automatic repair eligibility and orchestration | `ManagedLocalGatewayAutoRepairMonitor` + `ManagedLocalGatewayRepairCoordinator` | authoritative |
7172
| Permissions page state, settings commands, and exec-approvals presentation | `PermissionsPageViewModel` | authoritative |
@@ -160,6 +161,7 @@ leading and trailing pipe. Columns, in order:
160161
| setup-keepalive-process-manager | authoritative | src/OpenClaw.SetupEngine/SetupSteps.cs (StartKeepaliveStep) | setup-time WSL keepalive process discovery, start, marker read/write, command-line identity, and rollback cleanup | KeepaliveProcessManager (raw OS calls delegated to internal IKeepaliveProcessRuntime seam; StartKeepaliveStep is the only caller that reads SetupContext) | StartKeepaliveStep keeps Id/DisplayName and thin ExecuteAsync/RollbackAsync orchestration only | setup-time keepalive never hard-fails the pipeline on start failure (null PID or thrown exception both soft-fail identically); its marker path/JSON are the intentional handoff consumed by the tray keepalive service; rollback kills only wsl/wsl.exe processes whose command line matches this distro via WslCommandLineMatcher, leaves wrong-distro/unmatched command lines untouched, and deletes only its own marker/empty directory | KeepaliveProcessManagerTests.RollbackAsync_KillsOnlyMatchingDistroProcesses_LeavesOthersUntouched | behavioral | when StartKeepaliveStep contains no process/marker logic of its own |
161162
| wsl-distro-install-path | authoritative | OpenClaw.SetupEngine/SetupSteps.cs | inline Path.Combine wsl distro install-path derivation | DistroInstallPathPolicy | - | new installs use the strict supported name grammar; teardown accepts only unambiguous single-segment names whose canonical path is an immediate child of LocalDataDir\wsl with no aliases, case or Unicode collisions, or reparse points at the root or child | SetupStepsTests.DistroInstallPathPolicy_ResolvesImmediateChild | behavioral | - |
162163
| managed-local-provenance | authoritative | scattered connection, setup, browser, and reconnect call sites | implicit loopback trust and duplicated strong-credential listener checks | ManagedLocalGatewayPortProvenanceService | callers request inspection, authorization, or conflict repair only | unknown, incomplete, conflicting, or changed Windows listener ownership never receives strong credentials or destructive remediation; relayless ownership requires a complete empty Windows snapshot, expected-distro systemd MainPID proof, and immediate complete empty revalidation | ManagedLocalGatewayPortProvenanceServiceTests.InteractiveCredentialGate_ExpectedCacheThenOwnerChanges_FailsClosed | behavioral | - |
164+
| local-ai-gateway-distro-binding | authoritative | src/OpenClaw.Tray.WinUI/App.xaml.cs | hardcoded Local AI WSL distro selection | LocalAiGatewayDistroResolver | App loads the gateway registry and composes the resolver, provider coordinator, and runtime | the singleton Local AI installation binds to exactly one explicit setup-managed local no-SSH gateway record; its record ID and SetupManagedDistroName are pinned and revalidated before every WSL command, while missing, ambiguous, unavailable, or drifted ownership fails closed | LocalAiGatewayProviderCoordinatorTests.Quiesce_OwnerDriftsAfterInspection_BlocksFirstMutation | behavioral | - |
163165
| gateway-wizard-restart-recovery | authoritative | WizardPage + SetupWizardRunner reconnect call sites | duplicated exact-version terminal-restart classification and bounded provenance retry orchestration | GatewayWizardRestartRecoveryPolicy | WizardPage and SetupWizardRunner apply hosted and headless lifecycle and consume provenance inspection results | only managed-local restart-like disconnects may retry NoListener or the typed snapshot-changed race; other unknown or conflicting ownership fails immediately, retryable startup close 1013 stays inside the existing reconnect bound, and exact Gateway 2026.7.1 final model-check close 1012 completes only after a fresh hello-ok, and a terminal hosted-wizard payload completes on the exact TUI SIGTERM termination only when the request just sent answered the authoritative final done acknowledgement step | GatewayWizardRestartRecoveryPolicyTests.Exact2026_7_1TerminalModelCheckServiceRestart_IsExpected | behavioral | when the 2026.7.1 terminal-restart compatibility path is removed |
164166
| managed-local-repair | authoritative | src/OpenClaw.Tray.WinUI/App.xaml.cs and direct reconnect callbacks | repair eligibility, restart budgets, port remediation, and reconnect verification | ManagedLocalGatewayAutoRepairMonitor + ManagedLocalGatewayRepairCoordinator | App composition and dependency callbacks only | explicit disconnect and gateway switches abort repair before restart or reconnect | ManagedLocalGatewayRepairCoordinatorTests.UserDisconnectedIntent_AbortsBeforeProbeOrRestart | behavioral | - |
165167
| app-managed-local-repair-closed | closed | src/OpenClaw.Tray.WinUI/App.xaml.cs | managed-local repair loops, probing, restart budgeting, and verification implementation | ManagedLocalGatewayAutoRepairMonitor + ManagedLocalGatewayRepairCoordinator | service construction, callback adapters, and lifetime wiring only | App remains the composition root and does not regain repair implementation | AppRefactorContractTests.ManagedLocalGatewayRepair_StaysDelegatedToDedicatedOwners | source-shape | when App no longer constructs the managed-local repair services directly |
Lines changed: 302 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,302 @@
1+
using System.Text.Json;
2+
3+
namespace OpenClaw.Connection.LocalAi;
4+
5+
public sealed record LlamaServerRouterProbeResult(
6+
bool IsHealthy,
7+
LocalAiModelAvailabilityState ModelState,
8+
string? ReportedModelPath,
9+
string? Detail);
10+
11+
public sealed record LlamaServerModelStatusEvidence(
12+
LocalAiModelAvailabilityState State,
13+
string ModelPath,
14+
string ServerStatus);
15+
16+
/// <summary>
17+
/// Parses the router model metadata emitted by qualified llama-server builds.
18+
/// Unloaded preset models in b10488 report their path in status.args, while a
19+
/// loaded model may also expose the documented top-level path field.
20+
/// </summary>
21+
public static class LlamaServerModelStatusParser
22+
{
23+
public static LlamaServerModelStatusEvidence? Parse(
24+
JsonElement root,
25+
string modelAlias,
26+
string expectedModelPath)
27+
{
28+
ArgumentException.ThrowIfNullOrWhiteSpace(modelAlias);
29+
ArgumentException.ThrowIfNullOrWhiteSpace(expectedModelPath);
30+
if (root.ValueKind != JsonValueKind.Object ||
31+
!root.TryGetProperty("data", out JsonElement models) ||
32+
models.ValueKind != JsonValueKind.Array)
33+
{
34+
throw new InvalidDataException("The llama-server model status response has an invalid shape.");
35+
}
36+
37+
JsonElement? match = null;
38+
foreach (JsonElement model in models.EnumerateArray())
39+
{
40+
if (model.ValueKind != JsonValueKind.Object ||
41+
!model.TryGetProperty("id", out JsonElement id) ||
42+
id.ValueKind != JsonValueKind.String)
43+
{
44+
throw new InvalidDataException("The llama-server model status contains an invalid entry.");
45+
}
46+
if (!string.Equals(id.GetString(), modelAlias, StringComparison.Ordinal))
47+
continue;
48+
if (match is not null)
49+
throw new InvalidDataException("The llama-server model status contains duplicate aliases.");
50+
match = model;
51+
}
52+
53+
if (match is null)
54+
return null;
55+
56+
JsonElement selected = match.Value;
57+
if (!selected.TryGetProperty("status", out JsonElement statusElement) ||
58+
statusElement.ValueKind != JsonValueKind.Object ||
59+
!statusElement.TryGetProperty("value", out JsonElement valueElement) ||
60+
valueElement.ValueKind != JsonValueKind.String ||
61+
string.IsNullOrWhiteSpace(valueElement.GetString()))
62+
{
63+
throw new InvalidDataException("The llama-server model status does not contain a valid state.");
64+
}
65+
66+
string? topLevelPath = ReadOptionalTopLevelPath(selected);
67+
string? argumentPath = ReadOptionalModelArgument(statusElement);
68+
string reportedPath = topLevelPath ?? argumentPath
69+
?? throw new InvalidDataException("The llama-server model status does not identify the managed model path.");
70+
if (!PathsEqual(reportedPath, expectedModelPath) ||
71+
(topLevelPath is not null && argumentPath is not null && !PathsEqual(topLevelPath, argumentPath)))
72+
{
73+
throw new InvalidDataException("The llama-server model status does not match the managed model.");
74+
}
75+
76+
string status = valueElement.GetString()!;
77+
LocalAiModelAvailabilityState state = status switch
78+
{
79+
"loaded" => LocalAiModelAvailabilityState.Loaded,
80+
"unloaded" or "loading" or "sleeping" => LocalAiModelAvailabilityState.Verified,
81+
_ => LocalAiModelAvailabilityState.Unknown,
82+
};
83+
return new(state, reportedPath, status);
84+
}
85+
86+
private static string? ReadOptionalTopLevelPath(JsonElement selected)
87+
{
88+
if (!selected.TryGetProperty("path", out JsonElement path))
89+
return null;
90+
if (path.ValueKind != JsonValueKind.String || string.IsNullOrWhiteSpace(path.GetString()))
91+
throw new InvalidDataException("The llama-server model path is invalid.");
92+
return path.GetString();
93+
}
94+
95+
private static string? ReadOptionalModelArgument(JsonElement status)
96+
{
97+
if (!status.TryGetProperty("args", out JsonElement args))
98+
return null;
99+
if (args.ValueKind != JsonValueKind.Array)
100+
throw new InvalidDataException("The llama-server model arguments are invalid.");
101+
102+
string? modelPath = null;
103+
JsonElement[] values = args.EnumerateArray().ToArray();
104+
for (int index = 0; index < values.Length; index++)
105+
{
106+
if (values[index].ValueKind != JsonValueKind.String)
107+
throw new InvalidDataException("The llama-server model arguments contain a non-string value.");
108+
string? value = values[index].GetString();
109+
if (value is not ("--model" or "-m"))
110+
continue;
111+
if (modelPath is not null || index + 1 >= values.Length ||
112+
values[index + 1].ValueKind != JsonValueKind.String ||
113+
string.IsNullOrWhiteSpace(values[index + 1].GetString()))
114+
{
115+
throw new InvalidDataException("The llama-server model arguments contain an invalid model path.");
116+
}
117+
modelPath = values[++index].GetString();
118+
}
119+
return modelPath;
120+
}
121+
122+
private static bool PathsEqual(string left, string right)
123+
{
124+
try
125+
{
126+
return string.Equals(Path.GetFullPath(left), Path.GetFullPath(right), StringComparison.OrdinalIgnoreCase);
127+
}
128+
catch (Exception ex) when (ex is ArgumentException or NotSupportedException or PathTooLongException)
129+
{
130+
throw new InvalidDataException("The llama-server reported an invalid model path.", ex);
131+
}
132+
}
133+
}
134+
135+
internal interface ILlamaServerClient : IDisposable
136+
{
137+
Task<LlamaServerRouterProbeResult> ProbeRouterAsync(
138+
Uri endpoint,
139+
string modelAlias,
140+
string expectedModelPath,
141+
CancellationToken cancellationToken = default);
142+
}
143+
144+
/// <summary>Bounded, loopback-only health and model-state client for the managed llama-server router.</summary>
145+
public sealed class LlamaServerClient : ILlamaServerClient
146+
{
147+
private const int MaxEvidenceResponseBytes = 1024 * 1024;
148+
private readonly HttpClient _client;
149+
150+
public LlamaServerClient() : this(new SocketsHttpHandler
151+
{
152+
UseProxy = false,
153+
AllowAutoRedirect = false,
154+
ConnectTimeout = TimeSpan.FromSeconds(2),
155+
})
156+
{
157+
}
158+
159+
internal LlamaServerClient(HttpMessageHandler handler)
160+
{
161+
_client = new HttpClient(handler ?? throw new ArgumentNullException(nameof(handler)), disposeHandler: true)
162+
{
163+
Timeout = TimeSpan.FromSeconds(3),
164+
};
165+
}
166+
167+
public async Task<LlamaServerRouterProbeResult> ProbeRouterAsync(
168+
Uri endpoint,
169+
string modelAlias,
170+
string expectedModelPath,
171+
CancellationToken cancellationToken = default)
172+
{
173+
ArgumentNullException.ThrowIfNull(endpoint);
174+
ArgumentException.ThrowIfNullOrWhiteSpace(modelAlias);
175+
ArgumentException.ThrowIfNullOrWhiteSpace(expectedModelPath);
176+
ValidateManagedEndpoint(endpoint);
177+
178+
if (!await ProbeHealthAsync(endpoint, cancellationToken).ConfigureAwait(false))
179+
{
180+
return new(
181+
false,
182+
LocalAiModelAvailabilityState.Unknown,
183+
null,
184+
"The llama-server router health check did not succeed.");
185+
}
186+
187+
try
188+
{
189+
return await ProbeModelAsync(endpoint, modelAlias, expectedModelPath, cancellationToken)
190+
.ConfigureAwait(false);
191+
}
192+
catch (OperationCanceledException) when (!cancellationToken.IsCancellationRequested)
193+
{
194+
return new(true, LocalAiModelAvailabilityState.Unknown, null, "The model status check timed out.");
195+
}
196+
catch (Exception ex) when (ex is HttpRequestException or IOException or JsonException or InvalidDataException)
197+
{
198+
return new(true, LocalAiModelAvailabilityState.Unknown, null, "The model status response was invalid.");
199+
}
200+
}
201+
202+
private async Task<bool> ProbeHealthAsync(Uri endpoint, CancellationToken cancellationToken)
203+
{
204+
try
205+
{
206+
using var response = await _client.GetAsync(
207+
BuildEndpointUri(endpoint, "/health"),
208+
HttpCompletionOption.ResponseHeadersRead,
209+
cancellationToken)
210+
.ConfigureAwait(false);
211+
if (!response.IsSuccessStatusCode)
212+
return false;
213+
214+
byte[] payload = await ReadBoundedAsync(response.Content, cancellationToken).ConfigureAwait(false);
215+
using JsonDocument document = JsonDocument.Parse(payload, new JsonDocumentOptions { MaxDepth = 8 });
216+
return document.RootElement.ValueKind == JsonValueKind.Object &&
217+
document.RootElement.TryGetProperty("status", out JsonElement status) &&
218+
status.ValueKind == JsonValueKind.String &&
219+
string.Equals(status.GetString(), "ok", StringComparison.Ordinal);
220+
}
221+
catch (OperationCanceledException) when (!cancellationToken.IsCancellationRequested)
222+
{
223+
return false;
224+
}
225+
catch (Exception ex) when (ex is HttpRequestException or IOException or JsonException or InvalidDataException)
226+
{
227+
return false;
228+
}
229+
}
230+
231+
private async Task<LlamaServerRouterProbeResult> ProbeModelAsync(
232+
Uri endpoint,
233+
string modelAlias,
234+
string expectedModelPath,
235+
CancellationToken cancellationToken)
236+
{
237+
using var response = await _client.GetAsync(
238+
BuildEndpointUri(endpoint, "/models", "autoload=false"),
239+
HttpCompletionOption.ResponseHeadersRead,
240+
cancellationToken)
241+
.ConfigureAwait(false);
242+
if (!response.IsSuccessStatusCode)
243+
throw new HttpRequestException($"llama-server model status returned HTTP {(int)response.StatusCode}.");
244+
245+
byte[] payload = await ReadBoundedAsync(response.Content, cancellationToken).ConfigureAwait(false);
246+
using JsonDocument document = JsonDocument.Parse(payload, new JsonDocumentOptions { MaxDepth = 16 });
247+
LlamaServerModelStatusEvidence? evidence = LlamaServerModelStatusParser.Parse(
248+
document.RootElement,
249+
modelAlias,
250+
expectedModelPath);
251+
if (evidence is null)
252+
return new(true, LocalAiModelAvailabilityState.NotInstalled, null, "The configured model is not registered.");
253+
return new(
254+
true,
255+
evidence.State,
256+
evidence.ModelPath,
257+
$"llama-server reports the model as {evidence.ServerStatus}.");
258+
}
259+
260+
private static void ValidateManagedEndpoint(Uri endpoint)
261+
{
262+
if (!endpoint.IsAbsoluteUri ||
263+
endpoint.Scheme != Uri.UriSchemeHttp ||
264+
!string.Equals(endpoint.Host, "127.0.0.1", StringComparison.Ordinal) ||
265+
endpoint.Port is <= 0 or > 65535 ||
266+
endpoint.Port == 80 ||
267+
!string.Equals(endpoint.AbsolutePath, "/v1", StringComparison.Ordinal) ||
268+
!string.IsNullOrEmpty(endpoint.UserInfo) ||
269+
!string.IsNullOrEmpty(endpoint.Query) ||
270+
!string.IsNullOrEmpty(endpoint.Fragment))
271+
{
272+
throw new ArgumentException("The llama-server endpoint must use an explicit IPv4 loopback port.", nameof(endpoint));
273+
}
274+
}
275+
276+
private static Uri BuildEndpointUri(Uri endpoint, string path, string? query = null) =>
277+
new UriBuilder(Uri.UriSchemeHttp, "127.0.0.1", endpoint.Port, path)
278+
{
279+
Query = query ?? string.Empty,
280+
}.Uri;
281+
282+
private static async Task<byte[]> ReadBoundedAsync(HttpContent content, CancellationToken cancellationToken)
283+
{
284+
if (content.Headers.ContentLength is > MaxEvidenceResponseBytes)
285+
throw new InvalidDataException("The llama-server evidence response exceeds the size limit.");
286+
287+
await using Stream input = await content.ReadAsStreamAsync(cancellationToken).ConfigureAwait(false);
288+
using var output = new MemoryStream();
289+
var buffer = new byte[16 * 1024];
290+
while (true)
291+
{
292+
int read = await input.ReadAsync(buffer.AsMemory(), cancellationToken).ConfigureAwait(false);
293+
if (read == 0)
294+
return output.ToArray();
295+
if (output.Length + read > MaxEvidenceResponseBytes)
296+
throw new InvalidDataException("The llama-server evidence response exceeds the size limit.");
297+
output.Write(buffer, 0, read);
298+
}
299+
}
300+
301+
public void Dispose() => _client.Dispose();
302+
}

0 commit comments

Comments
 (0)