feat(core,cli): add support for SGLang and local OpenAI-compatible en… by shivajid · Pull Request #28681 · google-gemini/gemini-cli

shivajid · 2026-08-04T07:47:13Z

…dpoints

Summary

Details

Related Issues

How to Validate

Pre-Merge Checklist

…dpoints

github-actions · 2026-08-04T07:47:32Z

📊 PR Size: size/XL

Lines changed: 1336
Additions: +1329
Deletions: -7
Files changed: 10

gemini-code-assist · 2026-08-04T07:47:55Z

Summary of Changes

Hello, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request expands the core and CLI capabilities by introducing support for SGLang and OpenAI-compatible API endpoints. It includes a new content generator implementation, updates to the authentication detection mechanism, and adds support for the Kimi-K3 model, allowing users to leverage local or remote model servers beyond the existing Google-native options.

Highlights

SGLang Support: Added a new SglangContentGenerator to support local and remote OpenAI-compatible endpoints, enabling broader model integration.
Authentication Updates: Updated the authentication flow to detect SGLang configurations via environment variables (SGLANG_BASE_URL or OPENAI_BASE_URL) and added it as a selectable option in the AuthDialog.
Model Support: Introduced the Kimi-K3 model (moonshotai/Kimi-K3) and updated model validation logic to support it.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	`/gemini review`	Performs a code review for the current pull request in its current state.
Pull Request Summary	`/gemini summary`	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	`/gemini help`	Displays a list of available commands.

Customization

To customize the Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counterproductive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here.

Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

gemini-code-assist

Code Review

This pull request introduces support for SGLang servers (local or remote Kimi-K3) by adding a new SGLANG authentication type, updating environment variable detection, and implementing the SglangContentGenerator class. The code review identified a critical issue where generated tool_call_ids are not mapped to tool responses, which would lead to API errors. Additionally, the reviewer recommended refactoring generateContentStream to return Promise to match the ContentGenerator interface, allowing the removal of an unsafe 'as never' type assertion in contentGenerator.ts.

gemini-code-assist · 2026-08-04T07:50:43Z

+  private convertContentsToMessages(
+    contents: Content[] | string | undefined,
+    systemInstruction?: Content | string,
+  ) {
+    const messages: Array<{
+      role: string;
+      content?: string | null;
+      tool_calls?: Array<{
+        id: string;
+        type: 'function';
+        function: { name: string; arguments: string };
+      }>;
+      tool_call_id?: string;
+    }> = [];
+
+    if (systemInstruction) {
+      const text =
+        typeof systemInstruction === 'string'
+          ? systemInstruction
+          : systemInstruction.parts
+              ?.map((p: Part) => p.text)
+              .filter(Boolean)
+              .join('\n') || '';
+      if (text) {
+        messages.push({ role: 'system', content: text });
+      }
+    }
+
+    if (typeof contents === 'string') {
+      messages.push({ role: 'user', content: contents });
+      return messages;
+    }
+
+    if (Array.isArray(contents)) {
+      for (const c of contents) {
+        if (!c || !c.parts) continue;
+        const role = c.role === 'model' ? 'assistant' : 'user';
+
+        let textParts = '';
+        const toolCalls: Array<{
+          id: string;
+          type: 'function';
+          function: { name: string; arguments: string };
+        }> = [];
+
+        for (const part of c.parts) {
+          if (part.text) {
+            textParts += (textParts ? '\n' : '') + part.text;
+          }
+          if (part.functionCall) {
+            toolCalls.push({
+              id: `call_${Math.random().toString(36).substring(2, 9)}`,
+              type: 'function',
+              function: {
+                name: part.functionCall.name,
+                arguments: JSON.stringify(part.functionCall.args || {}),
+              },
+            });
+          }
+          if (part.functionResponse) {
+            messages.push({
+              role: 'tool',
+              content: JSON.stringify(part.functionResponse.response || {}),
+              tool_call_id:
+                (part.functionResponse as { id?: string }).id ||
+                `call_${part.functionResponse.name}`,
+            });
+          }
+        }
+
+        if (toolCalls.length > 0) {
+          messages.push({
+            role: 'assistant',
+            content: textParts || null,
+            tool_calls: toolCalls,
+          });
+        } else if (textParts) {
+          messages.push({ role, content: textParts });
+        }
+      }
+    }
+
+    return messages;
+  }


The current implementation of convertContentsToMessages generates a random tool_call_id for assistant tool calls, but does not map them to the corresponding tool responses. This will cause a mismatch between the assistant's tool_calls and the tool responses, leading to API errors (400 Bad Request) from SGLang/OpenAI-compatible endpoints.

We should maintain a map of function names to generated tool call IDs during conversion to ensure they match perfectly.

private convertContentsToMessages( contents: Content[] | string | undefined, systemInstruction?: Content | string, ) { const messages: Array<{ role: string; content?: string | null; tool_calls?: Array<{ id: string; type: 'function'; function: { name: string; arguments: string }; }>; tool_call_id?: string; }> = []; const toolCallIds = new Map<string, string>(); if (systemInstruction) { const text = typeof systemInstruction === 'string' ? systemInstruction : systemInstruction.parts ?.map((p: Part) => p.text) .filter(Boolean) .join('\n') || ''; if (text) { messages.push({ role: 'system', content: text }); } } if (typeof contents === 'string') { messages.push({ role: 'user', content: contents }); return messages; } if (Array.isArray(contents)) { for (const c of contents) { if (!c || !c.parts) continue; const role = c.role === 'model' ? 'assistant' : 'user'; let textParts = ''; const toolCalls: Array<{ id: string; type: 'function'; function: { name: string; arguments: string }; }> = []; for (const part of c.parts) { if (part.text) { textParts += (textParts ? '\n' : '') + part.text; } if (part.functionCall) { const callId = `call_${Math.random().toString(36).substring(2, 9)}`; toolCallIds.set(part.functionCall.name, callId); toolCalls.push({ id: callId, type: 'function', function: { name: part.functionCall.name, arguments: JSON.stringify(part.functionCall.args || {}), }, }); } if (part.functionResponse) { const callId = (part.functionResponse as { id?: string }).id || toolCallIds.get(part.functionResponse.name) || `call_${part.functionResponse.name}`; messages.push({ role: 'tool', content: JSON.stringify(part.functionResponse.response || {}), tool_call_id: callId, }); } } if (toolCalls.length > 0) { messages.push({ role: 'assistant', content: textParts || null, tool_calls: toolCalls, }); } else if (textParts) { messages.push({ role, content: textParts }); } } } return messages; }

gemini-code-assist · 2026-08-04T07:50:43Z

+  async *generateContentStream(
+    request: GenerateContentParameters,
+    _userPromptId: string,
+    _role: LlmRole,
+  ): AsyncGenerator<GenerateContentResponse> {
+    const messages = this.convertContentsToMessages(
+      request.contents as Content[] | string,
+      request.config?.systemInstruction as Content | string,
+    );
+    const tools = this.convertTools(request.config);
+
+    const payload: Record<string, unknown> = {
+      model: request.model || this.defaultModel,
+      messages,
+      temperature: request.config?.temperature ?? 0.7,
+      max_tokens: request.config?.maxOutputTokens ?? 4096,
+      stream: true,
+    };
+    if (tools) {
+      payload['tools'] = tools;
+    }
+
+    const res = await fetch(`${this.baseUrl}/chat/completions`, {
+      method: 'POST',
+      headers: { 'Content-Type': 'application/json' },
+      body: JSON.stringify(payload),
+    });
+
+    if (!res.ok || !res.body) {
+      throw new Error(`SGLang stream error (${res.status}): ${await res.text()}`);
+    }
+
+    const reader = res.body.getReader();
+    const decoder = new TextDecoder();
+    let buffer = '';
+
+    while (true) {
+      const { done, value } = await reader.read();
+      if (done) break;
+      buffer += decoder.decode(value, { stream: true });
+      const lines = buffer.split('\n');
+      buffer = lines.pop() || '';
+
+      for (const line of lines) {
+        const trimmed = line.trim();
+        if (!trimmed || !trimmed.startsWith('data: ')) continue;
+        if (trimmed === 'data: [DONE]') return;
+        try {
+          const json = JSON.parse(trimmed.slice(6));
+          const delta = json.choices?.[0]?.delta;
+          if (delta?.content) {
+            yield {
+              candidates: [
+                {
+                  content: { parts: [{ text: delta.content }], role: 'model' },
+                },
+              ],
+            } as GenerateContentResponse;
+          }
+        } catch {
+          // ignore partial chunks
+        }
+      }
+    }
+  }


The ContentGenerator interface defines generateContentStream as returning Promise<AsyncGenerator<GenerateContentResponse>>. By implementing it as an async * function (which returns AsyncGenerator), there is a type mismatch requiring an unsafe as never cast in contentGenerator.ts.

Additionally, using async * defers the fetch call and res.ok check until the caller starts iterating over the generator. Changing this to return Promise<AsyncGenerator> ensures connection/HTTP errors are thrown immediately when the stream is initiated.

async generateContentStream( request: GenerateContentParameters, _userPromptId: string, _role: LlmRole, ): Promise<AsyncGenerator<GenerateContentResponse>> { const messages = this.convertContentsToMessages( request.contents as Content[] | string, request.config?.systemInstruction as Content | string, ); const tools = this.convertTools(request.config); const payload: Record<string, unknown> = { model: request.model || this.defaultModel, messages, temperature: request.config?.temperature, max_tokens: request.config?.maxOutputTokens, stream: true, }; if (tools) { payload['tools'] = tools; } const res = await fetch(`${this.baseUrl}/chat/completions`, { method: 'POST', headers: { 'Content-Type': 'application/json' }, body: JSON.stringify(payload), }); if (!res.ok || !res.body) { throw new Error(`SGLang stream error (${res.status}): ${await res.text()}`); } const reader = res.body.getReader(); const decoder = new TextDecoder(); async function* generator() { let buffer = ''; try { while (true) { const { done, value } = await reader.read(); if (done) break; buffer += decoder.decode(value, { stream: true }); const lines = buffer.split('\n'); buffer = lines.pop() || ''; for (const line of lines) { const trimmed = line.trim(); if (!trimmed || !trimmed.startsWith('data: ')) continue; if (trimmed === 'data: [DONE]') return; try { const json = JSON.parse(trimmed.slice(6)); const delta = json.choices?.[0]?.delta; if (delta?.content) { yield { candidates: [ { content: { parts: [{ text: delta.content }], role: 'model' }, }, ], } as GenerateContentResponse; } } catch { // ignore partial chunks } } } } finally { reader.releaseLock(); } } return generator(); }

References

Rely on the schema as the single source of truth for configuration defaults, avoiding redundant nullish coalescing operators.

gemini-code-assist · 2026-08-04T07:50:43Z

+      return new LoggingContentGenerator(
+        new SglangContentGenerator(baseUrl, modelName) as never,
+        gcConfig,
+      );


With SglangContentGenerator properly implementing the ContentGenerator interface, we can remove the unsafe as never type assertion.

Suggested change

return new LoggingContentGenerator(

new SglangContentGenerator(baseUrl, modelName) as never,

gcConfig,

);

return new LoggingContentGenerator(

new SglangContentGenerator(baseUrl, modelName),

gcConfig,

);

…es robustly

…cs in SglangContentGenerator

…nAI API compatibility

…eliability

…P on stream end

…ons correctly

… Kimi-K3 - Return real GenerateContentResponse instances so SDK getters (functionCalls) used by turn.ts/geminiChat.ts dispatch tool calls - Convert parametersJsonSchema (used by all built-in tools), not just parameters, so tools are advertised with their schemas - Preserve tool call ids across turns (tool_calls[].id == tool_call_id) - Honor config.abortSignal so ESC cancels generation - Support responseJsonSchema/responseMimeType via response_format for internal generateJson calls (next-speaker, loop detection) - Stream usage via stream_options.include_usage; estimate countTokens - Map gemini-* model aliases to the served model name - Stable **Thinking** thought subject; never echo reasoning to server - Remove console.error calls that corrupt the Ink UI; raise max_tokens default to 32768 so large edit/write tool args aren't truncated

…g-support

gemini-cli · 2026-08-12T02:18:21Z

Hi there! Thank you for your interest in contributing to Gemini CLI.

To ensure we maintain high code quality and focus on our prioritized roadmap, we only guarantee review and consideration of pull requests for issues that are explicitly labeled as 'help wanted'.

This PR will be closed in 7 days if it remains without that designation. We encourage you to find and contribute to existing 'help wanted' issues in our backlog! Thank you for your understanding.

gemini-cli · 2026-08-19T02:03:09Z

This pull request is being closed as it has been open for 14 days without a 'help wanted' designation. We encourage you to find and contribute to existing 'help wanted' issues in our backlog! Thank you for your understanding.

feat(core,cli): add support for SGLang and local OpenAI-compatible en…

3b5d8ca

…dpoints

shivajid requested a review from a team as a code owner August 4, 2026 07:47

github-actions Bot added the size/l A large sized PR label Aug 4, 2026

gemini-code-assist Bot reviewed Aug 4, 2026

View reviewed changes

shivajid added 2 commits August 4, 2026 00:59

fix(cli): allow SGLang in auth validation and config initialization

76ed889

fix(core): ensure SGLang authType check supports string and enum valu…

2708a70

…es robustly

gemini-cli Bot added the priority/p1 Important and should be addressed in the near term. label Aug 4, 2026

shivajid added 7 commits August 4, 2026 01:07

fix(core): add reasoning_content and explicit network error diagnosti…

9d6e4e9

…cs in SglangContentGenerator

fix(core): convert uppercase schema types to lowercase for SGLang Ope…

caaacb2

…nAI API compatibility

fix(core): default to 127.0.0.1 instead of localhost for socket DNS r…

643f9d4

…eliability

fix(core): separate reasoning thoughts and guarantee finishReason STO…

28c8fd9

…P on stream end

fix(core): resolve TypeScript compiler types in SglangContentGenerator

1c1378a

fix(core): accumulate streaming tool calls and map OpenAI finish_reas…

5a17b12

…ons correctly

github-actions Bot added the size/xl An extra large PR label Aug 4, 2026

docs: add comprehensive SGLang server setup guide and update README

fb6de2c

shivajid requested a review from a team as a code owner August 4, 2026 15:30

docs: add Linux prerequisites and complete build steps for feat/sglan…

03112c9

…g-support

This was referenced Aug 4, 2026

📊 AI CLI 工具社区动态日报 2026-08-05 litang9/big_model_radar#316

Open

📊 AI CLI 工具社区动态日报 2026-08-05 junlinzhao327-oss/big_model_radar#255

Open

📊 AI CLI 工具社区动态日报 2026-08-05 96loveslife/big_model_radar#276

Open

gemini-cli Bot added the status/pr-nudge-sent label Aug 12, 2026

This was referenced Aug 12, 2026

📊 AI CLI 工具社区动态日报 2026-08-12 96loveslife/big_model_radar#312

Open

📊 AI CLI 工具社区动态日报 2026-08-13 litang9/big_model_radar#352

Open

gemini-cli Bot closed this Aug 19, 2026

Navigation Menu

Uh oh!

feat(core,cli): add support for SGLang and local OpenAI-compatible en… - #28681

feat(core,cli): add support for SGLang and local OpenAI-compatible en…#28681
shivajid wants to merge 12 commits into
google-gemini:mainfrom
shivajid:feat/sglang-support

shivajid commented Aug 4, 2026

Uh oh!

github-actions Bot commented Aug 4, 2026 •

edited

Loading

Uh oh!

gemini-code-assist Bot commented Aug 4, 2026

Uh oh!

gemini-code-assist Bot left a comment

Uh oh!

gemini-code-assist Bot Aug 4, 2026

Uh oh!

gemini-code-assist Bot Aug 4, 2026

Uh oh!

gemini-code-assist Bot Aug 4, 2026

Uh oh!

gemini-cli Bot commented Aug 12, 2026

Uh oh!

gemini-cli Bot commented Aug 19, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

shivajid commented Aug 4, 2026

Summary

Details

Related Issues

How to Validate

Pre-Merge Checklist

Uh oh!

github-actions Bot commented Aug 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

gemini-code-assist Bot commented Aug 4, 2026

Summary of Changes

Highlights

Footnotes

Uh oh!

gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

gemini-code-assist Bot Aug 4, 2026

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist Bot Aug 4, 2026

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist Bot Aug 4, 2026

Choose a reason for hiding this comment

Uh oh!

gemini-cli Bot commented Aug 12, 2026

Uh oh!

gemini-cli Bot commented Aug 19, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

github-actions Bot commented Aug 4, 2026 •

edited

Loading