Skip to content

Add Ollama backend (local LLM + embeddings) #1605

Description

@SkyeVault

Is your feature request related to a problem? Please describe.
All of Sycamore's current LLM (sycamore.llms) and embedding (sycamore.transforms.embed)
providers are hosted APIs - OpenAI, Anthropic, Bedrock, Gemini. There's no way to run the LLM
or embedding steps of a DocSet pipeline against a local model server. That's a real gap for
anyone building a fully local-first or offline-capable pipeline - related in spirit to #707,
though that issue is specifically about bundling partitioner model weights rather than
LLM/embedding providers.

I'm building a local-first RAG system (part of an MCP-based project) that already runs
entirely on Ollama for generation and embeddings, and I'd like to use Sycamore's
partitioning/DocSet pipeline without introducing a cloud dependency into an otherwise fully
local stack.

Describe the solution you'd like
Add:

  • OllamaLLM in a new sycamore/llms/ollama.py, subclassing the LLM ABC, using the
    official ollama PyPI client.
  • OllamaEmbedder in sycamore/transforms/embed.py, subclassing Embedder, matching
    OpenAIEmbedder's constructor/batching shape.
  • A new optional ollama extras group in pyproject.toml, so this has zero impact on
    existing installs.

I have a working draft of both, plus unit tests that mock the client, and I'm ready to open
a draft PR for early feedback if this is a welcome addition.

Describe alternatives you've considered
Ollama also exposes an OpenAI-compatible endpoint (/v1/chat/completions, /v1/embeddings),
so in principle OpenAI/OpenAIEmbedder could already point at a local Ollama server with
no new code. I think a native provider is still worth it - OLLAMA_HOST env var resolution,
Ollama-native options (num_ctx, mirostat, etc.) not exposed through the compat layer,
and clearer errors when the actual backend is Ollama rather than a generic OpenAI-client
exception - but flagging this in case maintainers would rather just document the
compat-endpoint route instead.

Additional context
For OllamaLLM I mirrored anthropic.py (direct client + __reduce__ for Ray-picklability)
rather than openai.py, since the latter carries Azure/Helicone/batch-API handling Ollama
has no equivalent of. Ollama's model catalog is just "whatever's been ollama pulled" rather
than a fixed list, so OllamaModels.from_name() falls back to constructing an ad hoc model
for unrecognized strings instead of raising, unlike the other providers - flagging that as an
intentional deviation in case it should work differently.

Happy to take this on if it's a welcome addition - I'll open a draft PR as soon as I get a
thumbs up.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions