Is your feature request related to a problem? Please describe.
All of Sycamore's current LLM (sycamore.llms) and embedding (sycamore.transforms.embed)
providers are hosted APIs - OpenAI, Anthropic, Bedrock, Gemini. There's no way to run the LLM
or embedding steps of a DocSet pipeline against a local model server. That's a real gap for
anyone building a fully local-first or offline-capable pipeline - related in spirit to #707,
though that issue is specifically about bundling partitioner model weights rather than
LLM/embedding providers.
I'm building a local-first RAG system (part of an MCP-based project) that already runs
entirely on Ollama for generation and embeddings, and I'd like to use Sycamore's
partitioning/DocSet pipeline without introducing a cloud dependency into an otherwise fully
local stack.
Describe the solution you'd like
Add:
OllamaLLM in a new sycamore/llms/ollama.py, subclassing the LLM ABC, using the
official ollama PyPI client.
OllamaEmbedder in sycamore/transforms/embed.py, subclassing Embedder, matching
OpenAIEmbedder's constructor/batching shape.
- A new optional
ollama extras group in pyproject.toml, so this has zero impact on
existing installs.
I have a working draft of both, plus unit tests that mock the client, and I'm ready to open
a draft PR for early feedback if this is a welcome addition.
Describe alternatives you've considered
Ollama also exposes an OpenAI-compatible endpoint (/v1/chat/completions, /v1/embeddings),
so in principle OpenAI/OpenAIEmbedder could already point at a local Ollama server with
no new code. I think a native provider is still worth it - OLLAMA_HOST env var resolution,
Ollama-native options (num_ctx, mirostat, etc.) not exposed through the compat layer,
and clearer errors when the actual backend is Ollama rather than a generic OpenAI-client
exception - but flagging this in case maintainers would rather just document the
compat-endpoint route instead.
Additional context
For OllamaLLM I mirrored anthropic.py (direct client + __reduce__ for Ray-picklability)
rather than openai.py, since the latter carries Azure/Helicone/batch-API handling Ollama
has no equivalent of. Ollama's model catalog is just "whatever's been ollama pulled" rather
than a fixed list, so OllamaModels.from_name() falls back to constructing an ad hoc model
for unrecognized strings instead of raising, unlike the other providers - flagging that as an
intentional deviation in case it should work differently.
Happy to take this on if it's a welcome addition - I'll open a draft PR as soon as I get a
thumbs up.
Is your feature request related to a problem? Please describe.
All of Sycamore's current LLM (
sycamore.llms) and embedding (sycamore.transforms.embed)providers are hosted APIs - OpenAI, Anthropic, Bedrock, Gemini. There's no way to run the LLM
or embedding steps of a DocSet pipeline against a local model server. That's a real gap for
anyone building a fully local-first or offline-capable pipeline - related in spirit to #707,
though that issue is specifically about bundling partitioner model weights rather than
LLM/embedding providers.
I'm building a local-first RAG system (part of an MCP-based project) that already runs
entirely on Ollama for generation and embeddings, and I'd like to use Sycamore's
partitioning/DocSet pipeline without introducing a cloud dependency into an otherwise fully
local stack.
Describe the solution you'd like
Add:
OllamaLLMin a newsycamore/llms/ollama.py, subclassing theLLMABC, using theofficial
ollamaPyPI client.OllamaEmbedderinsycamore/transforms/embed.py, subclassingEmbedder, matchingOpenAIEmbedder's constructor/batching shape.ollamaextras group inpyproject.toml, so this has zero impact onexisting installs.
I have a working draft of both, plus unit tests that mock the client, and I'm ready to open
a draft PR for early feedback if this is a welcome addition.
Describe alternatives you've considered
Ollama also exposes an OpenAI-compatible endpoint (
/v1/chat/completions,/v1/embeddings),so in principle
OpenAI/OpenAIEmbeddercould already point at a local Ollama server withno new code. I think a native provider is still worth it -
OLLAMA_HOSTenv var resolution,Ollama-native
options(num_ctx,mirostat, etc.) not exposed through the compat layer,and clearer errors when the actual backend is Ollama rather than a generic OpenAI-client
exception - but flagging this in case maintainers would rather just document the
compat-endpoint route instead.
Additional context
For
OllamaLLMI mirroredanthropic.py(direct client +__reduce__for Ray-picklability)rather than
openai.py, since the latter carries Azure/Helicone/batch-API handling Ollamahas no equivalent of. Ollama's model catalog is just "whatever's been
ollama pulled" ratherthan a fixed list, so
OllamaModels.from_name()falls back to constructing an ad hoc modelfor unrecognized strings instead of raising, unlike the other providers - flagging that as an
intentional deviation in case it should work differently.
Happy to take this on if it's a welcome addition - I'll open a draft PR as soon as I get a
thumbs up.