Skip to content

quickstart.md

Latest commit

 

History

History
227 lines (163 loc) · 6.06 KB

quickstart.md

File metadata and controls

227 lines (163 loc) · 6.06 KB
Raw

Quickstart

This guide gets a new llmff user from install to a working pipeline without requiring provider credentials. It uses the deterministic mock backend first, then shows the smallest real-backend configuration.

Use llmff when you want explicit, typed inference sub-pipelines that can be inspected, versioned, run as a subprocess, and supervised from files. Do not use it as an agent framework, model server, scheduler, memory system, or autonomous planner. See docs/when-to-use-llmff.md for the decision guide.

1. Install

Install the latest tagged release with Cargo:

cargo install --git https://github.com/syndicalt/llmff --tag v1.1.0 llmff

If you downloaded a release archive instead, unpack it and put the llmff binary on your PATH. Platform-specific release asset installation and checksum verification are documented in docs/github-release-installation.md.

Verify the binary:

llmff --version
llmff stages list

Expected version:

llmff 1.1.0

2. Run The Offline Example

Clone the repository, then run the JSON repair example:

git clone https://github.com/syndicalt/llmff.git
cd llmff

LLMFF_MOCK_BAD_RESPONSE='{"wrong":true}' \
LLMFF_MOCK_GOOD_RESPONSE='{"answer":"ok"}' \
llmff run examples/json-repair.yaml --trace /tmp/llmff-trace.jsonl

The run writes:

examples/answer.json

Confirm the output:

cat examples/answer.json

Expected output:

{"answer":"ok"}

The example intentionally asks one mock model for invalid JSON, validates it against examples/answer.schema.json, repairs it with a second mock model, and writes the repaired object.

3. Inspect Before Running

Use inspect when editing a pipeline. It validates references, stage requirements, type compatibility that can be proven statically, and backend availability without calling model servers.

llmff inspect examples/json-repair.yaml

Expected output:

ok

For agents, dashboards, and CI jobs that need a machine-readable preflight, use the JSON inspect report:

llmff inspect examples/json-repair.yaml --format json

The report includes the manifest hash, resolved inputs and outputs, execution stage order, model/backend aliases, per-stage capability constraints, plugin directories, stdout ownership, artifact paths, checkpoint/resume intent, and the requested default execution controls.

4. Run A One-Line Pipeline

Inline graphs are useful for shell workflows and quick experiments:

LLMFF_MOCK_GOOD_RESPONSE='{"answer":"ok"}' \
llmff run -i examples/question.txt \
  -g 'load | infer(model=mock:good) | write(-)'

Expected output:

{"answer":"ok"}

5. Run A Bounded Loop

llmff v1.1 adds op: loop for bounded refinement, retry, and agent-style sub-pipelines. Loops always require max_iterations, an explicit break_on condition, and a body DAG.

Inspect the self-refining answer loop before running it:

llmff inspect examples/loops/self-refining-answer-loop.yaml --format json

Run it with the mock backend and write a trace:

LLMFF_MOCK_GOOD_RESPONSE='{"answer":"Use llmff for bounded, inspectable LLM pipelines.","confidence":0.93}' \
llmff run examples/loops/self-refining-answer-loop.yaml \
  --trace /tmp/llmff-self-refining-answer.trace.jsonl

The loop writes:

examples/loops/self-refining-answer.output.json

Inspect the trace summary:

llmff trace /tmp/llmff-self-refining-answer.trace.jsonl

Loop body records include loop_id, loop_iteration, and loop_stage_id, so supervisors can explain which iteration produced each body-stage event.

More loop patterns are documented in examples/loops/README.md.

6. Use A Real Backend

Register an OpenAI-compatible backend explicitly:

export OPENAI_API_KEY='...'

llmff run examples/json-repair.yaml \
  --backend openai=https://api.openai.com/v1 \
  --api-key-env openai=OPENAI_API_KEY

Manifests reference that backend with model ids such as:

model: openai:gpt-4.1-mini

For a local Ollama server:

llmff run pipeline.yaml \
  --ollama ollama=http://localhost:11434

Then use model ids such as:

model: ollama:llama3.1

7. Read The Trace

The --trace file is JSONL. Summarize it with:

llmff trace /tmp/llmff-trace.jsonl

Trace output reports run status, stage status, durations, safe model metadata, validation errors, cache hits, and token usage when a backend reports it. It does not include full prompts, tool bodies, headers, cached values, or secrets.

Troubleshooting

  • llmff: command not found: confirm Cargo's bin directory is on PATH. Cargo usually installs to $HOME/.cargo/bin.
  • mock backend does not serve model: set the matching mock response variable for the model id in the manifest. mock:good uses LLMFF_MOCK_GOOD_RESPONSE; mock:bad uses LLMFF_MOCK_BAD_RESPONSE.
  • backend alias is not registered: pass --backend alias=url or --ollama alias=url, and make sure the manifest model id uses the same alias before the first colon.
  • missing API key environment variable: export the secret and pass --api-key-env alias=ENV_NAME.
  • Windows and macOS release installers are unsigned in v1.1.0; expect normal OS trust prompts until paid signing and notarization are added.

Next Steps