Skip to content

getting-started.md

Latest commit

 

History

History
239 lines (183 loc) · 6.18 KB

getting-started.md

File metadata and controls

239 lines (183 loc) · 6.18 KB
Raw

Getting Started

This guide starts Park Bench from a clean checkout with an empty application SQLite database.

Prerequisites

  • Docker and Docker Compose.
  • Credentials for the coding provider you want to benchmark.
  • Credentials for the judge provider required by that coding provider.
  • A pullable Docker task image containing /task/task-manifest.json.

For local development outside Docker you also need Python 3.12+, uv, and Node.js 22+ for frontend work.

1. Clone and configure

git clone <repository-url>
cd park-bench
cp .env.example .env

Edit .env with the credentials needed by your selected provider. Common pairs:

Coding provider Coding credential Judge credential
anthropic-api ANTHROPIC_API_KEY OPENAI_API_KEY
anthropic-max CLAUDE_CODE_OAUTH_TOKEN OPENAI_API_KEY
openai / openai-api OPENAI_API_KEY ANTHROPIC_API_KEY
openai-plus Pi auth.json entry openai-codex ANTHROPIC_API_KEY
google GEMINI_API_KEY ANTHROPIC_API_KEY
openrouter OPENROUTER_API_KEY ANTHROPIC_API_KEY

See LLM Clients for the full provider registry and auth-json mount options.

2. Start the stack

docker compose up -d --build

Open:

On first startup, bootstrap_database() creates schema version 14. The task-image catalog starts empty.

Older Park Bench application schemas are not migrated by this release. Start with a fresh data/results.db when upgrading from an older schema family.

3. Register a task image

The image must already be available to the Docker daemon used by the server. Register it in the catalog:

curl -X POST http://localhost:8080/api/task-images \
  -H "Content-Type: application/json" \
  -d '{
    "name": "oas-generation",
    "catalog_image_ref": "ghcr.io/danballance/park-bench-task-exercise-oas-generation:v1",
    "enabled": true
  }'

Response shape:

{
  "id": 1,
  "name": "oas-generation",
  "catalog_image_ref": "ghcr.io/danballance/park-bench-task-exercise-oas-generation:v1",
  "effective_image_ref": "ghcr.io/danballance/park-bench-task-exercise-oas-generation:v1",
  "enabled": true,
  "created_at": "2026-04-28T12:00:00",
  "updated_at": "2026-04-28T12:00:00"
}

catalog_image_ref is the stored value. effective_image_ref is the value after an optional local-registry override.

4. Create a task configuration

A task configuration is the immutable benchmark identity used by ad hoc runs, suites, and collections.

curl -X POST http://localhost:8080/api/task-configurations \
  -H "Content-Type: application/json" \
  -d '{
    "name": "oas-generation-pytest",
    "task_image_id": 1,
    "scorers": ["pytest", "complexity"]
  }'

The backend checks that the task image exists and is enabled, and that the scorer list is non-empty, unique, registered, and dependency-complete. The scorer set is stored in canonical sorted order for uniqueness.

5. Trigger an ad hoc benchmark run

curl -X POST http://localhost:8080/api/runs/benchmark \
  -H "Content-Type: application/json" \
  -d '{
    "task_configuration_id": 1,
    "iterations": 3,
    "provider": "anthropic-api",
    "model": "claude-sonnet-4-6",
    "llm_timeout": 7200,
    "container_timeout": 7200,
    "max_workers": 1
  }'

The endpoint returns 202 Accepted when the run is accepted for background execution. RunLock allows only one active benchmark or suite run at a time.

Ad hoc benchmark requests do not include a reasoning field; the runtime uses the default Pi thinking level xhigh.

6. Create and run a suite

A suite runs one task configuration across one or more provider/model lanes. Provider entries can include reasoning; when omitted, Pydantic defaults it to xhigh.

curl -X POST http://localhost:8080/api/suites \
  -H "Content-Type: application/json" \
  -d '{
    "name": "nightly-oas-generation",
    "task_configuration_id": 1,
    "iterations": 3,
    "max_workers": 1,
    "providers": [
      {"provider": "anthropic-api", "model": "claude-sonnet-4-6", "reasoning": "xhigh"},
      {"provider": "openai-api", "model": "gpt-5.4"}
    ]
  }'

Trigger the suite:

curl -X POST http://localhost:8080/api/runs/suite \
  -H "Content-Type: application/json" \
  -d '{"suite_id": 1}'

7. View results

Use the UI or these API endpoints:

  • GET /api/batches
  • GET /api/batches/{batch_id}
  • GET /api/batches/{batch_id}/iterations
  • GET /api/iterations/{iteration_id}
  • GET /api/iterations/{iteration_id}/artefacts
  • GET /api/iterations/{iteration_id}/code
  • GET /api/iterations/{iteration_id}/code/content?path=...

Iteration payloads include normalized scores rows of the form {score_id, value, detail}. Aggregate stats are keyed by the same score_id values.

8. Schedule a suite

curl -X POST http://localhost:8080/api/schedules \
  -H "Content-Type: application/json" \
  -d '{
    "suite_id": 1,
    "cron": "0 3 * * *",
    "enabled": true
  }'

Enabled schedules are loaded into APScheduler during application startup and manual schedule updates update the in-memory APScheduler job list.

Local task-image registry mode

For local task-image development, start the registry override:

docker compose \
  -f docker-compose.yaml \
  -f docker-compose.local-task-registry.yaml \
  up -d --build

The override starts a registry at 127.0.0.1:5000 and sets:

TASK_IMAGE_LOCAL_REGISTRY=127.0.0.1:5000
TASK_IMAGE_LOCAL_TAG=dev

When both are set, Park Bench rewrites effective refs in memory by preserving the final repository component and replacing the registry/tag. For example:

ghcr.io/danballance/park-bench-task-exercise-oas-generation:v1

becomes:

127.0.0.1:5000/park-bench-task-exercise-oas-generation:dev

The database row is not modified.

Local development commands

curl -LsSf https://astral.sh/uv/install.sh | sh
uv sync

uv run pytest --cov park_bench --cov-report term-missing tests
uv run ruff check --fix park_bench
uv run ruff format park_bench
uv run python .github/scripts/check_complexity.py

For UI work:

cd ui
npm install
npm run build