Skip to content

api-reference.md

Latest commit

 

History

History
609 lines (504 loc) · 15.5 KB

api-reference.md

File metadata and controls

609 lines (504 loc) · 15.5 KB
Raw

API Reference

The Litestar API is served under /api. Paths below are shown without a trailing slash.

Endpoint summary

Task images

Method Path Purpose
GET /api/task-images List task images. Defaults to enabled rows only.
GET /api/task-images/{task_image_id} Get one task image with stored and effective refs.
POST /api/task-images Create a task-image catalog row.
PUT /api/task-images/{task_image_id} Replace a task-image catalog row.
DELETE /api/task-images/{task_image_id} Delete an unreferenced task image.

GET /api/task-images query parameters:

Parameter Type Meaning
enabled bool or null Filter by enabled state; default is true.
include_disabled bool When true, disables the enabled filter.

Task configurations

Method Path Purpose
GET /api/task-configurations List benchmark definitions.
GET /api/task-configurations/{task_configuration_id} Get one benchmark definition.
POST /api/task-configurations Create a benchmark definition.
DELETE /api/task-configurations/{task_configuration_id} Delete an unreferenced benchmark definition.

Runs

Method Path Purpose
POST /api/runs/benchmark Start an ad hoc benchmark run.
POST /api/runs/suite Start a suite run.

Batches and iterations

Method Path Purpose
GET /api/batches List batches.
GET /api/batches/{batch_id} Get a batch with aggregate stats.
DELETE /api/batches/{batch_id} Delete a batch and its child rows/workspaces.
GET /api/batches/{batch_id}/iterations List iterations in one batch.
GET /api/iterations List all iterations with aggregate stats and filter options.
GET /api/iterations/{iteration_id} Get one iteration.
GET /api/iterations/{iteration_id}/code List persisted workspace files, or legacy code rows.
GET /api/iterations/{iteration_id}/code/content?path=... Read one persisted file.
GET /api/iterations/{iteration_id}/artefacts List non-code artefacts, including Pi crash diagnostics when present.
DELETE /api/iterations/{iteration_id} Delete one iteration and its child rows/workspace.

GET /api/iterations query parameters:

Parameter Meaning
task Exact batches.task_name filter.
provider Exact iteration provider filter.
model Exact iteration model filter.

Suites

Method Path Purpose
GET /api/suites List suites.
GET /api/suites/{suite_id} Get one suite.
POST /api/suites Create a suite.
PUT /api/suites/{suite_id} Update mutable suite fields.
DELETE /api/suites/{suite_id} Delete a suite that has no schedules.
POST /api/suites/{suite_id}/duplicate Duplicate a suite under a new name.
GET /api/suites/{suite_id}/runs List suite runs.
GET /api/suites/{suite_id}/runs/{run_id} Get one suite run.
POST /api/suites/{suite_id}/runs/{run_id}/complete Complete/cancel an active suite run.
DELETE /api/suites/{suite_id}/runs/{run_id} Delete a suite run and child batches.

Schedules

Method Path Purpose
GET /api/schedules List schedules with runtime next-run state.
GET /api/schedules/{schedule_id} Get one schedule.
GET /api/schedules/{schedule_id}/history List schedule dispatch history.
POST /api/schedules Create a cron schedule.
PUT /api/schedules/{schedule_id} Update cron and/or enabled state.
DELETE /api/schedules/{schedule_id} Delete a schedule and dispatch history.

Collections

Method Path Purpose
GET /api/collections List collections.
POST /api/collections Create a collection.
GET /api/collections/{collection_id} Get a collection with aggregate stats.
PUT /api/collections/{collection_id} Rename a collection.
DELETE /api/collections/{collection_id} Delete a collection.
GET /api/collections/{collection_id}/iterations List iterations in dynamically matched batches.

Metadata

Method Path Purpose
GET /api/scorers List registered scorer definitions.
GET /api/score-definitions List canonical score definitions.
GET /api/providers List supported provider IDs.
GET /api/models List model suggestions.

GET /api/models accepts optional provider=<provider-id>. Unknown providers return 400 Bad Request.

Request schemas

CreateTaskImageRequest

{
  "name": "oas-generation",
  "catalog_image_ref": "ghcr.io/example/park-bench-task-oas-generation:v1",
  "enabled": true
}

name and catalog_image_ref must be non-empty. The repository enforces unique name and unique stored image_ref.

UpdateTaskImageRequest

Same fields as create; the row is replaced with the supplied values.

CreateTaskConfigurationRequest

{
  "name": "oas-generation-pytest",
  "task_image_id": 1,
  "scorers": ["pytest", "complexity"]
}

Validation:

  • name must be non-empty
  • task image must exist and be enabled
  • scorer list must be non-empty
  • scorer names must be unique and registered
  • registry-declared scorer dependencies must be included
  • duplicate (task_image_id, canonical scorer set) identities are rejected

BenchmarkRunRequest

{
  "task_configuration_id": 1,
  "iterations": 3,
  "provider": "anthropic-api",
  "model": "claude-sonnet-4-6",
  "llm_timeout": 7200,
  "container_timeout": 7200,
  "max_workers": 1
}

Defaults:

Field Default
iterations 1
provider ""
model ""
llm_timeout 7200
container_timeout 7200
max_workers 1

When a provider string is supplied, it must exist in GET /api/providers.

SuiteRunRequest

{
  "suite_id": 1
}

CreateSuiteRequest

{
  "name": "nightly-oas-generation",
  "task_configuration_id": 1,
  "iterations": 5,
  "max_workers": 2,
  "providers": [
    {"provider": "anthropic-api", "model": "claude-sonnet-4-6", "reasoning": "xhigh"},
    {"provider": "openai-api", "model": "gpt-5.4"}
  ]
}

Provider entries are parsed as ProviderConfig with fields provider, model, and reasoning. reasoning defaults to xhigh if omitted. Provider IDs are validated when non-empty.

UpdateSuiteRequest

All fields are optional:

{
  "name": "nightly-oas-generation-updated",
  "iterations": 10,
  "max_workers": 4,
  "providers": [
    {"provider": "openai-api", "model": "gpt-5.4", "reasoning": "xhigh"}
  ]
}

Suite updates cannot change task_configuration_id.

DuplicateSuiteRequest

{
  "name": "copy-of-nightly-oas-generation"
}

name must be non-empty.

CreateScheduleRequest

{
  "suite_id": 1,
  "cron": "0 3 * * *",
  "enabled": true
}

The cron expression is parsed with CronTrigger.from_crontab.

UpdateScheduleRequest

Both fields are optional:

{
  "cron": "30 2 * * 1",
  "enabled": false
}

CreateCollectionRequest

{
  "name": "anthropic-sonnet-baseline",
  "task_configuration_id": 1,
  "provider": "anthropic-api",
  "model": "claude-sonnet-4-6"
}

The controller verifies that the task configuration exists. Provider/model values are stored as supplied; collection creation does not validate provider IDs.

UpdateCollectionRequest

{
  "name": "renamed-collection"
}

name must be non-empty.

Response schemas

Task image list item

{
  "id": 1,
  "name": "oas-generation",
  "image_ref": "ghcr.io/example/park-bench-task-oas-generation:v1",
  "enabled": true
}

image_ref is the effective ref after local-registry override, when configured.

Task image detail

{
  "id": 1,
  "name": "oas-generation",
  "catalog_image_ref": "ghcr.io/example/park-bench-task-oas-generation:v1",
  "effective_image_ref": "127.0.0.1:5000/park-bench-task-oas-generation:dev",
  "enabled": true,
  "created_at": "2026-04-28T12:00:00",
  "updated_at": "2026-04-28T12:00:00"
}

Task configuration

{
  "id": 1,
  "name": "oas-generation-pytest",
  "task_image_id": 1,
  "task_name": "oas-generation",
  "task_image_ref": "ghcr.io/example/park-bench-task-oas-generation:v1",
  "scorers": ["complexity", "pytest"],
  "created_at": "2026-04-28T12:00:00"
}

Batch summary/detail

{
  "id": 10,
  "suite_run_id": null,
  "task_configuration_id": 1,
  "run_timestamp": "2026-04-28T12:30:00",
  "task_name": "task",
  "provider": "anthropic-api",
  "model": "claude-sonnet-4-6",
  "user_prompt": "/harness",
  "system_prompt": "",
  "user_template_prompt": "",
  "task_image_ref": "ghcr.io/example/park-bench-task-oas-generation:v1",
  "stats": {},
  "total_samples": 0
}

GET /api/batches/{id} adds stats and total_samples. stats is keyed by canonical score_id values and uses each score definition's null_aggregation policy when aggregating persisted null values. task_name is the runtime Task.name captured when the batch was created.

Iteration

{
  "id": 100,
  "batch_id": 10,
  "iteration": 1,
  "provider": "anthropic-api",
  "model": "claude-sonnet-4-6",
  "reasoning": "xhigh",
  "duration": 182,
  "sessionStats": {
    "sessionFile": "/tmp/session.jsonl",
    "sessionId": "abc123",
    "userMessages": 1,
    "assistantMessages": 6,
    "toolCalls": 4,
    "toolResults": 4,
    "totalMessages": 11,
    "tokens": {
      "input": 1234,
      "output": 456,
      "cacheRead": 0,
      "cacheWrite": 0,
      "total": 1690
    },
    "cost": 0.09
  },
  "outcome": "success",
  "terminal_state": "completed",
  "scoreable": true,
  "error_message": null,
  "exit_code": 0,
  "scores": [
    {"score_id": "duration", "value": 182, "detail": ""},
    {"score_id": "percentage_passed", "value": 100, "detail": ""}
  ],
  "workspace_path": "/data/runs/100"
}

For Pi non-zero exits, error_message contains a concise extracted crash summary. Detailed stderr diagnostics are exposed as artefacts, not inline on the iteration response.

Iteration list

GET /api/iterations returns:

{
  "iterations": [],
  "stats": {"metrics": {}, "total_samples": 0},
  "filters": {"tasks": [], "providers": [], "models": []}
}

Code file summary/detail

GET /api/iterations/{id}/code returns summaries:

{
  "path": "solution/app.py",
  "kind": "file",
  "size_bytes": 1234,
  "has_changes": true,
  "added_lines": 12,
  "deleted_lines": 3
}

GET /api/iterations/{id}/code/content?path=solution/app.py returns:

{
  "path": "solution/app.py",
  "kind": "file",
  "contents": "print('hello')\n",
  "diff": "--- original/solution/app.py\n+++ iteration/solution/app.py\n"
}

Artefacts

GET /api/iterations/{id}/artefacts returns non-code artefacts:

[
  {
    "label": "pi_crash_diagnostics",
    "content_type": "application/json",
    "body": "{\n  \"returncode\": 1,\n  \"summary\": \"Error: write ENOBUFS; code: 'ENOBUFS'; syscall: 'write'\"\n}"
  },
  {
    "label": "pi_stderr_tail",
    "content_type": "text/plain",
    "body": "Error: write ENOBUFS\ncode: 'ENOBUFS'\nsyscall: 'write'"
  }
]

Common Pi labels are pi_session_jsonl, pi_chat_export, pi_crash_diagnostics, and pi_stderr_tail. Task images can also expose a configured status artefact such as pied_pi_status.

Suite

{
  "id": 1,
  "name": "nightly-oas-generation",
  "task_configuration_id": 1,
  "config": {
    "iterations": 5,
    "max_workers": 2,
    "providers": [
      {"provider": "anthropic-api", "model": "claude-sonnet-4-6", "reasoning": "xhigh"}
    ]
  },
  "created_at": "2026-04-28T12:00:00",
  "updated_at": "2026-04-28T12:00:00",
  "run_count": 3,
  "latest_run_status": "completed"
}

Suite run detail

{
  "id": 12,
  "suite_id": 1,
  "status": "completed",
  "accepted_at": "2026-04-28T12:00:00",
  "launched_at": "2026-04-28T12:00:01",
  "running_at": "2026-04-28T12:00:02",
  "completed_at": "2026-04-28T12:10:00",
  "batch_count": 2,
  "total_cost": 0.42,
  "trigger_source": "manual",
  "schedule_dispatch_id": null,
  "schedule_id": null,
  "config_snapshot": {
    "name": "nightly-oas-generation",
    "iterations": 5,
    "max_workers": 2,
    "providers": [
      {"provider": "anthropic-api", "model": "claude-sonnet-4-6", "reasoning": "xhigh"}
    ],
    "task_configuration": {
      "id": 1,
      "name": "oas-generation-pytest",
      "task_image_id": 1,
      "task_name": "oas-generation",
      "scorers": ["complexity", "pytest"],
      "task_image_ref": "ghcr.io/example/park-bench-task-oas-generation:v1"
    }
  },
  "batch_ids": [10, 11]
}

Suite run status values are accepted, launching, running, completed, or failed.

Schedule

{
  "id": 1,
  "suite_id": 1,
  "suite_name": "nightly-oas-generation",
  "cron": "0 3 * * *",
  "enabled": true,
  "next_run": "2026-04-29T03:00:00+00:00",
  "last_dispatch_status": "accepted",
  "last_dispatch_message": "Suite run started for nightly-oas-generation",
  "last_dispatched_at": "2026-04-28T03:00:00+00:00",
  "created_at": "2026-04-28T12:00:00",
  "updated_at": "2026-04-28T12:00:00"
}

Schedule dispatch statuses are accepted, busy, invalid, not_found, or error.

Scorer response

{
  "id": "complexity",
  "label": "Cyclomatic Complexity",
  "description": "Computes total cognitive complexity across source files.",
  "requires_judge_client": false,
  "requires_original_codebase": false,
  "dependencies": [],
  "scores": [
    {
      "id": "cognitive_complexity",
      "source_kind": "scorer",
      "source_id": "complexity",
      "label": "Cognitive Complexity",
      "description": "Aggregate cognitive complexity score.",
      "group": {"id": "quality", "label": "Code Quality"},
      "unit": {"kind": "plain"},
      "format": {"kind": "integer"},
      "direction": "lower",
      "expected_range": null,
      "thresholds": [
        {"min": null, "max": 10.0, "tone": "good"},
        {"min": 10.0, "max": 20.0, "tone": "warning"},
        {"min": 20.0, "max": null, "tone": "bad"}
      ],
      "compact_visible": false,
      "detail_visible": true,
      "chartable": true,
      "trendable": true,
      "null_aggregation": "skip"
    }
  ]
}

Collection summary/detail

{
  "id": 1,
  "name": "anthropic-sonnet-baseline",
  "task_configuration_id": 1,
  "task_configuration_name": "oas-generation-pytest",
  "task_name": "oas-generation",
  "scorers": ["complexity", "pytest"],
  "provider": "anthropic-api",
  "model": "claude-sonnet-4-6",
  "batch_count": 3,
  "iteration_count": 15,
  "created_at": "2026-04-28T12:00:00",
  "updated_at": "2026-04-28T12:00:00"
}

Collection detail adds batch_ids and stats.

Run trigger response

{
  "status": "accepted",
  "message": "Benchmark started for oas-generation-pytest"
}

Common error responses

  • 400 Bad Request: invalid provider, invalid cron, invalid run dispatch, missing task configuration during suite/collection creation, or request validation errors that the controller maps to a client error.
  • 404 Not Found: requested row does not exist.
  • 409 Conflict: run lock is busy, duplicate task image values, duplicate task configuration identity, attempted deletion of referenced rows, or completing a suite run that is already terminal.