Design: CI benchmark golden templates + per-run working copies for mutable datasets

[Xuanwo]

Summary

Mutable ci_benchmarks targets (today: merge_insert_*) should live as immutable golden templates under a fixed storage prefix. Each benchmark run materializes a private working copy, runs all mutate / restore traffic against that copy only, and deletes the copy when finished.

Golden creation remains the existing datagen path (write data, build indexes, stamp merge_insert_base). When a usable golden already exists, the run copies it instead of rebuilding or mutating it in place.

This keeps cold-start cost on the order of a one-time full gen (~15 minutes measured on the regression runner for the five merge_insert_* targets), keeps steady-state setup to a prefix copy rather than index rebuild, and removes cross-run commit conflicts on a shared tip.

Related:

• CI failure example: https://github.com/lance-format/lance/actions/runs/30971041362/job/92195205095
• Orthogonal OOM issue (not solved by this design): bug: ci_benchmarks merge_insert 100pct-v1_indexed OOM under 150MB mem pool #8269, Benchmark failure: merge-insert hash join exceeds bounded memory pool #8129

Problem

merge_insert benchmarks must mutate a target, then return it to a pristine baseline between measured rounds. The suite does this with a merge_insert_base tag and checkout_version + restore before each round, and cleans up versions at module teardown.

Those targets currently resolve to shared fixed URIs (gs://lance-benchmarks-ci-datasets/merge_insert_* in CI). Multiple processes (or leftover state from failed teardowns) therefore commit Restore and Update against the same tip.

Lance treats Update vs concurrent Restore as a hard conflict. Cleanup can also remove manifests another handle still expects. The regression job has failed repeatedly with those errors.

Datagen already skips recreation when a tagged baseline with the expected row count exists, so steady-state runs do not rebuild multi-GB datasets; they share and rewrite the same ones. Isolation is currently approximated by workflow concurrency serialization, which is brittle and does not protect golden content.

Proposed Design

Roles

Golden template. Canonical, durable dataset for a logical name (e.g. merge_insert_narrow). Contains data files, indexes, and a baseline tag (merge_insert_base). After successful creation, no benchmark path commits to a golden URI.

Working copy. Per-run private dataset obtained by copying a golden. Benchmarks open only working URIs. Round reset (restore to baseline tag) and teardown cleanup run only on the working copy.

Run scope. One CI job (or local invocation) owns one run_id and one working prefix. Concurrent jobs use different run_ids and never share working tips.

URI layout

{base}/templates/{name}           # golden
{base}/runs/{run_id}/{name}       # working copy

• {base} remains environment-selected (gs://lance-benchmarks-ci-datasets/ on GCP; ~/lance-benchmarks-ci-datasets/ locally).
• {name} is unchanged (merge_insert_narrow, merge_insert_wide, …).
• {run_id} is required when preparing working copies. CI uses github.run_id + github.run_attempt (or equivalent unique token); local runs use an explicit env var or a generated id.

Read-only datasets that are never restored or merge-inserted (e.g. basic, count_rows, wikipedia) keep today’s single-URI layout under {base}/{name} unless later opted into the same model. This design applies to mutable names only; the mutable set is declared explicitly (initially the five merge_insert_* generators).

Lifecycle

prepare(run_id):
  for each mutable name:
    if golden missing or not usable:
      generate golden at templates/{name}   # existing write + index + tag
    copy golden → runs/{run_id}/{name}      # full Lance tree (data, _versions, indexes, tags)

benchmarks:
  open only runs/{run_id}/{name}
  per round: checkout baseline tag + restore on that handle
  module teardown: restore + cleanup_old_versions on working copy only

finish(run_id):
  delete runs/{run_id}/                     # best-effort; safe to leave orphans for a janitor

Usable golden means the same class of checks datagen already uses: dataset opens, baseline tag exists, row count at the tagged version matches the generator contract. Schema or generator contract changes require a golden invalidation rule (below); “path exists” alone is insufficient after a definition change.

Copy means a storage-level recursive copy of the Lance directory tree so the working dataset is byte-equivalent and immediately openable—no index rebuild. Implementation may use the object-store recursive copy available in the environment; semantics are “new independent dataset URI with the same logical contents,” not a shared-manifest shallow clone.

Generate remains the current generator implementation, writing to the template URI with mode=overwrite only when the golden is missing or unusable. Generators do not write under runs/.

API surface (ci_benchmarks)

Minimal contract change:

• get_template_uri(name) → golden path for generators and prepare.
• get_dataset_uri(name) → for mutable names, the working URI for the current run_id; for read-only names, the existing single path. Benchmarks keep calling get_dataset_uri.
• Prepare step (datagen / workflow) ensures working copies exist before pytest starts; it is not latent inside the first test that opens a dataset.
• run_id is supplied by environment (e.g. LANCE_CI_BENCH_RUN_ID) so URI resolution stays free of GitHub-specific imports.

Invariants:

1. No merge_insert, restore, tag update, or cleanup_old_versions against a template URI from the benchmark suite.
2. Working copies are never reused across different run_ids.
3. A successful prepare leaves every mutable name openable at the working URI with a valid baseline tag.
4. Failure mid-prepare must not leave benchmarks pointing at a half-copied tree (copy to a temp prefix then rename/swap, or delete incomplete working name before exit).

Cost (measured / expected)

On the regression runner writing to the shared GCS bucket:

Action	Order of magnitude
Full generate of all five merge_insert_* goldens	~15 min measured (dominated by merge_insert_frags: 10M rows / 10k fragments)
Steady-state prepare when all goldens exist	copy only — expected well below full gen; object-count heavy for frags
Current skip-only “generate” when shared datasets already exist	~3–5 s measured (no isolation)

Logical table payload is on the order of ~5 GB raw across the five targets, plus BTREE indexes (string keys on 10M rows are material). Per-run storage is one extra full copy until teardown.

Scope boundary

• Does not change merge_insert product semantics, memory-pool limits, or which shapes are parametrized (100pct-v1_indexed OOM is separate: bug: ci_benchmarks merge_insert 100pct-v1_indexed OOM under 150MB mem pool #8269).
• Does not require a Lance native “dataset clone” API; storage recursive copy is sufficient.
• Does not mandate removing workflow concurrency immediately; private working prefixes make overlap safe, so serialization becomes an optional cost control rather than a correctness requirement.
• Golden rebuilds that race two jobs on the same template URI still need a single-writer discipline (conditional create, lease, or keep workflow concurrency for the prepare-golden phase only).

Compatibility and Migration

Existing shared paths {base}/merge_insert_* become legacy. Migration options:

1. Treat current URIs as goldens: one-time move/rename into templates/, or configure template URI to the legacy path until a cutover.
2. Leave legacy in place unused: first run after deploy recreates goldens under templates/ (~15 min once), then copy forever after.

Preference for a clean tip: new templates/ + one cold generate, rather than carrying thousands of dirty versions from the legacy shared datasets.

Local developers: without LANCE_CI_BENCH_RUN_ID, prepare generates a local run id; templates and runs both live under the local base directory. Behavior matches CI with smaller operational risk.

Benchmark result continuity (bencher time series) is unaffected: measured code paths and logical dataset shapes stay the same; only the storage URI and isolation model change.

Golden invalidation: when generator parameters (row counts, schemas, indexed columns, tag semantics) change, bump an explicit template revision (constant or hash of the generator contract) stored alongside the golden (e.g. a small sidecar object or dataset tag). Prepare treats revision mismatch as “unusable” and regenerates the golden. Row-count-only checks remain necessary but not sufficient.

Open choices for discussion

1. Scope: mutable merge_insert_* only first, or all ci_benchmarks datasets?
2. Legacy path: migrate existing shared datasets into templates/, or cold-create clean templates once?
3. Copy implementation: workflow shell (gcloud storage cp -r) vs in-process object-store copy for one path across local and CI?
4. Concurrency: keep full workflow serialization, or only serialize golden rebuild?
5. merge_insert_frags: accept object-heavy copy first, or special-case (fewer fragments / lighter private gen) if prepare is too slow?

Feedback welcome—especially on migration of the existing GCS datasets and whether copy should live in the workflow or in ci_benchmarks itself.

[westonpace]

I think we can get rid of the bench_regress job.

However, if you want to pull creation of bench datasets out of the benchmark itself then I'm conditionally in favor. We already do this for the image_eda dataset. However, we do the following:

1. Detect if the code is currently running in a VM on the Google cloud (we do this by trying to ping the metadata endpoint)
2. If the runner is on the Google cloud then we expect the GCS dataset to already exist and be created and we skip the dataset creation phase
3. If the runner is not on the Google cloud then we either skip the benchmark or we create a small local copy of the dataset, ideally the second.

In other words, it is good to allow our benchmarks to run against massive datasets and cloud storage. We don't want to be constantly recreating those datasets. However, it is also good to allow our benchmarks to be run locally as developers are tweaking things. So we should have a fallback to something that can be generated on the fly.

[Xuanwo]

I'm open to just remove bench_regress