Skip to content

Latest commit

 

History

History
170 lines (136 loc) · 7.4 KB

File metadata and controls

170 lines (136 loc) · 7.4 KB

Testing strategy

The repository uses deterministic tests as the merge gate and records live integration limits separately. The post-tag runtime/CI evidence baseline is 6b8e9e5; documentation commits may advance the checkout after this ledger is written. See docs/release.md and docs/ci.md for the checkpoint ledger.

Test layers

Contracts

The contracts package builds its test TypeScript and runs its Node test suite. Build contracts before API tests because the API consumes the workspace package:

pnpm --filter @repomentor/contracts build
pnpm --filter @repomentor/contracts test

API

The API test script builds test TypeScript and runs Node's test runner over the compiled test tree. It covers auth, health, guest review, AI policy/provider adapters, review processing, Redis primitives/executors, quota admission, usage/history, security transport/CSRF provenance, and Prisma repository seams. Most integration-shaped tests use in-memory repositories, fake Luna, or deterministic Redis executors.

pnpm --filter @repomentor/api test

Security transport tests cover the configured browser Origin, Referer fallback, absent headers in development/test compatibility mode, production rejection of absent provenance, malformed/null/disallowed origins, malformed Referer values, and Origin/Referer mismatches. The check is limited to the two cookie-authenticated mutation routes.

Web

The web shell tests exercise route and UI contracts, including authenticated settings and review detail behavior. The production build and TypeScript/lint checks are separate gates:

pnpm --filter @repomentor/web test
pnpm --filter @repomentor/web typecheck
pnpm --filter @repomentor/web lint
pnpm --filter @repomentor/web build

Playwright discovery is available through pnpm --filter @repomentor/web e2e, but no browser execution is claimed when the required Chromium revision is not installed.

Opt-in live Compose review path

The repository also has a deliberately separate, opt-in persistence check:

pnpm test:live-compose-review-path

Implementation anchor: ef10db3; the hosted workflow anchor is 6b8e9e5.

The command generates a unique Compose project and reserves unique currently free loopback ports for PostgreSQL and Redis. It applies the separate docker-compose.live-review.yml override only for this temporary project so those services also join the non-internal edge network; the production docker-compose.yml is unchanged. It starts only those two services, applies the checked-in Prisma migrations from the host, and runs the Nest API HTTP surface in-process with FakeAiReviewProvider injected through the existing testing seam. The child environment omits Luna, OpenAI, and DeepSeek credentials and fails if the Luna variables are present. The test registers and authenticates a user, admits and processes a QUICK review, checks persisted result/usage and usage history, verifies Redis-backed quota denial, and blocks unexpected global fetch calls.

Cleanup validates the generated Compose project label on its containers, volumes, and networks before running docker compose down --volumes --remove-orphans for that project, then verifies those labeled resources are gone. This is local integration evidence for the exact run only; it does not prove the production API/web images, external Luna behavior, multi-instance operation, deployment, or production readiness.

The opt-in real-browser journey uses the same harness and infrastructure:

pnpm test:live-browser-review-journey

Implementation anchor: 7ee0ddb. This is an opt-in browser path, not a hosted browser CI run.

This mode keeps PostgreSQL and Redis in the unique Compose project, starts the Nest API in a child process with the existing FakeAiReviewProvider seam, and lets Playwright start the Next web server on another unique loopback port. The browser calls the live API over HTTP; the journey does not intercept or mock its auth, review, history, usage, or quota routes. It covers registration, login, Monaco editor submission, persisted completion/history/usage display, and a second QUICK admission that returns 429 and renders the workspace retry state. The API child omits Luna, OpenAI, and DeepSeek credentials and replaces global fetch with a throwing external-call guard; the run requires one Fake AI request and zero external fetches. Chromium must already be installed for the Playwright package. The browser's configured web Origin is allowlisted by the API, so the journey also exercises the permitted cookie provenance path. This remains local evidence for the exact run and does not certify production images, external-provider behavior, deployment, or production readiness.

The non-browser persistence path is also exercised by the credential-free .github/workflows/live-compose-integration.yml workflow on relevant pull requests, pushes to main, and manual dispatch. That job uses the same pnpm test:live-compose-review-path command on a Docker-enabled hosted runner with a 30-minute bound and no provider or registry credentials. Its pass is hosted live PostgreSQL/Redis, migration, persistence, usage/history, quota, and Fake AI guard evidence for that exact run; it is not browser or production deployment evidence.

The hosted evidence is Live Compose run 31292112691 and Application Gates run 31292112688, both at source head 6b8e9e5. The post-tag CSRF provenance tests are anchored at a783b28; the timer-global fix at a57ea9d keeps the live harness lint-safe on Linux. These anchors do not claim external Luna behavior or deployment.

Static and package gates

The root checks are:

pnpm format:check
pnpm lint
pnpm typecheck
pnpm build
pnpm package:check
pnpm db:validate
pnpm db:generate
pnpm audit --audit-level=high

package:check inspects package payloads without publishing. The dependency audit is fail-closed; a high/critical advisory is a visible blocker, not a reason to bypass the check. Credential-shaped scans and git diff --check are also required for integration.

The canonical CI ordering and permissions are in application-gates.yml. The container workflow separately validates Dockerfiles/Compose and performs no-publish image builds when a Docker runner is available.

Evidence boundaries

Passing deterministic tests proves the tested code paths with their fakes and in-memory seams. The bounded live Compose path adds one real PostgreSQL/Redis and Fake AI evidence point, but it does not prove:

  • production PostgreSQL transaction isolation, backup/restore, or reliability beyond the bounded migration and persistence run;
  • full live Redis authentication/EVAL semantics, streams, or a multi-instance process-lock lease beyond the bounded quota path;
  • external Luna HTTP access, model output quality, provider quota, or latency;
  • hosted browser execution, production UI/API serving behind a proxy, or multi-instance browser behavior; the browser journey remains opt-in/local;
  • production image behavior, registry publication, GitHub release, or deployment. The hosted job proves only its disposable Compose project.

Those claims require a separate environment with explicit credentials and recorded run identity. Do not add real secrets to make a local deterministic test appear live.