feat(db): add global soft-delete query filter

[daryllimyt]

Summary

• Adds a global ORM soft-delete filter: a do_orm_execute listener on Session applies with_loader_criteria(SoftDeleteMixin, deleted_at IS NULL, include_aliases=True) to top-level ORM SELECTs, so soft-deleted rows disappear from product queries by default.
• Opt-out is explicit and narrow: with_deleted(stmt) / the include_deleted execution option, reserved for staging/provenance/sync-reconciliation/erasure helpers (later PRs). Other uses are a review smell.
• Ships a behavior-pinning coverage suite (9 tests) so the filter's exact semantics are documented executable behavior, not tribal knowledge.

Part of ENG-1517 (version-less resource refs). Stacks on #2931, which introduced SoftDeleteMixin (columns-only) on AgentPreset. This addresses the with_loader_criteria suggestion raised in #2931's review — including its caveats, which are pinned as tests here.

Review guide

• Start with tracecat/db/soft_delete.py (~80 lines, the whole mechanism): listener guards (is_select and not is_column_load and not is_relationship_load), propagate_to_loaders=False, the INCLUDE_DELETED execution option, and assert_soft_delete_listener_registered().
• The contract, stated in the module docstring: relationship loads are deliberately unfiltered (UUID/historical paths must keep reaching tombstones); warm Session.get() identity-map hits bypass SQL and therefore the filter; UPDATE/DELETE are unfiltered; Core/text statements are unfiltered.
• Wiring (db/engine.py, api/app.py): registration is an import side-effect mirroring session_events; the API lifespan asserts registration at startup so a missing import fails fast in prod instead of silently no-op'ing. Worker processes get registration transitively via engine.py.
• tests/unit/test_soft_delete_filter.py: one test per pinned behavior — entity selects, mapped-column tuple selects (filtered — verified), aliased joins, relationship loads (see tombstones), cold vs warm Session.get(), RLS-bypass sessions (still filtered), UPDATE/DELETE (unfiltered), direct-engine sessions (import wiring canary).
• Audit: AgentPresetService.get_preset(include_deleted=True) / get_preset_by_slug(include_deleted=True) now also set the execution option. All other deleted_at predicates were audited and deliberately left in place — redundant filtering is rollback-safe, and they're stripped only in a later cleanup once this suite has baked. File/case-comment deleted_at columns don't inherit the mixin and are untouched.

Note

We will be updating the explicit filters introduced in the original PR upstream

Back-compat / rollback

• No schema change, no API change. Existing explicit predicates keep every active-only path correct even if this PR is reverted.
• Behavior-neutrality verified: 281 existing tests across preset/skill/channels/folders/sync-contract/session-versioning suites pass unchanged with the listener active.

Validation

• uv run pytest tests/unit/test_soft_delete_filter.py — 9 passed
• Regression suites (preset, skill, channels, folders, sync acceptance contract, session versioning) — 281 passed
• uv run ruff check . / format --check clean; uv run basedpyright --warnings — 0 errors, 0 warnings

---

Summary by cubic

Adds a global ORM SELECT filter to hide soft-deleted rows by default, with a with_deleted() escape hatch. Part of ENG-1519.

• New Features

  • Global criteria on SoftDeleteMixin adds deleted_at IS NULL to ORM SELECTs; relationship loads and UPDATE/DELETE remain unfiltered.
  • with_deleted(stmt) for narrow opt-outs; AgentPresetService uses it when include_deleted=True.
  • Listener registered via side-effect import in tracecat.db.engine; api/app.py asserts with assert_soft_delete_listener_registered(). Tests cover SELECTs, joins, relationship loads, cold/warm Session.get(), RLS-bypass, direct-engine sessions, and a subprocess check for engine wiring.

• Migration

  • No action needed. Only wrap queries with with_deleted(stmt) when tombstones are intentionally required.

^{Written for commit ab533d6. Summary will update on new commits.}

[daryllimyt]

• refactor(agent): contract legacy resource ownership #3024
• feat(agent): resolve resource graph at dispatch #3008
• feat(agent): record per-turn resource provenance #3007
• refactor(agent): align skill slugs with published packages #3025
• refactor(agent): resolve preset resources through heads #3006
• refactor(agent): normalize preset subagent head edges #3017
• feat(agent): make preset skill bindings head-only #3005
• feat(agent): add pinned_version_id columns and pin API #3004
• refactor(agent): resolve skill identifiers by id then live slug #3003
• feat(agent): add stable skill slugs with live uniqueness #3001
• feat(agent): expand skill soft delete to deleted_at #2999
• feat(db): add global soft-delete query filter #2998 👈 (View in Graphite)
• main

This stack of pull requests is managed by Graphite. Learn more about stacking.

[zeropath-ai]

✅ No security or compliance issues detected. Reviewed everything up to ab533d6.

Security Overview

• 🔎 Scanned files: 6 changed file(s)
• 🔗 Scan Link: https://zeropath.com/app/repositories/00dffd6c-8834-4dc9-b6d8-b44cd1622986?scanId=ecf5a6cd-d923-4e44-bbfa-92a522849d95&codeScanTypes=PrScan&tab=issues

Detected Code Changes

Change Type	Relevant files
Enhancement	► tracecat/agent/preset/service.py       import with_deleted from tracecat.db.soft_delete       use with_deleted when include_deleted is true in get_preset/get_preset_by_slug
Enhancement	► tracecat/api/app.py       import assert_soft_delete_listener_registered       call assert_soft_delete_listener_registered() in lifespan
Enhancement	► tracecat/db/engine.py       ensure soft_delete module is imported for listener registration (side effect import)
Enhancement	► tracecat/db/models.py       update SoftDeleteMixin docstring to mention global ORM SELECT filtering lives in tracecat.db.soft_delete
Enhancement	► tracecat/db/soft_delete.py ▸ new file: implement global ORM criteria for soft-delete, with_deleted helper, and listener registration assertion
Enhancement	► tracecat/db/engine.py ▸ (relation) import side-effect to register listeners via soft_delete
Bug Fix / Refactor	► tracecat/api/app.py ▸ import and use assert_soft_delete_listener_registered to ensure listener presence at startup
Other	► test file addition: tests/unit/test_soft_delete_filter.py (new tests covering soft-delete filtering behavior under various scenarios)

[daryllimyt]

@codex review

[cubic-dev-ai]

All reported issues were addressed across 6 files

Confidence score: 5/5

• Safe to merge after the addressed issues were fixed.

<sub>Reply with feedback, questions, or to request a fix.

Re-trigger cubic</sub>

[chatgpt-codex-connector]

Codex Review: Didn't find any major issues. You're on a roll.

Reviewed commit: bb95f5a935

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[jordan-umusu]

lgtm

[daryllimyt]

Merge activity

• Jul 8, 7:14 PM UTC: A user started a stack merge that includes this pull request via Graphite.