Remove continual tasks (CT) feature by antiguru · Pull Request #35967 · MaterializeInc/materialize

antiguru · 2026-04-11T15:08:13Z

Summary

Remove the entire continual tasks (CT) feature from the codebase
Add catalog migration v81→v82 that deletes any existing CT items and comments
Preserve audit log ObjectType::ContinualTask for backward compatibility with persisted entries

Details

Catalog migration (v81→v82):

Iterates over all persisted items and deletes any whose create_sql starts with CREATE CONTINUAL TASK
Deletes any comments on continual task objects
Builtin CT removal is handled automatically by the builtin item migration (they're no longer in BUILTINS_STATIC)

What was removed (~4,600 lines deleted across 99 files):

SQL parser: CREATE/DROP/SHOW/ALTER/COMMENT ON CONTINUAL TASK syntax, ContinualTaskStmt, CreateContinualTaskSugar (TRANSFORM/RETAIN)
SQL planning: CreateContinualTaskPlan, feature flags (enable_continual_task_create/transform/retain), max_continual_tasks session var, CatalogItemType::ContinualTask, ObjectType::ContinualTask, CommentObjectId::ContinualTask
Adapter: sequence_create_continual_task, CT bootstrap/dataflow logic, CT quota checks, ExecuteResponse::CreatedContinualTask
Compute: ContinualTaskCtx, ContinualTaskConnection, CT sink rendering, render/continual_task.rs
Catalog builtins: BuiltinContinualTask, 3 system CTs (mz_cluster_replica_metrics_history_ct, mz_cluster_replica_status_history_ct, mz_wallclock_lag_history_ct), mz_continual_tasks table/view/indexes
BuiltinsConfig emptied (was only used for include_continual_tasks); BUILTINS::iter() no longer takes a config parameter
All CT-specific test files (9 SLT files, 1 testdrive file, 1 parser test file)

What was preserved:

mz_audit_log::ObjectType::ContinualTask — audit log entries are immutable
audit_log_event_v1::ObjectType::ContinualTask in catalog protos — same reason
Historical catalog proto snapshots (v74–v81) — frozen for migration support

https://claude.ai/code/session_01Rx3uSReMsvp3zDTxnXuNUd

github-actions · 2026-04-11T15:08:26Z

Thanks for opening this PR! Here are a few tips to help make the review process smooth for everyone.

PR title guidelines

Use imperative mood: "Fix X" not "Fixed X" or "Fixes X"
Be specific: "Fix panic in catalog sync when controller restarts" not "Fix bug" or "Update catalog code"
Prefix with area if helpful: compute: , storage: , adapter: , sql:

Pre-merge checklist

The PR title is descriptive and will make sense in the git log.
This PR has adequate test coverage / QA involvement has been duly considered. (trigger-ci for additional test/nightly runs)
If this PR includes major user-facing behavior changes, I have pinged the relevant PM to schedule a changelog post.
This PR has an associated up-to-date design doc, is a design doc (template), or is sufficiently small to not require a design.
If this PR evolves an existing $T ⇔ Proto$T mapping (possibly in a backwards-incompatible way), then it is tagged with a T-proto label.
If this PR will require changes to cloud orchestration or tests, there is a companion cloud PR to account for those changes that is tagged with the release-blocker label (example).

def- · 2026-04-15T15:02:36Z

Probably fixes https://github.com/MaterializeInc/database-issues/issues/11284 & https://github.com/MaterializeInc/database-issues/issues/11279

def- · 2026-04-15T15:04:07Z

I'd like to see a platform check that creates a CT if the version is old enough to allow it and then checks that we handle the migration properly, could be done with the existing tests in misc/python/materialize/checks/all_checks/continual_task.py

teskje

Got halfway through, but need to get lunch now!

teskje

Looks good, except for the merge skew!

def-

How do we know that no one is using continual tasks in self-managed btw? I'm wondering if the panic is the best approach here.

antiguru · 2026-04-23T19:26:29Z

This should be good for another look.

ggevay · 2026-04-24T14:01:58Z

How do we know that no one is using continual tasks in self-managed btw?

We provide no guarantees for such highly experimental features, so I'd say it's fine. But there might be a higher chance of some staging env or prod canary having CTs.

Anyhow, if an env has CTs, this would come out during the Friday upgrade-check (or when bringing up the read-only env during a 0dt upgrade), at which point one can still manually drop the CT in the old env, so we wouldn't get into an unrecoverable situation, right?

ggevay

Thank you! This will alleviate some complications in the catalog implications work!

One more minor thing: Can we also remove force_source_non_monotonic?

DAlperin · 2026-04-27T22:23:31Z

 Connection
 Connections
 Constraint
-Continual


can we not nuke Task here too? Did a quick grep through parser.rs and not sure we use it anywhere else.

Remove the continual tasks feature from Materialize. CTs were an experimental feature that is no longer needed. Changes: * Remove CT SQL syntax from parser * Remove CT catalog items, builtins, and system tables * Remove CT rendering in compute * Add catalog migration v81→v82 that panics if CTs are found * Remove BuiltinsConfig (empty after CT removal) * Add platform check for CT migration verification * Clean up related comments and dead code Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Removes the materialized-view optimizer's force_source_non_monotonic field, which was only used by continual tasks. Also removes the Continual, Task, and Tasks SQL keywords (no longer referenced) and a stale TODO(ct) comment in CatalogCollectionEntry. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

antiguru force-pushed the claude/remove-continual-tasks-akICD branch from e9990a5 to 9187f22 Compare April 15, 2026 14:50

antiguru marked this pull request as ready for review April 15, 2026 15:00

antiguru requested review from a team as code owners April 15, 2026 15:00

antiguru requested review from SangJunBak, aljoscha and teskje and removed request for a team and aljoscha April 15, 2026 15:00

antiguru force-pushed the claude/remove-continual-tasks-akICD branch from 4a2664a to b36e831 Compare April 15, 2026 16:22

teskje reviewed Apr 15, 2026

View reviewed changes

Comment thread misc/wasm/Cargo.lock Outdated

Comment thread src/adapter/src/catalog/open.rs Outdated

Comment thread src/catalog/src/durable/upgrade/v81_to_v82.rs Outdated

Comment thread src/catalog/src/builtin.rs Outdated

Comment thread src/compute-client/src/controller.rs Outdated

antiguru force-pushed the claude/remove-continual-tasks-akICD branch 2 times, most recently from 3a75b2c to 8e95b61 Compare April 16, 2026 07:46

antiguru requested a review from a team as a code owner April 16, 2026 07:46

antiguru force-pushed the claude/remove-continual-tasks-akICD branch from 8e95b61 to e80bd1d Compare April 16, 2026 07:48

teskje reviewed Apr 16, 2026

View reviewed changes

Comment thread doc/user/content/releases/v26.20.md

Comment thread src/arrow-util/src/reader.rs

teskje reviewed Apr 16, 2026

View reviewed changes

Comment thread src/persist/src/azure.rs

Comment thread src/sql-parser/src/parser.rs

Comment thread src/testdrive/src/action/s3.rs

Comment thread src/testdrive/src/action.rs

Comment thread LICENSE

def- reviewed Apr 18, 2026

View reviewed changes

Comment thread misc/python/materialize/checks/all_checks/continual_task.py

def- reviewed Apr 18, 2026

View reviewed changes

Comment thread misc/python/materialize/checks/all_checks/continual_task.py

antiguru force-pushed the claude/remove-continual-tasks-akICD branch from e80bd1d to b6d075c Compare April 23, 2026 19:21

antiguru requested a review from a team as a code owner April 23, 2026 19:21

antiguru force-pushed the claude/remove-continual-tasks-akICD branch from b6d075c to 22e7cc4 Compare April 23, 2026 19:54

antiguru force-pushed the claude/remove-continual-tasks-akICD branch from 22e7cc4 to ea9254a Compare April 24, 2026 07:35

ggevay approved these changes Apr 24, 2026

View reviewed changes

DAlperin approved these changes Apr 27, 2026

View reviewed changes

antiguru force-pushed the claude/remove-continual-tasks-akICD branch 2 times, most recently from 321c1b2 to b49dd7c Compare April 28, 2026 08:43

antiguru force-pushed the claude/remove-continual-tasks-akICD branch from b49dd7c to f8f440f Compare April 28, 2026 09:07

antiguru enabled auto-merge (squash) April 28, 2026 09:08

antiguru merged commit 9d0a7c3 into main Apr 28, 2026
129 checks passed

antiguru deleted the claude/remove-continual-tasks-akICD branch April 28, 2026 09:30

antiguru mentioned this pull request Jun 4, 2026

design: recorders — recorded, non-deterministic, at-least-once transformations #36909

Draft

Conversation

antiguru commented Apr 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Details

Uh oh!

github-actions Bot commented Apr 11, 2026

PR title guidelines

Pre-merge checklist

Uh oh!

def- commented Apr 15, 2026

Uh oh!

def- commented Apr 15, 2026

Uh oh!

teskje left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

teskje left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

def- left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

antiguru commented Apr 23, 2026

Uh oh!

ggevay commented Apr 24, 2026

Uh oh!

ggevay left a comment

Choose a reason for hiding this comment

Uh oh!

DAlperin Apr 27, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

antiguru commented Apr 11, 2026 •

edited

Loading