dbt State model concurrency - #9795
Open
luna-bianca wants to merge 6 commits into
Open
Conversation
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
joellabes
requested changes
Aug 17, 2026
| - **`build_after` vs `lag_tolerance`**: Both configs reduce how often a model runs when upstream data is frequently fresh, but they work differently: | ||
| - `freshness.build_after` (for example, `{count: 4, period: hour}`) skips the model unless the configured interval has elapsed _and_ upstream sources have new data since the last run. A SQL change alone does not trigger a rebuild; both conditions must be met. | ||
| - `state.lag_tolerance` (for example, `4h`) skips the model unless upstream data is newer than the model's last run by at least the configured interval. Unlike `build_after`, a detected SQL change triggers a rebuild. | ||
| - **No concurrent build detection**: In state-aware orchestration, if a model was being built by one job, any other job that included the same model would skip it. dbt State doesn't have this mechanism; if two jobs include the same model and run at the same time (for example, an hourly and a daily job that overlap), both jobs build the model independently. This doesn't cause data corruption because the warehouse handles concurrent writes, but it does result in duplicate compute. |
Contributor
There was a problem hiding this comment.
This doesn't cause data corruption because the warehouse handles concurrent writes, but it does result in duplicate compute.
This can cause data corruption in snapshots and incremental models if they execute simultaneously on two jobs.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What are you changing in this pull request and why?
Closes https://dbtlabs.atlassian.net/browse/PRODDOCS-1256
Documents a known gap in dbt State: unlike state-aware orchestration, dbt State doesn't detect when the same model is already being built by another concurrent job. Both jobs run independently, resulting in duplicate compute (though no data corruption). Adds a new "No concurrent build detection" entry to the Known differences section of the migration guide, including the workaround of configuring job schedules or avoiding overlapping manually triggered runs.
Source: https://dbt-labs.slack.com/archives/C0B0L9R47HB/p1786483730719469?thread_ts=1781736611.990089&cid=C0B0L9R47HB
Checklist
🚀 Deployment available! Here are the direct links to the updated files: