-
Notifications
You must be signed in to change notification settings - Fork 1.2k
Docs updates #9830
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Docs updates #9830
Changes from 2 commits
470c7d8
abac3d7
ae889be
67e4864
42548bd
ae5ae77
7c563c3
51ede7d
12ef901
7fd2cfc
db32fcd
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change | ||||
|---|---|---|---|---|---|---|
|
|
@@ -122,111 +122,104 @@ As of 1.8.1, the default host is `127.0.0.1`. For versions 1.8.0 and prior, the | |||||
|
|
||||||
| <VersionBlock firstVersion="2.0"> | ||||||
|
|
||||||
| The <Constant name="fusion_engine" /> uses the `--write-catalog` flag instead of the `dbt docs generate` command for generating your [`catalog.json`](/reference/artifacts/catalog-json) file and hydrating metadata. When you use `dbt build --write-catalog`, you're using a flag that performs better because it's built for the Fusion engine. To see the latest metadata in <Constant name="catalog" />, run a job in <Constant name="dbt_platform" /> which uploads the metadata. | ||||||
| With the <Constant name="fusion_engine" /> and <Constant name="core_v2" />, [dbt Docs v2](/docs/build/view-documentation#dbt-docs-v2) is the recommended way to generate and view your project's documentation. Use `dbt docs generate` to build the documentation site and `dbt docs serve` to preview it locally. | ||||||
|
|
||||||
| ## --write-catalog flag | ||||||
| If you only need to hydrate catalog metadata (`catalog.json`) for <Constant name="catalog" /> in <Constant name="dbt_platform" />, without building the documentation site, use the [`--write-catalog` flag](#--write-catalog-flag) instead. | ||||||
|
|
||||||
| The `--write-catalog` flag generates the [`catalog.json`](/reference/artifacts/catalog-json) artifact, which contains metadata about the tables and <Term id="view">views</Term> produced by the models in your project. <Constant name="fusion" /> jobs running in <Constant name="dbt_platform" />, dbt automatically runs `write-catalog`, `build`, and `run`, and hydrates your Catalog, so you don't need to manually include it. You can use this flag with the following commands: | ||||||
| ## dbt Docs v2 <Lifecycle status="beta"/> | ||||||
|
|
||||||
| - `dbt build` | ||||||
| - `dbt run` | ||||||
| - `dbt parse` | ||||||
| - `dbt compile` | ||||||
| Instead of loading a static `manifest.json` in the browser, v2 builds a compact binary index of your project. `dbt docs generate` exports a documentation site made of plain static files (a single-page app plus the index artifacts) that any file host can serve. The browser reads those artifacts directly using DuckDB-WASM, so you don't need to run a stateful server to view your docs. This keeps the experience fast even for large projects. | ||||||
|
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Can we link to something that explains DuckDB-WASM? Or at least spell out WASM for those of us who don't know it stands for WebAssembly.
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Again no company binary index. It just produces the parquet artifacts instead of manifest json |
||||||
|
|
||||||
| ### Generate the site | ||||||
|
|
||||||
| **Examples**: | ||||||
| `dbt docs generate` compiles your project, writes the index, and exports the documentation site in a single command. You no longer need to run a separate `--write-index` command first: | ||||||
|
|
||||||
| ```shell | ||||||
| dbt build --write-catalog | ||||||
| dbt docs generate | ||||||
| ``` | ||||||
|
|
||||||
| By default, dbt writes the site into your `target/` directory (`target/index.html`, `target/assets/`, and the index under `target/index/`), matching the layout of <Constant name="core_v1" />. You can serve `index.html` from `target/` the same way you did in v1, so existing CI such as `dbt docs generate && mv target public` continues to work. | ||||||
|
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Do you mean Docs v1?
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. v1 in this case is referring to Core v1 but they're sort of synonymous at this stage |
||||||
|
|
||||||
| ### Platform behavior | ||||||
|
|
||||||
| In <Constant name="dbt_platform" /> jobs running on <Constant name="fusion" />, you don't need to change anything. When `dbt docs generate` is called (either as a job step or separate command), the platform automatically uses `--write-catalog` instead. Additionally, for <Constant name="fusion" /> jobs running in the platform, dbt will run `write-catalog` automatically with `build` or `run`, so you don't need to run a separate command to hydrate your metadata. In the platform, you can optionally choose to include it when running `dbt parse` or `dbt compile`. | ||||||
| Use `--output-dir` to write a self-contained copy of the site to a different directory: | ||||||
|
|
||||||
| Note: | ||||||
|
|
||||||
| ### Local usage | ||||||
| ```shell | ||||||
| dbt docs generate --output-dir site | ||||||
|
runleonarun marked this conversation as resolved.
|
||||||
| ``` | ||||||
|
|
||||||
| When running <Constant name="fusion" /> locally, add the `--write-catalog` flag to your command to generate the catalog: | ||||||
| To skip compilation and export whatever index is already on disk, use `--no-compile`. This errors if no index exists: | ||||||
|
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
|
||||||
|
|
||||||
| ```shell | ||||||
| dbt build --write-catalog | ||||||
| dbt docs generate --no-compile | ||||||
| ``` | ||||||
|
|
||||||
| ### What's different from docs generate | ||||||
|
|
||||||
| The `--write-catalog` flag focuses solely on metadata hydration, generating the `catalog.json` file that powers [Catalog](/docs/explore/build-and-view-your-docs) and metadata APIs. It does not generate the static documentation website files (`index.html`). | ||||||
| #### Column lineage and richer metadata | ||||||
|
|
||||||
| ## dbt Docs v2 <Lifecycle status="alpha"/> | ||||||
| Column-level lineage and richer column metadata require an index built with [`--static-analysis strict`](/docs/build/about-static-analysis). Because `dbt docs generate` runs a standard compile by default, build the index with strict static analysis first when you want column lineage, then export it: | ||||||
|
|
||||||
| The <Constant name="fusion_engine" /> and <Constant name="core_v2" /> deliver a new version of `dbt docs serve` that powers [dbt Docs v2](/docs/build/view-documentation#dbt-docs-v2). | ||||||
| ```shell | ||||||
| dbt build --write-index --static-analysis strict | ||||||
| dbt docs generate --no-compile | ||||||
| ``` | ||||||
|
|
||||||
| Instead of loading a static `manifest.json` in the browser, v2 builds a compact binary index of your project and serves it through a local HTTP server with a REST API. This makes the experience fast even for large projects, and makes metadata queryable by AI agents and external tooling. | ||||||
| If the site was generated without column lineage, dbt Docs v2 hides those features instead of showing empty data. | ||||||
|
|
||||||
| ### Generate the index | ||||||
| ### Serve dbt Docs v2 | ||||||
|
|
||||||
| Before serving, build your project with the `--write-index` flag. You can add this flag to dbt `build`, `run`, `parse`, or `compile` commands. It writes index files to the `target/index/` directory which is what `dbt docs serve` reads from: | ||||||
| To preview the site locally, run: | ||||||
|
|
||||||
| ```shell | ||||||
| dbt compile --write-index | ||||||
| dbt docs serve | ||||||
| ``` | ||||||
|
|
||||||
| `dbt docs serve` generates the site if it's missing or older than the index, then serves the static files. The server starts on port `8580` by default and opens in your browser. Use `--port` to change the port: | ||||||
|
|
||||||
| ```shell | ||||||
| dbt build --write-index | ||||||
| dbt docs serve --port 8081 | ||||||
| ``` | ||||||
|
|
||||||
| Add [`--static-analysis strict`](/docs/build/about-static-analysis) to for column lineage and richer column metadata from your warehouse: | ||||||
| Use the `--target-path` flag to change the path where dbt reads artifacts from: | ||||||
|
|
||||||
| ```shell | ||||||
| dbt build --write-index --static-analysis strict | ||||||
| dbt docs serve --target-path ~/Developer/internal-analytics/target | ||||||
| ``` | ||||||
|
|
||||||
| ```shell | ||||||
| dbt build --write-index --static-analysis strict | ||||||
| ``` | ||||||
| Because the generated site is a set of static files, you can also host it on any static file host — such as cloud object storage or a static site host — instead of serving it locally. | ||||||
|
|
||||||
| ### Serve dbt Docs v2 | ||||||
| ### Project overview page | ||||||
|
|
||||||
| :::note Login for full capabilities | ||||||
| When using <Constant name="fusion" />, run `dbt login` before serving to unlock all capabilities. Some features, such as column lineage, require authentication to display. | ||||||
| ::: | ||||||
| dbt Docs v2 renders your project's `__overview__` doc block as the landing page, the same as dbt Docs v1. dbt discovers overview content by scanning your `docs-paths` for `{% docs %}` blocks, so a block in `models/overview.md` is found by default. A file at `docs/overview.md` is only picked up when your project sets `docs-paths: ["docs"]`. If your project defines no overview, dbt renders its default overview content. | ||||||
|
|
||||||
| Once the index is built, start the local documentation server: | ||||||
| ## --write-catalog flag | ||||||
|
|
||||||
| ```shell | ||||||
| dbt docs serve | ||||||
| ``` | ||||||
| The `--write-catalog` flag generates the [`catalog.json`](/reference/artifacts/catalog-json) artifact, which contains metadata about the tables and <Term id="view">views</Term> produced by the models in your project. It focuses solely on metadata hydration and does not build the documentation site — use [dbt Docs v2](#dbt-docs-v2) for that. | ||||||
|
|
||||||
| You can pass the `--target-path` flag to change the path where dbt pulls artifacts from: | ||||||
| For <Constant name="fusion" /> jobs running in <Constant name="dbt_platform" />, dbt automatically runs `write-catalog` with `build` and `run` and hydrates your Catalog, so you don't need to include it manually. You can use this flag with the following commands: | ||||||
|
|
||||||
| ```shell | ||||||
| dbt docs serve --target-path ~/Developer/internal-analytics/target | ||||||
| ``` | ||||||
| - `dbt build` | ||||||
| - `dbt run` | ||||||
| - `dbt parse` | ||||||
| - `dbt compile` | ||||||
|
|
||||||
| The server starts on port `8580` by default and opens in your browser. Use `--port` to change the port: | ||||||
| **Example**: | ||||||
|
|
||||||
| ```shell | ||||||
| dbt docs serve --port 8081 | ||||||
| dbt build --write-catalog | ||||||
| ``` | ||||||
|
|
||||||
| ### REST API | ||||||
| ### Platform behavior | ||||||
|
|
||||||
| In <Constant name="dbt_platform" /> jobs running on <Constant name="fusion" />, you don't need to change anything. When `dbt docs generate` is called (either as a job step or separate command), the platform automatically uses `--write-catalog` instead. Additionally, for <Constant name="fusion" /> jobs running in the platform, dbt runs `write-catalog` automatically with `build` or `run`, so you don't need to run a separate command to hydrate your metadata. In the platform, you can optionally choose to include it when running `dbt parse` or `dbt compile`. | ||||||
|
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Instead of generating the docs site? I'm confused because I thought
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. @eliasdefaria good question here. ARe the changes you've made explicitly for the local experience? Does anything change platform side?
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Platform side should just work if you run docs generate or turn it on in your jobs. Otherwise it does not produce the static site |
||||||
|
|
||||||
| ### Local usage | ||||||
|
|
||||||
| dbt Docs v2 exposes a REST API at `/api/v1/` that AI agents, MCP servers, and external tooling can query directly, all without a browser. Key endpoints include: | ||||||
| When running <Constant name="fusion" /> locally, add the `--write-catalog` flag to your command to generate the catalog: | ||||||
|
|
||||||
| | Endpoint | Description | | ||||||
| |---|---| | ||||||
| | `GET /api/v1/health` | Server status | | ||||||
| | `GET /api/v1/capabilities` | Feature flags (for example, `has_column_lineage`) | | ||||||
| | `GET /api/v1/models` | Paginated model list with filters | | ||||||
| | `GET /api/v1/models/:id` | Model detail including catalog metadata | | ||||||
| | `GET /api/v1/sources/:id` | Source detail | | ||||||
| | `GET /api/v1/nodes/counts` | Resource type counts (models, sources, tests, etc.) | | ||||||
| | `GET /api/v1/nodes/:id/lineage` | Model-level lineage graph | | ||||||
| | `GET /api/v1/nodes/:id/column-lineage` | Column-level lineage (Fusion-only capability) | | ||||||
| ```shell | ||||||
| dbt build --write-catalog | ||||||
| ``` | ||||||
|
|
||||||
| See the [dbt Docs v2 API contracts](https://github.com/dbt-labs/fs/blob/main/fs/sa/crates/dbt-docs-server/API-CONTRACTS.md#get-apiv1sources) for the full list of available endpoints. | ||||||
| ### What's different from docs generate | ||||||
|
|
||||||
| This makes dbt Docs v2 a natural context source for MCP servers. If you're using a coding agent like Claude Code, you can point it at a running dbt Docs v2 instance to give it rich, structured metadata about your dbt project without installing dbt locally. | ||||||
| The `--write-catalog` flag focuses solely on metadata hydration, generating the `catalog.json` file that powers [Catalog](/docs/explore/build-and-view-your-docs) and metadata APIs. It does not generate the static documentation website files (`index.html`). | ||||||
|
|
||||||
| </VersionBlock> | ||||||
Uh oh!
There was an error while loading. Please reload this page.