Skip to content

Commit f05a2a2

Browse files
pieternTanishqDatabricks
pietern
authored and
TanishqDatabricks
committed
Add postgres_synced_tables bundle resource (databricks#5268)
## Changes New `postgres_synced_tables` resource that syncs a Unity Catalog Delta table into a Postgres table on a Lakebase Autoscaling branch. Supported on both direct and terraform deployment engines. ## Tests Acceptance coverage: `basic` and `recreate` exercise each engine, plus the existing `no_drift` and `migrate` invariants pick up the new resource. Both engines produce identical human-readable output and identical wire bodies. Verified end to end on a live workspace: the bundle deploys a project, lakebase catalog, pipeline-storage schema, and synced table; the pipeline materializes in under a minute; `SELECT` against the destination through the UC federated view returns the rows from the source Delta table; and `bundle destroy` cleans up the full chain. This pull request and its description were written by Isaac.
1 parent c5d5981 commit f05a2a2

45 files changed

Lines changed: 1197 additions & 36 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎NEXT_CHANGELOG.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -18,6 +18,7 @@
1818
* Make sure warnings asking for approval are understood by agents ([#5239](https://github.com/databricks/cli/pull/5239))
1919
* Support `replace_existing: true` on `postgres_branches` and `postgres_endpoints` so bundles can manage the implicitly-created production branch and primary read-write endpoint of a Lakebase project.
2020
* Add `postgres_catalogs` resource to bind a Unity Catalog catalog to a Postgres database on a Lakebase Autoscaling branch ([#5265](https://github.com/databricks/cli/pull/5265)).
21+
* Add `postgres_synced_tables` resource to sync a Unity Catalog Delta table into a Postgres table on a Lakebase Autoscaling branch ([#5268](https://github.com/databricks/cli/pull/5268)).
2122
* engine/direct: Changes to state file now persisted to .wal file right away instead of being saved in the end ([#5149](https://github.com/databricks/cli/pull/5149))
2223

2324
### Dependency updates

‎acceptance/bundle/invariant/configs/postgres_synced_table.yml.tmpl‎

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,16 @@
1+
bundle:
2+
name: test-bundle-$UNIQUE_NAME
3+
4+
resources:
5+
postgres_synced_tables:
6+
foo:
7+
synced_table_id: lakebase_$UNIQUE_NAME.public.trips_synced
8+
source_table_full_name: main.raw.trips
9+
primary_key_columns: [id]
10+
scheduling_policy: SNAPSHOT
11+
postgres_database: appdb
12+
branch: projects/test-pg-project-$UNIQUE_NAME/branches/production
13+
create_database_objects_if_missing: true
14+
new_pipeline_spec:
15+
storage_catalog: main
16+
storage_schema: pipelines

‎acceptance/bundle/invariant/continue_293/out.test.toml‎

Lines changed: 1 addition & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎acceptance/bundle/invariant/migrate/out.test.toml‎

Lines changed: 1 addition & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎acceptance/bundle/invariant/no_drift/out.test.toml‎

Lines changed: 1 addition & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎acceptance/bundle/invariant/test.toml‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -47,6 +47,7 @@ EnvMatrix.INPUT_CONFIG = [
4747
"postgres_catalog.yml.tmpl",
4848
"postgres_endpoint.yml.tmpl",
4949
"postgres_project.yml.tmpl",
50+
"postgres_synced_table.yml.tmpl",
5051
"registered_model.yml.tmpl",
5152
"schema.yml.tmpl",
5253
"schema_grant_ref.yml.tmpl",
@@ -69,6 +70,7 @@ no_postgres_project_on_cloud = ["CONFIG_Cloud=true", "INPUT_CONFIG=postgres_proj
6970
no_postgres_branch_on_cloud = ["CONFIG_Cloud=true", "INPUT_CONFIG=postgres_branch.yml.tmpl"]
7071
no_postgres_endpoint_on_cloud = ["CONFIG_Cloud=true", "INPUT_CONFIG=postgres_endpoint.yml.tmpl"]
7172
no_postgres_catalog_on_cloud = ["CONFIG_Cloud=true", "INPUT_CONFIG=postgres_catalog.yml.tmpl"]
73+
no_postgres_synced_table_on_cloud = ["CONFIG_Cloud=true", "INPUT_CONFIG=postgres_synced_table.yml.tmpl"]
7274

7375
# External locations require actual storage credentials with cloud IAM setup
7476
# which are environment-specific, so we only test locally with the mock server

‎acceptance/bundle/refschema/out.fields.txt‎

Lines changed: 43 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2850,6 +2850,49 @@ resources.postgres_projects.*.permissions[*].group_name string ALL
28502850
resources.postgres_projects.*.permissions[*].level iam.PermissionLevel ALL
28512851
resources.postgres_projects.*.permissions[*].service_principal_name string ALL
28522852
resources.postgres_projects.*.permissions[*].user_name string ALL
2853+
resources.postgres_synced_tables.*.branch string ALL
2854+
resources.postgres_synced_tables.*.create_database_objects_if_missing bool ALL
2855+
resources.postgres_synced_tables.*.create_time *time.Time REMOTE
2856+
resources.postgres_synced_tables.*.existing_pipeline_id string ALL
2857+
resources.postgres_synced_tables.*.id string INPUT
2858+
resources.postgres_synced_tables.*.lifecycle resources.Lifecycle INPUT
2859+
resources.postgres_synced_tables.*.lifecycle.prevent_destroy bool INPUT
2860+
resources.postgres_synced_tables.*.modified_status string INPUT
2861+
resources.postgres_synced_tables.*.name string REMOTE
2862+
resources.postgres_synced_tables.*.new_pipeline_spec *postgres.NewPipelineSpec ALL
2863+
resources.postgres_synced_tables.*.new_pipeline_spec.budget_policy_id string ALL
2864+
resources.postgres_synced_tables.*.new_pipeline_spec.storage_catalog string ALL
2865+
resources.postgres_synced_tables.*.new_pipeline_spec.storage_schema string ALL
2866+
resources.postgres_synced_tables.*.postgres_database string ALL
2867+
resources.postgres_synced_tables.*.primary_key_columns []string ALL
2868+
resources.postgres_synced_tables.*.primary_key_columns[*] string ALL
2869+
resources.postgres_synced_tables.*.scheduling_policy postgres.SyncedTableSyncedTableSpecSyncedTableSchedulingPolicy ALL
2870+
resources.postgres_synced_tables.*.source_table_full_name string ALL
2871+
resources.postgres_synced_tables.*.status *postgres.SyncedTableSyncedTableStatus REMOTE
2872+
resources.postgres_synced_tables.*.status.detailed_state postgres.SyncedTableState REMOTE
2873+
resources.postgres_synced_tables.*.status.last_processed_commit_version int64 REMOTE
2874+
resources.postgres_synced_tables.*.status.last_sync *postgres.SyncedTablePosition REMOTE
2875+
resources.postgres_synced_tables.*.status.last_sync.delta_table_sync_info *postgres.DeltaTableSyncInfo REMOTE
2876+
resources.postgres_synced_tables.*.status.last_sync.delta_table_sync_info.delta_commit_time *time.Time REMOTE
2877+
resources.postgres_synced_tables.*.status.last_sync.delta_table_sync_info.delta_commit_version int64 REMOTE
2878+
resources.postgres_synced_tables.*.status.last_sync.sync_end_time *time.Time REMOTE
2879+
resources.postgres_synced_tables.*.status.last_sync.sync_start_time *time.Time REMOTE
2880+
resources.postgres_synced_tables.*.status.last_sync_time *time.Time REMOTE
2881+
resources.postgres_synced_tables.*.status.message string REMOTE
2882+
resources.postgres_synced_tables.*.status.ongoing_sync_progress *postgres.SyncedTablePipelineProgress REMOTE
2883+
resources.postgres_synced_tables.*.status.ongoing_sync_progress.estimated_completion_time_seconds float64 REMOTE
2884+
resources.postgres_synced_tables.*.status.ongoing_sync_progress.latest_version_currently_processing int64 REMOTE
2885+
resources.postgres_synced_tables.*.status.ongoing_sync_progress.sync_progress_completion float64 REMOTE
2886+
resources.postgres_synced_tables.*.status.ongoing_sync_progress.synced_row_count int64 REMOTE
2887+
resources.postgres_synced_tables.*.status.ongoing_sync_progress.total_row_count int64 REMOTE
2888+
resources.postgres_synced_tables.*.status.pipeline_id string REMOTE
2889+
resources.postgres_synced_tables.*.status.project string REMOTE
2890+
resources.postgres_synced_tables.*.status.provisioning_phase postgres.ProvisioningPhase REMOTE
2891+
resources.postgres_synced_tables.*.status.unity_catalog_provisioning_state postgres.ProvisioningInfoState REMOTE
2892+
resources.postgres_synced_tables.*.synced_table_id string ALL
2893+
resources.postgres_synced_tables.*.timeseries_key string ALL
2894+
resources.postgres_synced_tables.*.uid string REMOTE
2895+
resources.postgres_synced_tables.*.url string INPUT
28532896
resources.quality_monitors.*.assets_dir string ALL
28542897
resources.quality_monitors.*.baseline_table_name string ALL
28552898
resources.quality_monitors.*.custom_metrics []catalog.MonitorMetric ALL

‎acceptance/bundle/resources/postgres_synced_tables/basic/databricks.yml.tmpl‎

Lines changed: 38 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,38 @@
1+
bundle:
2+
name: deploy-postgres-synced-table-$UNIQUE_NAME
3+
4+
sync:
5+
paths: []
6+
7+
resources:
8+
schemas:
9+
pipeline_storage:
10+
name: pipeline_storage_$UNIQUE_NAME
11+
catalog_name: main
12+
comment: "Pipeline storage for the synced-table test"
13+
14+
postgres_projects:
15+
my_project:
16+
project_id: test-pg-proj-$UNIQUE_NAME
17+
display_name: "Test Project for Synced Table"
18+
pg_version: 17
19+
20+
postgres_catalogs:
21+
my_catalog:
22+
catalog_id: lakebase_test_$UNIQUE_NAME
23+
branch: ${resources.postgres_projects.my_project.id}/branches/production
24+
postgres_database: appdb
25+
create_database_if_missing: true
26+
27+
postgres_synced_tables:
28+
my_table:
29+
synced_table_id: ${resources.postgres_catalogs.my_catalog.catalog_id}.public.trips_synced
30+
source_table_full_name: main.source_$UNIQUE_NAME.trips_source
31+
primary_key_columns: ["tpep_pickup_datetime"]
32+
scheduling_policy: SNAPSHOT
33+
postgres_database: appdb
34+
branch: ${resources.postgres_projects.my_project.id}/branches/production
35+
create_database_objects_if_missing: true
36+
new_pipeline_spec:
37+
storage_catalog: ${resources.schemas.pipeline_storage.catalog_name}
38+
storage_schema: ${resources.schemas.pipeline_storage.name}

‎acceptance/bundle/resources/postgres_synced_tables/basic/out.requests.json‎

Lines changed: 54 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,54 @@
1+
{
2+
"method": "POST",
3+
"path": "/api/2.0/postgres/projects",
4+
"q": {
5+
"project_id": "test-pg-proj-[UNIQUE_NAME]"
6+
},
7+
"body": {
8+
"spec": {
9+
"display_name": "Test Project for Synced Table",
10+
"pg_version": 17
11+
}
12+
}
13+
}
14+
{
15+
"method": "POST",
16+
"path": "/api/2.0/postgres/catalogs",
17+
"q": {
18+
"catalog_id": "lakebase_test_[UNIQUE_NAME]"
19+
},
20+
"body": {
21+
"spec": {
22+
"branch": "projects/test-pg-proj-[UNIQUE_NAME]/branches/production",
23+
"create_database_if_missing": true,
24+
"postgres_database": "appdb"
25+
}
26+
}
27+
}
28+
{
29+
"method": "POST",
30+
"path": "/api/2.0/postgres/synced_tables",
31+
"q": {
32+
"synced_table_id": "lakebase_test_[UNIQUE_NAME].public.trips_synced"
33+
},
34+
"body": {
35+
"spec": {
36+
"branch": "projects/test-pg-proj-[UNIQUE_NAME]/branches/production",
37+
"create_database_objects_if_missing": true,
38+
"new_pipeline_spec": {
39+
"storage_catalog": "main",
40+
"storage_schema": "pipeline_storage_[UNIQUE_NAME]"
41+
},
42+
"postgres_database": "appdb",
43+
"primary_key_columns": [
44+
"tpep_pickup_datetime"
45+
],
46+
"scheduling_policy": "SNAPSHOT",
47+
"source_table_full_name": "main.source_[UNIQUE_NAME].trips_source"
48+
}
49+
}
50+
}
51+
{
52+
"method": "GET",
53+
"path": "/api/2.0/postgres/synced_tables/lakebase_test_[UNIQUE_NAME].public.trips_synced"
54+
}

‎acceptance/bundle/resources/postgres_synced_tables/basic/out.test.toml‎

Lines changed: 6 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

0 commit comments

Comments
 (0)