New feature: Add Salesforce Data360 database and pipeline connectors

[rahulcodewiz]

Describe your changes:

Adds a new Salesforce Data360 (formerly known as Data cloud) data warehouse connector and a Data360 pipeline connector, since Data360 is not currently supported as a metadata source in OpenMetadata.

This adds:

• ingestion/src/metadata/ingestion/source/database/data360/ — database source (client, connection, metadata, utils)
• ingestion/src/metadata/ingestion/source/pipeline/data360pipeline/ — pipeline source (connection, lineage, metadata, models, operational, exceptions)
• New JSON schemas for both connections (data360Connection.json, data360PipelineConnection.json) and registration in databaseService.json / pipelineService.json
• UI wiring to register the new service types (icon, locale docs, constants, generated types)
• get_metadata_by_type only fetched a single page of DataCloud objects, silently dropping data past the first page. It now paginates fully, following the same limit/offset/totalSize convention as the other ssot/* endpoints in this client.

Type of change:

• Salesforce Data 360 Connector

High-level design:

• Follows the existing Source→Sink connector pattern used by other database/pipeline connectors in this repo (see docs/design-patterns.md).
• Database connector: client.py wraps the Data360 REST API (via simple_salesforce), connection.py builds the connection, metadata.py implements the ingestion source, utils.py/constant.py hold helpers/constants.
• Pipeline connector: adds pipeline metadata, lineage extraction (lineage.py), and operational status (operational.py). data360pipeline/exceptions.py holds the connector-local exceptions used by lineage/operational.
• Schema-first: new connection schemas added under openmetadata-spec/, then databaseService.json/pipelineService.json updated to reference them; generated TS/Java types picked up via the normal codegen step.
• No migration or backward-compatibility concerns — this only adds a new, opt-in service type.

Tests:

Unit tests

• ingestion/tests/unit/source/database/data360/test_client.py, test_utils.py
• ingestion/tests/unit/topology/database/test_data360.py
• ingestion/tests/unit/topology/pipeline/test_data360pipeline.py
• All passing: python -m pytest ingestion/tests/unit/topology/database/test_data360.py ingestion/tests/unit/source/database/data360/ ingestion/tests/unit/topology/pipeline/test_data360pipeline.py -q → 68 passed.

Checklist:

• I have read the CONTRIBUTING document
• I have added unit tests for the new connectors
• I have verified against a live Data360 instance

Filename	Overview
ingestion/src/metadata/ingestion/source/database/data360/client.py	Adds shared endpoint pagination and selective not-found handling, but missing pages are returned as authoritative partial discovery.
ingestion/src/metadata/ingestion/source/database/data360/metadata.py	Maps dataspaces, object categories, tables, columns, constraints, partitions, and tags into the database topology.
ingestion/src/metadata/ingestion/source/pipeline/data360pipeline/metadata.py	Creates pipeline entities for DataStreams, Calculated Insights, and DataTransforms with lifecycle metadata and tags.
ingestion/src/metadata/ingestion/source/pipeline/data360pipeline/lineage.py	Resolves mapped source entities and Data360 objects to emit table, column, and pipeline-associated lineage.
ingestion/src/metadata/ingestion/source/pipeline/data360pipeline/operational.py	Extracts pipeline run statuses, but absent optional timestamps currently cause parsing errors and dropped status records.
openmetadata-spec/src/main/resources/json/schema/entity/services/connections/database/data360Connection.json	Defines the Data360 database connection contract and pagination configuration.
openmetadata-spec/src/main/resources/json/schema/entity/services/connections/pipeline/data360PipelineConnection.json	Defines the Data360 pipeline connection contract, including optional database-service lineage resolution.
openmetadata-ui/src/main/resources/ui/src/constants/ServiceType.constant.ts	Registers Data360 database and pipeline service types in the UI service metadata.

_{Reviews (1): Last reviewed commit: "fix(ingestion): fix error masking, pagin..." | Re-trigger Greptile}

Greptile also left 2 inline comments on this PR.

Context used (3)

• Knowledge Base — Metadata ingestion
• Knowledge Base — Ingestion connectors
• Knowledge Base — Metadata schema contracts

Greptile Summary

The PR adds Salesforce Data360 database and pipeline connectors across ingestion, shared schemas, generated models, and UI registration.

• Adds paginated Data360 metadata discovery, table modeling, tags, constraints, and deletion-failure safeguards.
• Adds Data360 pipeline metadata, operational status, and lineage extraction.
• Registers the new service connection types and generated consumers across the server and UI.

Confidence Score: 4/5

The PR is not yet safe to merge because omitted optional Data360 run timestamps can still abort operational-status extraction.

DataTransform, Calculated Insight, and DataStream timestamp fields are optional, but the operational extractors still substitute 0 and pass it to ISO-8601 parsing, causing an exception and dropping the affected pipeline status.

Files Needing Attention: ingestion/src/metadata/ingestion/source/pipeline/data360pipeline/operational.py, ingestion/src/metadata/ingestion/source/pipeline/data360pipeline/metadata.py

Important Files Changed

Filename	Overview
ingestion/src/metadata/ingestion/source/database/data360/client.py	Adds endpoint-specific pagination and raises on missing pages so failed discovery is not returned as an authoritative partial listing.
ingestion/src/metadata/ingestion/source/database/data360/metadata.py	Implements Data360 database topology and now records failed schema discovery so deletion reconciliation skips affected schemas.
ingestion/src/metadata/ingestion/source/pipeline/data360pipeline/metadata.py	Implements Data360 pipeline entity extraction and shared timestamp parsing used by operational status ingestion.
ingestion/src/metadata/ingestion/source/pipeline/data360pipeline/operational.py	Extracts pipeline run statuses, but optional timestamps still reach the ISO parser as 0 and can drop status records.
ingestion/src/metadata/ingestion/source/pipeline/data360pipeline/lineage.py	Resolves Data360 source and target entities into table, column, and pipeline lineage.
openmetadata-spec/src/main/resources/json/schema/entity/services/connections/database/data360Connection.json	Defines the Data360 database connection and pagination contract.
openmetadata-spec/src/main/resources/json/schema/entity/services/connections/pipeline/data360PipelineConnection.json	Defines Data360 pipeline authentication, pagination, and optional database-service lineage configuration.
openmetadata-ui/src/main/resources/ui/src/constants/ServiceType.constant.ts	Registers both Data360 service types in the UI metadata catalog.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart LR
  D360[Salesforce Data360 APIs] --> DB[Data360 database source]
  D360 --> PL[Data360 pipeline source]
  DB --> Tables[Databases, schemas, tables, columns and tags]
  PL --> Pipelines[Pipelines, lineage and run status]
  Tables --> Sink[OpenMetadata sink]
  Pipelines --> Sink
  Schemas[Connection JSON schemas] --> DB
  Schemas --> PL
  Schemas --> UI[Server and UI service registration]

Loading

_{Reviews (11): Last reviewed commit: "fix(ingestion): resolve ruff checkstyle ..." | Re-trigger Greptile}

Context used (3)

• Knowledge Base — Metadata ingestion
• Knowledge Base — Ingestion connectors
• Knowledge Base — Metadata schema contracts

[github-actions]

❌ PR checklist incomplete

This PR cannot be merged until the following are addressed on its linked issue:

• No GitHub issue is linked. Link an issue in the Development section of the PR (or add Fixes #12345 to the description). For a same-org cross-repo issue, add Fixes open-metadata/<repo>#123 to the description.

The fields live on the linked issue in the Shipping project (open the issue → right sidebar → Projects). After you set them, re-run this check (or push a commit) — issue/project changes do not re-trigger it automatically.

Maintainers can bypass this check by adding the skip-pr-checks label.

[github-actions]

Hi there 👋 Thanks for your contribution!

The OpenMetadata team will review the PR shortly! Once it has been labeled as safe to test, the CI workflows
will start executing and we'll be able to make sure everything is working as expected.

Let us know if you need any help!

[github-actions]

Hi there 👋 Thanks for your contribution!

The OpenMetadata team will review the PR shortly! Once it has been labeled as safe to test, the CI workflows
will start executing and we'll be able to make sure everything is working as expected.

Let us know if you need any help!

[github-actions]

The Python checkstyle failed.

Please run make py_format and py_format_check in the root of your repository and commit the changes to this PR.
You can also use pre-commit to automate the Python code formatting.

You can install the pre-commit hooks with make install_test precommit_install.

[github-actions]

⚠️ TypeScript Types Need Update

The generated TypeScript types are out of sync with the JSON schema changes.

Since this is a pull request from a forked repository, the types cannot be automatically committed.
Please generate and commit the types manually:

cd openmetadata-ui/src/main/resources/ui
./json2ts-generate-all.sh -l true
git add src/generated/
git commit -m "Update generated TypeScript types"
git push

After pushing the changes, this check will pass automatically.

[github-actions]

Jest test Coverage

UI tests summary

Lines	Statements	Branches	Functions
	70.42% (95179/135141)	54.95% (56309/102455)	56.34% (18800/33364)

[github-actions]

Hi there 👋 Thanks for your contribution!

The OpenMetadata team will review the PR shortly! Once it has been labeled as safe to test, the CI workflows
will start executing and we'll be able to make sure everything is working as expected.

Let us know if you need any help!

[github-actions]

Hi there 👋 Thanks for your contribution!

The OpenMetadata team will review the PR shortly! Once it has been labeled as safe to test, the CI workflows
will start executing and we'll be able to make sure everything is working as expected.

Let us know if you need any help!

[rahulcodewiz]

@pmbrull thanks for the thorough pass on issues( pydantic-model paginator, the lastRunsLimit/regex/logging fixes, the BaseConnection refactor, and the setup.py dedup) what I had.

One regression I ran into: PaginatedPage.from_payload still raises unconditionally when totalSize is missing, even when items is present:

total_size = body.get(paging.total_size_field)
if total_size is None:
raise Data360ResponseError(f"Missing '{paging.total_size_field}' in {context}")

and it's locked in by test_run_paginator_raises_when_total_size_is_missing.

I hit this against a real Salesforce Data 360 org (10k+ tables): the live ssot/metadata endpoint omits totalSize entirely not 0 for a legitimate empty result, e.g. {"metadata": []} with no totalSize key at
all. That response should be treated as "zero items," not malformed, but this raises on it regardless of whether items is present and empty, which reproduces the exact RuntimeError/Data360ResponseError I
originally reported from production.

I've pushed a fix on top of your work in 3476abf (fix(ingestion): don't fail Data360 pagination when totalSize is omitted with items present): it only raises when both totalSize and the items field are absent from the body, and otherwise derives total_size from len(items). Updated/replaced the test accordingly and verified against 3 live ingestion runs against the real org with zero recurrence, all 121 tests passing across the Data360 unit + topology suites. Let me know if you'd rather approach it differently

[github-actions]

Hi there 👋 Thanks for your contribution!

The OpenMetadata team will review the PR shortly! Once it has been labeled as safe to test, the CI workflows
will start executing and we'll be able to make sure everything is working as expected.

Let us know if you need any help!

[gitar-bot]

Code Review ⚠️ Changes requested 4 resolved / 5 findings

Adds Salesforce Data360 database and pipeline connectors with full metadata discovery, lineage extraction, and operational status ingestion. The refactor to convert databaseSchemaLoaders to loadConnectionSchema(...) accidentally deleted the SapBw4Hana loader entry, leaving that service type without connection form fields—restore it alongside the new Data360 entries.

⚠️ Bug: Refactor drops SapBw4Hana connection schema loader

📄 openmetadata-ui/src/main/resources/ui/src/utils/DatabaseServicePureUtils.ts:81-84

In converting databaseSchemaLoaders from static import() calls to loadConnectionSchema(...), the [DatabaseServiceType.SapBw4Hana] entry was deleted rather than converted (the diff removes both the Data360 and SapBw4Hana import blocks but only re-adds Data360). SapBw4Hana is still a valid DatabaseServiceType and a supported service in ServiceUtilClassBase, and sapBw4HanaConnection.json still exists under the public schemas. With no loader, getDatabaseConfig(SapBw4Hana) falls through to the empty-schema branch, so the SapBw4Hana add/edit connection form renders with no connection fields. Restore the entry.

Re-add the SapBw4Hana schema loader that was dropped during the refactor.

[DatabaseServiceType.Data360]: () =>
  loadConnectionSchema('connections/database/data360Connection.json'),
[DatabaseServiceType.SapBw4Hana]: () =>
  loadConnectionSchema('connections/database/sapBw4HanaConnection.json'),
[DatabaseServiceType.SingleStore]: () =>
  loadConnectionSchema('connections/database/singleStoreConnection.json'),

✅ 4 resolved

✅ Bug: get_source_url produces malformed URLs via fragile 'my' split

📄 ingestion/src/metadata/ingestion/source/pipeline/data360pipeline/metadata.py:235-243
The if "my" in host: heuristic matches any domain containing the substring "my" (e.g. mycompany.my, mydomain), and host.split('my')[0] takes everything before the FIRST occurrence of "my". For the documented My Domain form mycompany.my, split('my')[0] returns an empty string, so the deep link becomes https://lightning.force.com/lightning/r/... with the instance host dropped. Result: broken/incorrect Salesforce sourceUrl deep links for essentially all My Domain hosts. Parse the host explicitly (e.g. strip a trailing .my suffix or use the configured domain directly) instead of substring splitting.

✅ Edge Case: Paginator can raise AttributeError on CI response missing 'collection'

📄 ingestion/src/metadata/ingestion/source/database/data360/client.py:94-98 📄 ingestion/src/metadata/ingestion/source/database/data360/client.py:114-116
For CALCULATED_INSIGHT, response = response.get(ResponseConstant.COLLECTION) (and the same on subsequent pages) can return None if the collection key is absent or null in a valid-but-unexpected API response. The very next line response.get(json_config.get(...)) then raises AttributeError on NoneType, aborting the whole ingestion instead of degrading gracefully. Guard the unwrap: if the collection is missing, log a warning and return the items collected so far.

✅ Bug: Empty PRIMARY_KEY constraint attached to tables without primary keys

📄 ingestion/src/metadata/ingestion/source/database/data360/metadata.py:299-302 📄 ingestion/src/metadata/ingestion/source/database/data360/utils.py:41-48
For every non-CI table, get_table_constraints(table.get(PRIMARY_KEYS, [])) is always called and unconditionally returns a TableConstraint(constraintType=PRIMARY_KEY, columns=[]) even when there are no primary keys (confirmed by test_get_table_constraints_with_no_primary_keys). This attaches a meaningless empty primary-key constraint to every DLO/DMO table. Return an empty list when there are no primary keys so no constraint is emitted.

✅ Quality: mark_tables_as_deleted duplicates base logic and can drift

📄 ingestion/src/metadata/ingestion/source/database/data360/metadata.py:436-450
The override copies the base DatabaseServiceSource.mark_tables_as_deleted verbatim (context check, markDeletedTables guard, _get_filtered_schema_names loop, delete_entity_from_source call) and only adds the failed_schema_fqns skip. If the base method's deletion semantics change later, this copy will silently drift out of sync. Consider factoring the per-schema skip into the base (e.g., an overridable _should_skip_schema_deletion(schema_fqn) hook returning False by default) so this class only overrides the small delta rather than re-implementing the whole method.

🤖 Prompt for agents

Code Review: Adds Salesforce Data360 database and pipeline connectors with full metadata discovery, lineage extraction, and operational status ingestion. The refactor to convert `databaseSchemaLoaders` to `loadConnectionSchema(...)` accidentally deleted the `SapBw4Hana` loader entry, leaving that service type without connection form fields—restore it alongside the new Data360 entries.

1. ⚠️ Bug: Refactor drops SapBw4Hana connection schema loader
   Files: openmetadata-ui/src/main/resources/ui/src/utils/DatabaseServicePureUtils.ts:81-84

   In converting `databaseSchemaLoaders` from static `import()` calls to `loadConnectionSchema(...)`, the `[DatabaseServiceType.SapBw4Hana]` entry was deleted rather than converted (the diff removes both the Data360 and SapBw4Hana import blocks but only re-adds Data360). SapBw4Hana is still a valid `DatabaseServiceType` and a supported service in `ServiceUtilClassBase`, and `sapBw4HanaConnection.json` still exists under the public schemas. With no loader, `getDatabaseConfig(SapBw4Hana)` falls through to the empty-schema branch, so the SapBw4Hana add/edit connection form renders with no connection fields. Restore the entry.

   Fix (Re-add the SapBw4Hana schema loader that was dropped during the refactor.):
   [DatabaseServiceType.Data360]: () =>
     loadConnectionSchema('connections/database/data360Connection.json'),
   [DatabaseServiceType.SapBw4Hana]: () =>
     loadConnectionSchema('connections/database/sapBw4HanaConnection.json'),
   [DatabaseServiceType.SingleStore]: () =>
     loadConnectionSchema('connections/database/singleStoreConnection.json'),

Options

Display: compact → Showing less information.

Comment with these commands to change the behavior for this request:

Compact

gitar display:verbose

_{Was this helpful? React with 👍 / 👎 | Powered by Gitar — free for open source}

[sonarqubecloud]

Quality Gate passed for 'open-metadata-ui'

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

[sonarqubecloud]

Quality Gate passed for 'open-metadata-ingestion'

Issues
9 New issues
0 Accepted issues

Measures
0 Security Hotspots
69.6% Coverage on New Code
0.9% Duplication on New Code

See analysis details on SonarQube Cloud

[rahulcodewiz]

Regarding the SapBw4Hana loader entry Bot's comment- checked the diff and DatabaseServicePureUtils.ts only adds the Data360 entry here (2 lines), nothing is removed from there.

The only other SapBw4Hana-related diff in this PR is in the generated workflow.json, where sapBw4HanaType's oneOf index shifts from 56 -> 57 I think that's the expected so no fix needed here

[pmbrull]

@rahulcodewiz yes bot hallucinated, i think there was some past code I cleaned and stayed stuck