Skip to content

PLUGIN-1438: Added support for destination table write preference in BQ Execute Plugin#1473

Merged
itsankit-google merged 1 commit into
developfrom
bigQuery-Execute-Plugin
Dec 12, 2024
Merged

PLUGIN-1438: Added support for destination table write preference in BQ Execute Plugin#1473
itsankit-google merged 1 commit into
developfrom
bigQuery-Execute-Plugin

Conversation

@anshumanks

@anshumanks anshumanks commented Dec 11, 2024

Copy link
Copy Markdown
Contributor

Added support for destination table write preference in BQ Execute Plugin

Jira : PLUGIN-1438

Description

This update introduces the ability for users to select a destination table write preference when storing results in BigQuery. Users can now choose from the following options:

  • Write If Empty: Write to the table only if it is empty. If the table contains data, the operation will fail.
  • Append To Table: Append results to an existing table, preserving its schema.
  • Overwrite Table: Overwrite the existing table, replacing both the data and schema.

UI Field

  • Modified BigQueryExecute-action.json

Docs

  • Modified BigQueryExecute-action.md

Code change

  • Modified BigQueryExecute.java

Unit Tests

  • Modified BigQueryExecuteTest.java

Tested

image

@anshumanks anshumanks changed the title Added support for destination table write preference in BQ Execute Plugin PLUGIN-1438: Added support for destination table write preference in BQ Execute Plugin Dec 11, 2024

@itsankit-google itsankit-google left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please add evidence/screenshots in PR description under Tested section.

A good PR description example: #1444 (comment)

Comment on lines +406 to +409
@Description("Specifies the write preference for the destination table. Options are: " +
"Write if Empty: Only write if the table is empty. " +
"Append to Table: Add results to existing data. " +
"Overwrite Table: Replace all existing data.")

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Description/Tooltip should be crisp: Specifies whether a job should overwrite or append the existing destination table if it already exists.

Please add these details in markdown: https://github.com/data-integrations/google-cloud/blob/develop/docs/BigQueryExecute-action.md

@itsankit-google itsankit-google added the build Trigger unit test build label Dec 11, 2024
Comment thread src/main/java/io/cdap/plugin/gcp/bigquery/action/BigQueryExecute.java Outdated
Comment thread src/main/java/io/cdap/plugin/gcp/bigquery/action/BigQueryExecute.java Outdated
Comment thread src/test/java/io/cdap/plugin/gcp/bigquery/action/BigQueryExecuteTest.java Outdated

@itsankit-google itsankit-google left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, please squash commits before merge.

@anshumanks anshumanks force-pushed the bigQuery-Execute-Plugin branch 2 times, most recently from 40b5f74 to d7b5253 Compare December 12, 2024 12:25
…ugin

Addressed PR comments: updated description, added documentation for the added feature

Formatted to fix failing tests

Refactor: Replace hardcoded write preference strings with JobInfo.WriteDisposition constants

Refactor: Simplify error message for invalid write preference validation

Refactor: Centralize valid write preferences in BigQueryExecute.Config

Refactor: Dynamically generate valid write preferences from JobInfo.WriteDisposition
@anshumanks anshumanks force-pushed the bigQuery-Execute-Plugin branch from d7b5253 to a467a7a Compare December 12, 2024 12:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

build Trigger unit test build

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants