Skip to content

Fixed printing the tasks in job output in DAG execution order - #377

Merged
andrewnester merged 2 commits into
mainfrom
output-dag-order
May 8, 2023
Merged

Fixed printing the tasks in job output in DAG execution order#377
andrewnester merged 2 commits into
mainfrom
output-dag-order

Conversation

@andrewnester

@andrewnester andrewnester commented May 8, 2023

Copy link
Copy Markdown
Contributor

Fixes #259

Changes

Sort task output in an execution order based on task end time

Tests

Added TestTaskJobOutputOrderToString unit test.

Bundle configuration used for manual testing

bundle:
  name: multiple-tasks

workspace:
  host: ***

environments:
  development: {
    default: true
  }

  production: {}

resources:
  jobs:
    my_job_with_file_task:
      name: "[${bundle.environment}] Job with multiple tasks"

      tasks:
        - task_key: Task1
          existing_cluster_id: ***
          spark_python_task:
            python_file: ./test.py

        - task_key: Task2
          depends_on:
            - task_key: Task1
          existing_cluster_id: ***
          spark_python_task:
            python_file: ./test.py

        - task_key: Task3
          depends_on:
            - task_key: Task2
          existing_cluster_id: ***
          spark_python_task:
            python_file: ./test.py

        - task_key: Task4
          depends_on:
            - task_key: Task1
          existing_cluster_id: ***
          spark_python_task:
            python_file: ./test.py

Output

> bricks bundle run my_job_with_file_task

Run URL: ***

2023-05-08 16:08:00 "[development] Job with multiple tasks" RUNNING 
2023-05-08 16:08:23 "[development] Job with multiple tasks" TERMINATED SUCCESS 
Output:
=======
Task Task1:
Hello

=======
Task Task4:
Hello

=======
Task Task2:
Hello

=======
Task Task3:
Hello

@andrewnester
andrewnester requested review from a team and pietern May 8, 2023 13:58

@pietern pietern left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks! 🚀

LGTM bar the comment.

Comment thread bundle/run/output/job.go Outdated
)

type TaskOutput struct {
Name string

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is the task key so can be named TaskKey instead to mirror the API objects (or simply Key).

@andrewnester
andrewnester requested a review from pietern May 8, 2023 14:12
@andrewnester
andrewnester merged commit 1916bc9 into main May 8, 2023
@andrewnester
andrewnester deleted the output-dag-order branch May 8, 2023 14:35
@shreyas-goenka shreyas-goenka mentioned this pull request May 15, 2023
shreyas-goenka added a commit that referenced this pull request May 15, 2023
## Changes
## 0.0.32
* Add support for variables in bundle config. Introduces 4 ways of
setting variable values, which in decreasing order of priority are:
([#383](https://github.com/databricks/bricks/pull/383))([#359](https://github.com/databricks/bricks/pull/359)).
	1. Command line flag. For example: `--var="foo=bar"`
	2. Environment variable. eg: BUNDLE_VAR_foo=bar
	3. Default value as defined in the applicable environments block
	4. Default value defined in variable definition
* Make the git details bundle config block optional
([#372](#372)).
* Fix api post integration tests
([#371](#371)).
* Fix table of content by removing not required top-level item
([#366](#366)).
* Fix printing the tasks in job output in DAG execution order
([#377](#377)).
* Improved error message when 'bricks bundle run' is executed before
'bricks bundle deploy'
([#378](#378)).
denik pushed a commit that referenced this pull request May 20, 2026
Fixes #259

## Changes
Sort task output in an execution order based on task end time

## Tests
Added `TestTaskJobOutputOrderToString` unit test.
denik pushed a commit that referenced this pull request May 20, 2026
## Changes
## 0.0.32
* Add support for variables in bundle config. Introduces 4 ways of
setting variable values, which in decreasing order of priority are:
([#383](https://github.com/databricks/bricks/pull/383))([#359](https://github.com/databricks/bricks/pull/359)).
	1. Command line flag. For example: `--var="foo=bar"`
	2. Environment variable. eg: BUNDLE_VAR_foo=bar
	3. Default value as defined in the applicable environments block
	4. Default value defined in variable definition
* Make the git details bundle config block optional
([#372](#372)).
* Fix api post integration tests
([#371](#371)).
* Fix table of content by removing not required top-level item
([#366](#366)).
* Fix printing the tasks in job output in DAG execution order
([#377](#377)).
* Improved error message when 'bricks bundle run' is executed before
'bricks bundle deploy'
([#378](#378)).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Print the tasks in job output in DAG execution order

2 participants