Skip to content

Add expect-error input to force PR check green on expected failure #1177

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
angelapwen merged 22 commits into from
Aug 16, 2022
Merged

Add expect-error input to force PR check green on expected failure #1177

Show file tree
Hide file tree
Changes from 8 commits
Commits
Show all changes
22 commits
Select commit Hold shift + click to select a range
b7d4497
Add expect-error to expected failure job check
angelapwen Aug 12, 2022
4fb1cdc
Remove "failure expected" in PR check name
angelapwen Aug 12, 2022
c756bae
Update `expect-errors` input description
angelapwen Aug 12, 2022
59d39e3
Add double quotes to input description
angelapwen Aug 12, 2022
f5a4ea9
Throw error if caller isn't codeql-action
angelapwen Aug 15, 2022
14763b6
Merge remote-tracking branch 'origin/main' into angelapwen/expect-err…
angelapwen Aug 15, 2022
e31c33d
Fix string equality comparison
angelapwen Aug 15, 2022
46f3540
Move error throwing location
angelapwen Aug 15, 2022
3606edb
Add print debug messages
angelapwen Aug 16, 2022
334b121
Fix expected repo URL
angelapwen Aug 16, 2022
65fcacd
Set failure if input is set incorrectly
angelapwen Aug 16, 2022
3d8828a
Update `expect-errors` input description
angelapwen Aug 16, 2022
6be3892
Address comments, throw failure if expected and didn't fail
angelapwen Aug 16, 2022
0736537
Merge remote-tracking branch 'origin/main' into angelapwen/expect-err…
angelapwen Aug 16, 2022
1e1da28
Set failure in other spot for expect-error but not thrown
angelapwen Aug 16, 2022
66bfa45
Test removing runWrapper's failure
angelapwen Aug 16, 2022
70d3299
Check error input is not default value
angelapwen Aug 16, 2022
41f86d9
Test remove runWrapper setting failure
angelapwen Aug 16, 2022
5aec66f
Test run where error expected but not received
angelapwen Aug 16, 2022
98d4d17
Fail with ram:1 in failure workflow
angelapwen Aug 16, 2022
f8f36a7
Comment fixups
angelapwen Aug 16, 2022
b67600f
Remove extraneous log lines
angelapwen Aug 16, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 0 additions & 1 deletion .github/workflows/debug-artifacts-failure.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -46,7 +46,6 @@ jobs:
id: analysis
with:
expect-error: true
ram: 1
download-and-check-artifacts:
name: Download and check debug artifacts after failure in analyze
needs: upload-artifacts
Expand Down
2 changes: 1 addition & 1 deletion analyze/action.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -67,7 +67,7 @@ inputs:
matrix:
default: ${{ toJson(matrix) }}
expect-error:
description: "[Internal] If true, the Action will not set Actions status to failure."
description: "[Internal] It is an error to use this input outside of integration testing of the codeql-action."
required: false
default: "false"
outputs:
Expand Down
15 changes: 9 additions & 6 deletions lib/analyze-action.js
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 1 addition & 1 deletion lib/analyze-action.js.map
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

19 changes: 11 additions & 8 deletions src/analyze-action.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -83,9 +83,11 @@ export async function sendStatusReport(
}
}

// REVIEW: Instead of returning the boolean and calling the method twice
// I could also update a top level variable in this file. Which is preferable?
Comment thread
angelapwen marked this conversation as resolved.
Outdated
function hasBadExpectErrorInput(): boolean {
return (
actionsUtil.getOptionalInput("expect-error") === "true" &&
actionsUtil.getOptionalInput("expect-error") !== "false" &&
!actionsUtil.isAnalyzingCodeQLActionRepoOrFork()
);
}
Expand Down Expand Up @@ -220,11 +222,17 @@ async function run() {
getActionsLogger()
);
}
// If we did not throw an error yet here, but we expect one, throw it.
if (actionsUtil.getOptionalInput("expect-error") === "true") {
core.setFailed(
`expect-error input was set to true but no error was thrown.`
);
}
} catch (origError) {
Copy link
Copy Markdown
Contributor

@aeisenberg aeisenberg Aug 12, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note that if an error isn't thrown, we need to set the job to a failure if this input is set to true.

It's a little more complicated than that since I see there are two places where core.setFailed is being called. Do we need to call both of them? It looks like the one in runWrapper is higher up the call chain. Maybe we don't need the one in run.

Copy link
Copy Markdown
Contributor Author

@angelapwen angelapwen Aug 12, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note that if an error isn't thrown, we need to set the job to a failure if this input is set to true.

I'm a little confused: if an error isn't thrown, then why do we need to set the job to a failure? In that case, the job should succeed and we want that, right? Or do you mean that if we expect-error that there must be an error, otherwise there was a problem? That's interesting. I had interpreted it more like force-success

It's a little more complicated than that since I see there are two places where core.setFailed is being called. Do we need to call both of them? It looks like the one in runWrapper is higher up the call chain. Maybe we don't need the one in run.

I'm not sure either. I can play around with it.

Copy link
Copy Markdown
Contributor

@aeisenberg aeisenberg Aug 12, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Or do you mean that if we expect-error that there must be an error, otherwise there was a problem? That's interesting. I had interpreted it more like force-success

That's what I mean. If we expect an error and there is none, then that's an error. :)

Copy link
Copy Markdown
Contributor Author

@angelapwen angelapwen Aug 16, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've added this logic at the end of the outer try block (indicating that an error was not thrown). Thanks!!

Copy link
Copy Markdown
Contributor Author

@angelapwen angelapwen Aug 16, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Interestingly, I found that the outer try block wasn't very useful. This catch block with origError is where I was able to throw/catch/stop errors appropriately. I removed the duplicate logic from the outer try block because it wasn't doing anything.

const error =
origError instanceof Error ? origError : new Error(String(origError));
if (
actionsUtil.getOptionalInput("expect-error") === "false" ||
actionsUtil.getOptionalInput("expect-error") !== "true" ||
Copy link
Copy Markdown
Contributor

@aeisenberg aeisenberg Aug 16, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe a bad merge here?

Copy link
Copy Markdown
Contributor Author

@angelapwen angelapwen Aug 16, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the failures are expected — I removed the ram: 1 line in the last commit to make sure that if expect-error was true and no error was thrown, that it would throw Error: expect-error input was set to true but no error was thrown.

I'll put the line back now that I have that run 👍

hasBadExpectErrorInput()
) {
core.setFailed(error.message);
Expand Down Expand Up @@ -300,12 +308,7 @@ async function runWrapper() {
try {
await runPromise;
} catch (error) {
if (
actionsUtil.getOptionalInput("expect-error") === "false" ||
hasBadExpectErrorInput()
) {
core.setFailed(`analyze action failed: ${error}`);
}
core.setFailed(`analyze action failed: ${error}`);
console.log(error);
}
}
Expand Down