[SNOW-1462584] Implement setup script validation

[sfc-gh-fcampbell]

Pre-review checklist

• I've confirmed that instructions included in README.md are still correct after my changes in the codebase.
• I've added or updated automated unit tests to verify correctness of my new code.
• I've added or updated integration tests to verify correctness of my new code.
• I've confirmed that my changes are working by executing CLI's commands manually.
• I've confirmed that my changes are up-to-date with the target branch.
• I've described my changes in the release notes.
• I've described my changes in the section below.

Changes description

Calls system$validate_native_app_setup() to validate the setup script in the stage on snow app deploy, snow app run, and the new snow app validate. The validation is done server-side, all we need to do is raise an error and show messages if the validation failed.

For snow app validate, we use a scratch stage to avoid clobbering the app's dev stage in between deploys. This stage is deleted after every validation run and on snow app teardown to avoid leaving behind any garbage.

Scenarios

App has warning but no errors (exit code 0):

App has validation errors (exit code 1):

Deploy-time validation error (exit code 1, notice it's using the app's stage and not the scratch stage):

Skip validation at deploy time (exit code 0):

JSON output (exit code 0 regardless of status):

[sfc-gh-bgoel]

Since the App Pkg does not exist error is a specific error, we should possibly tighten the constraint on this condition with an and on the error number.

Once case I'm thinking of but is not necessarily true today: "does not exist or not authorized" could be thrown if some setup script is missing or permissions change that setup script cannot be accessed.

[sfc-gh-fcampbell]

if a permission issue hides the visibility of the app package, wouldn't that be functionally equivalent to it not existing, from the PoV of the user? I can update the exception's error message to add "or is not authorized" (i'll add the errno check too)

[sfc-gh-bdufour]

any reason we shouldn't print the errors as we do with the warnings, and fail with a "validation failed" generic message?

[sfc-gh-fcampbell]

there was no specific reason, but looking back I like it like that since it separates errors and warnings, and it makes the errors really obvious by printing them inside the ASCII box art (see the screenshot in the PR description). I'm not opposed to changing it if it's a blocker

[sfc-gh-bdufour]

I think we'll want to control how we format these messages, and this way of outputting them will prevent us from doing that. I'll do a pass on the overall output this week or next week, it's long overdue. We should sync on that, but I'm fine if validate becomes slightly inconsistent (but better) while the other commands catch up.

IMHO we keep the box small and generic here (something like "Validation failed, re-run with --no-validate to skip this check"). We output errors the same way as we do warnings, but maybe we make them visually stand out in the output (maybe we use shell colors to highlight errors?)

[sfc-gh-fcampbell]

I updated it to print the messages before the error:

[sfc-gh-bdufour]

Is validation a phase? It looks like it will output nested messages, hence the question.

[sfc-gh-fcampbell]

yeah it could be considered a phase, but the output looks a little wonky since the diff log uses "\n.join on a bunch of strings, so they don't come out indented:

[sfc-gh-bdufour]

The diff output is bad, I have a fix locally that I will submit in the next few days. So it's not a really good comparison.

[sfc-gh-fcampbell]

changed it to a phase

[sfc-gh-bdufour]

I think we'll want to control how we format these messages, and this way of outputting them will prevent us from doing that. I'll do a pass on the overall output this week or next week, it's long overdue. We should sync on that, but I'm fine if validate becomes slightly inconsistent (but better) while the other commands catch up.

IMHO we keep the box small and generic here (something like "Validation failed, re-run with --no-validate to skip this check"). We output errors the same way as we do warnings, but maybe we make them visually stand out in the output (maybe we use shell colors to highlight errors?)