[ci] introduce mode: experimental flag

[connorhu]

@thePanz This was the best solution I found. If mode: experimental is enabled, the exit status is printed but exits the shell with a zero.

This changes is because of php8.4 testing. We can now mark php version as experimental.

[connorhu]

If this works, we can roll out the solution to other repos and start testing it with php 8.4.

[thirsch]

Would continue-on-error work here as well? Maybe with something like

- name: Run Tests
  run: ...
  continue-on-error: ${{ matrix.mode == "experimental"}}

[connorhu]

It is possible that it works. Why didn't I find it in the documentation when I searched?

[mentalstring]

I believe the continue-on-error will only make sure the other PHP versions still run and not abort as soon as one fails. But if one fails, the CI will still be marked as failed.

[connorhu]

It's true! The status is also important here to keep the result green.

[mentalstring]

IMHO, I'm not sure this hack is worth the time and added complexity, given that a) php8.4 is at least 8 months away from .0 release (why test something that may still change?) and b) our tests are already passing with php8.4 anyway.
We could just let #334 test php8.4 (or wait until rc1 before merging it), and iff it starts failing, at that point either fix our code to make it pass, or temporarily stop testing php8.4 until we have a fix.

[connorhu]

I think in our case (seeing our reaction time) it is very important to foresee what the new version will bring (and run tests with that version). I'm not sure if it's worth it to make it switchable either, but in the end it's not a big change (I'd rather test it with phpunit).

[alquerci]

Like a feature flag

The overall test suite should pass when the feature flag is enabled or disabled.

Regarding this principle, a failed test should always fail the job.

But how to progressively make symfony1 compatible with next PHP version?

Test case after test case.

1. Starts with one test file, passes on next PHP version.
2. Ends with the overall test suite, passes on next PHP version.

With this plan, each contributor can provide a small PR with one test file executed on the next PHP version.