Create usage tests in the CI

[talregev]

Create usage tests in the CI
@dg0yt @autoantwort
What do you think? How we can implement such a thing?

[autoantwort]

I have an idea how to implement that, but I don't think that the vcpkg team has time to review and merge it...

[talregev]

I have an idea how to implement that, but I don't think that the vcpkg team has time to review and merge it...

Maybe they will invest the extra time review it, because it will save them a lot of time in review.

[autoantwort]

There is also microsoft/vcpkg-tool#802 which would same them a lot of time in review and no response since a year ...

[talregev]

@BillyONeal What do you think?

[dg0yt]

For now, the solution is: test ports.
AFAICT it scales well, it is accepted without discussions, and with very little code, you can test more than the vcpkg crew would be able to test manually: You choose which headers to include, which symbols to use, and which extra cmake code to call.
And for very important ports, you list them with =pass in the CI baseline.

Slightly extended example/template:
https://github.com/microsoft/vcpkg/tree/master/scripts/test_ports/vcpkg-ci-soci

[talregev]

You must be talking about different things.

We are talking the same thing. For example on the ls-qpack they test the usage only on windows, but there was a problem the usage on linux: #40710

Because you don't spend 60 seconds on a test port which would pay back quickly.

You don't spend the 60 seconds yourself when you update ports.

Their work is full of surprises. Obviously, they write test projects. Everytime. Again. Even they would save time if they would do it only once as an added test port.

You can write them and ask them to change the policy, but I don't think is the solution. I think automatic tests is the solution.

[dg0yt]

For example on the ls-qpack they test the usage only on windows, but there was a problem the usage on linux: #40710

So you talk about manual tests. I talk about test ports which run in CI. Different things.

You don't spend the 60 seconds yourself when you update ports.

No. Apart from pointed wording, this was explicitly about copying an existing test port and doing tiny adjustment to 4 files.

You can write them and ask them to change the policy, but I don't think is the solution. I think automatic tests is the solution.

The problem is: The solution will not be available in 60 seconds, and probably not even in 60 weeks.
And the delay for some relevant tool PRs is actively discouraging contributions.

OTOH every second spend on writing a test port quickly pays back.

[dg0yt]

(Obsolete, but I will reduce my further input.)

[talregev]

already submitted PRs to run simple usage checks as part of the portfile.

So you talk about manual tests. I talk about test ports which run in CI. Different things.

We both talk about the manual tests when submitted a PR, and I continue to take about that.

And my main talk is about automatic tests.

[talregev]

Most of the people and most of the ports lake usage tests on the CI. People not do that even if 60 seconds and there is no policy that enforce that.

I think this is not the way, and we need to think about things that will scale up, meaning that it will not add more work for people when add or update ports.