Merge ktxtools into main

[VaderDev]

PR to merge the contents of the ktxtools branch into main.
This PR should be functionally identical to the ktxtools branch. Changes other than the rebase itself are in the separate second commit.

Briefs:

• Rebase ktxtools onto main
• Update the CTS golden files to pickup the changes in ASTC and ZSTD versions
• Add CTS testing to MinGW and Windows CI builds
• Pickup changes in image.hpp from main to ktxtools's image.hpp
• Fix any remaining warnings

Please complete the final review of this scope including the CTS.

[MarkCallow]

Looks good. Two very minor change requests. One unrelated to your work but in a place you are already changing.

[MarkCallow]

I'm mystified by the macOS build failures when BASISU_SUPPORT_SSE=ON:

1. The PR has no changes to the warning disables c/f main which is succeeding.
2. The compiler version is the same as in the main CI build.
3. I don't recall ever seeing this warning before. I saw deprecated-copy warnings which are due to having a user-defined destructor but no user-defined copy operator. I disabled that one for the basisu encoder.

[MarkCallow]

There are no CMake commands either in the root CMakeLists.txt or tools/ktx/CMakeLists.txt to install the new tool or add it to the cpack tools component. Are they somewhere else that I missed?

[VaderDev]

I'm mystified by the macOS build failures when BASISU_SUPPORT_SSE=ON:

Yeah, I was also puzzled by it. Thank you for disabling it.

I didn't disable anything. What are you talking about? Note that BASISU_SUPPORT_SSE will be overridden and turned off if the build detects the CPU is not x86_64 but otherwise it will use whatever value is passed to CMake.

[VaderDev]

I have rebased the PR to the main again.

[VaderDev]

I successfully whacked every compiler warnings.

2 job has a single failing test with (what looks like some stdin issue on Windows with ktx2check):

 Issues in: stdin
    ERROR: Size of image data in file does not match size calculated from
    levelIndex.

    ERROR: ktxTexture2 creation failed: Not a KTX file..
/usr/bin/bash: line 1:  1315 Broken pipe             cat color_grid_uastc_zstd.ktx2
      1316 Segmentation fault      | D:/a/KTX-Software/KTX-Software/build/Release/ktx2check.exe

The ARM tests most likely have to be disabled until there is a way to make the checks happen.
Can I disable the CTS test on that build?

[MarkCallow]

I successfully whacked every compiler warnings.

2 job has a single failing test with (what looks like some stdin issue on Windows with ktx2check):

 Issues in: stdin
    ERROR: Size of image data in file does not match size calculated from
    levelIndex.

    ERROR: ktxTexture2 creation failed: Not a KTX file..
/usr/bin/bash: line 1:  1315 Broken pipe             cat color_grid_uastc_zstd.ktx2
      1316 Segmentation fault      | D:/a/KTX-Software/KTX-Software/build/Release/ktx2check.exe

This is very strange as all tests pass in main. You haven't changed ktx2check, right? Further investigation is needed.

The ARM tests most likely have to be disabled until there is a way to make the checks happen.

"make the checks happen"?

Can I disable the CTS test on that build?

You can disable the CTS checks on the arm64 job but not on the entire Linux build.

[VaderDev]

This is very strange as all tests pass in main. You haven't changed ktx2check, right? Further investigation is needed.

We did not change ktx2check. I checked it locally and it also fails in the current main on Windows with both MSVC and MinGW.

"make the checks happen"?

By either making the output deterministic or having checks with tolerance (like ktx diff).

You can disable the CTS checks on the arm64 job but not on the entire Linux build.

I will only disable the CTS tests for that specific build, so even the unittests will keep running.

[MarkCallow]

This is very strange as all tests pass in main. You haven't changed ktx2check, right? Further investigation is needed.

We did not change ktx2check. I checked it locally and it also fails in the current main on Windows with both MSVC and MinGW.

Something happened yesterday. The GitHub action logs show successful PR builds with the now failing test passing until sometime yesterday. Later yesterday, at build 113, they started failing. Unfortunately I can't find either the time or commit being built in the logs. Most irritating.

"make the checks happen"?

By either making the output deterministic or having checks with tolerance (like ktx diff).

Thanks.

You can disable the CTS checks on the arm64 job but not on the entire Linux build.

I will only disable the CTS tests for that specific build, so even the unittests will keep running.

Sounds good. As the CI services use "build" to refer to the all the configurations that are run and "job" to refer to a single configuration. I make my comment to make sure you would only disable the tests in the single configuration.

[MarkCallow]

Something happened yesterday. The GitHub action logs show successful PR builds with the now failing test passing until sometime yesterday. Later yesterday, at build 113, they started failing. Unfortunately I can't find either the time or commit being built in the logs. Most irritating.

I've started seeing the same issue in my Windows builds. So far I've been unable to reproduce it locally. However when I inadvertently ran the command used by the test in PowerShell instead of Bash

cat color_grid_uastc_zstd.ktx2 | ktx2check

I got a very similar error

Issues in: stdin
    FATAL: Not a KTX2 file.
    Aborting validation.

I wonder if the tests are somehow being run in PowerShell. The configuration output shows it finding the Git 4 Windows bash so $BASH_EXECUTABLE should be set correctly so I don't see how that could be possible.

[MarkCallow]

The pipe read test failure is happening in all builds now. Git for Windows in both the vs2022 and vs2019 runner images was updated from 2.40.1 to 2.41.0 on June 6th. I suspect something in this update is the cause of the failure.

[MarkCallow]

I suspect something in this update is the cause of the failure.

Yup. I updated Git on my Windows 11 to 2.41.0 and now I see the failure locally. I'll look into it more later today.

[VaderDev]

I suspect something in this update is the cause of the failure.

Yup. I updated Git on my Windows 11 to 2.41.0 and now I see the failure locally. I'll look into it more later today.

I suspected something similar. Thank you.

[VaderDev]

Sounds good. As the CI services use "build" to refer to the all the configurations that are run and "job" to refer to a single configuration. I make my comment to make sure you would only disable the tests in the single configuration.

I have disabled the CTS testing on the single ARM job. Yes, We were talking about the same thing but I was not aware the terminology difference. Thank you for being explicit and letting me know.

[MarkCallow]

Please fix the merge conflicts. I'm not sure where they've come from. The changes I just checked in to main are small. Sorry for the trouble. Also if you rebase, you'll get the fix to not run the failing pipe test on Windows.

The 4.2.0 release is done. It is building now. Once the rebase is complete I can merge this.

[MarkCallow]

I'm not sure where they've come from.

It's RELEASE_NOTES.md. Just accept the file in main. The bit you changed (master->main) no longer exists in the new release notes.

[VaderDev]

I have rebased the PR to the main.