Optimized IsFileValid

[takahiro0327]

Fixed IsFileValid was slow for large files over 200MB. Please review and merge.

400ms became about 80ms.
Converted the token search algorithm to BoyerMoore and to run on a thread pool.
When using NVMe, the major cost was token lookup on the CPU.

[ManlyMarco]

It looks like for very large scenes there may be hundreds of threads created, which will be slower and could possibly cause issues by making thread pool fill up.

There's also a possible race condition where the last chunk completes before a previous chunk with the tag in it completes, causing a false negative.

Maybe it would be possible to use RunParallel? It would avoid the race condition and only use as many threads as necessary.

[takahiro0327]

Thanks for review.

It looks like for very large scenes there may be hundreds of threads created, which will be slower and could possibly cause issues by making thread pool fill up.

The PoolAllocator was restricted by setting an upper limit on the number of instances.

There's also a possible race condition where the last chunk completes before a previous chunk with the tag in it completes, causing a false negative.

Sorry, I was an idiot. Fixed.

Maybe it would be possible to use RunParallel? It would avoid the race condition and only use as many threads as necessary.

It's in the form of making a list at the offset of the position to be read and executing it in parallel, right?

In many cases, it is faster to do nothing but sequential reads. This is especially true for HDDs with physical moving parts and slow SDDs.
It should be better to call fs.Read() sequentially in the main thread to get better speed in many environments.
It is not the fastest in certain environments, but considering the penalty in other environments, I think the current form is better.

Incidentally, in my environment, it appears that a maximum of 5 pools are used for a 200 MB scene.

[ManlyMarco]

Makes sense. Thanks for the fixes, everything seems to be good now.