Add Futhark + Pyopencl-based GPU-bound entropy calculation #488
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…periments in injecting code into the generated entropy.py pyopencl library to allow progress tracking
…s. no_gpu still needs work. Set up ofrak_gpu to be a proper module. It now has a makefile, requerements, setup.py, etc. The dockerstub automatically installs the latest release of futhark and is used to create entropy.py from entropy.fut at build time. Unfortunately, this will probably get rejected, which sucks because making that Dockerstub was a pain. Had to split up the entropy tests into two separate files, one that runs the tests using a fixture to always trick the system into thinking ofrak_gpu is installed, to test the DataSummaryAnalyzer's behavior when ofrak_gpu is present, and one that uses a fixture to always trick the system into thinking ofrak_gpu has not been installed, to ensure DataSummaryAnalyzer's current behavior remains unaffected. These fixtures were incredibly annoying and confusing to get working consistently. As of this commit, no_gpu is still in need of more research. However, these efforts will allow us to fully test DataSummaryAnalyzer's behavior consistently, regardless of whether or not the ofrak_gpu module has been installed on the testing environment, which is very important for reliability (and unforuntately very difficult to do).
It is generated at build-time from entropy.fut, thanks to the Dockerstub and Makefile. It's hugely messy for VCS tracking
…talled now work Have since realized that we now have a numpy problem. Will have to figure out how to mock this in test_entropy_component_gpu
… with and without ofrak_gpu, while testing the fallback behavior of DataSummaryAnalyzer correctly
…balloonsecurity-master
…thing :( test_enteopy_component method works
…stalled using patching and MagicMock to stop breaking the pjson tests. Still need to redo the tests that act like ofrak_gpu is NOT installed
…frak_gpu on ARM, finish writing tests for ofrak_gpu
|
Tests fail with zero / null bytes on Apple Silicon M3 Mac and pocl/pyopencl installed from conda-forge. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
One sentence summary of this PR (This should go in the CHANGELOG!)
Adds an
ofrak_gpumodule containing GPU-bound entropy calculation, and updates theDataSummaryAnalyzerto try usingofrak_gpu's implementation before the C/Python implementations inofrak_core.Link to Related Issue(s)
N/A
Please describe the changes in your request.
ofrak_gpumodule containing Futhark source code to compute the entropy of given data.DataSummaryAnalyzerto try usingofrak_gpu'spyopencl library, then try the C implementation, then finally fall back on the Python implementationentropy_gpu's correctness on certain edge cases and againstofrak_core's C/Python implementationsofrak_core/test_ofrak/components/test_entropy_component.pyto testDataSummaryAnalyzer's behavior, asserting thatDataSummaryAnalyzerwill useofrak_gpu's entropy calculation, if installed, before attempting to use C/Python. This was written using mocks in a way that ensures the test will run, pass, and correctly assessDataSummaryAnalyzer, whether or notofrak_gpuis present in the testing environment.Anyone you think should look at this, specifically?
@whyitfor
** Potential TODOs:**
x86_64device that does not have a GPU, so only CPU-based OpenCL runtimes have been tested. Would be smart to test this on ARM, real GPUs, etc.