-
Notifications
You must be signed in to change notification settings - Fork 46
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
CeedRequestWait is documented but not implemented #700
Comments
This is a good point. We envisioned this as a consistent way to handle CPU threads and GPU streams, but now I'm less confident that this is the right interface (at least for GPUs). We've recently been experimenting in PETSc with offloading the scalar math for Krylov methods, such that the host doesn't need to block. We might want to switch to a stream model, something like |
I think the community is slowly moving away from streams/queues and towards more declarative/graph based APIs. Good examples here are the (relatively recent) CUDA graph management APIs and the command list approach taken by Intel's Level Zero (which is probably the gold standard in terms of HPC API design at the moment, albeit being somewhat low level). The difficulty I've always had with designing these kind of interfaces (and we have our own queues in PyFR) is how to handle CPUs. If we execute the kernel immediately from the API then the call is not asynchronous (whereas it would be on a GPU if you launched it in a stream). This confuses users who expect you to return immediately so they can go off and do some I/O while waiting for the kernel to finish. If you defer execution until a later wait/sync call then you may catch out some users who expect forward progress to be made. (MPI libraries often run into this problem with non-blocking communication where if you're not probing/waiting then nothing happens.) Finally, if you spawn a thread you may and do the work there you're increasing overhead and may be considered a bad citizen. The problem is even more fiddly when you're a library and need to integrate with the models of a large number of stakeholders. |
I agree this is a tricky issue for a library like libCEED, which may need to use various task/execution queue models (through backends targeting different architecture), while also being used by other libraries/applications. Currently, the CUDA/HIP/MAGMA backends all use the NULL stream (and only the NULL stream). For the CUDA/HIP backends, this is baked into the kernel launch functions, see e.g. |
I am currently reviewing libCEED as part of openjournals/joss-reviews#2945.
The method CeedRequestWait is documented and seems extremely interesting. However, looking through the source it does not appear to be actually implemented. It might be prudent to relegate this method and associated paraphernalia to a feature branch until it is ready for mainline. If nothing else it may be that when implementing that that a blocker raises its head or a simpler approach shows itself.
The text was updated successfully, but these errors were encountered: