python tests failing in ctests - No module named 'ncepbufr'

[PatrickHildreth-NOAA]

After running cmake with:

cmake -DENABLE_PYTHON=ON -DCMAKE_INSTALL_PREFIX=./ncep_libs -DMASTER_TABLE_DIR=./ncep_tables ../NCEPLIBS-bufr-bufr_v12.0.0

and then running 'make -j4' ; 'ctests' all pass except for the python related tests which all seem to be unable to locate the 'ncepbufr' module (below)

I've gotten the same result on MacOS 10.15.7 and Redhat8 linux. I also note that the directories './ncep_libs' and './ncep_tables' do not get built under 'build' Am I specifying those correctly?

Is this a known issue and/or is there a workaround?

Failed tests output:
ctest --rerun-failed --output-on-failure
Test project /Users/patrick.hildreth/projects/python/venv/ncep_bufr_lib_proj/ncep_bufr_lib_root/build
Start 1: test_pyncepbufr_checkpoint
1/7 Test #1: test_pyncepbufr_checkpoint .......***Failed 0.03 sec
Traceback (most recent call last):
File "/Users/patrick.hildreth/projects/python/venv/ncep_bufr_lib_proj/ncep_bufr_lib_root/NCEPLIBS-bufr-bufr_v12.0.0/python/test/test_checkpoint.py", line 2, in
import ncepbufr
ModuleNotFoundError: No module named 'ncepbufr'

Start 2: test_pyncepbufr_gps

2/7 Test #2: test_pyncepbufr_gps ..............***Failed 0.02 sec
Traceback (most recent call last):
File "/Users/patrick.hildreth/projects/python/venv/ncep_bufr_lib_proj/ncep_bufr_lib_root/NCEPLIBS-bufr-bufr_v12.0.0/python/test/test_gps.py", line 2, in
import ncepbufr
ModuleNotFoundError: No module named 'ncepbufr'

Start 3: test_pyncepbufr_prepbufr

3/7 Test #3: test_pyncepbufr_prepbufr .........***Failed 0.03 sec
Traceback (most recent call last):
File "/Users/patrick.hildreth/projects/python/venv/ncep_bufr_lib_proj/ncep_bufr_lib_root/NCEPLIBS-bufr-bufr_v12.0.0/python/test/test_prepbufr.py", line 2, in
import ncepbufr
ModuleNotFoundError: No module named 'ncepbufr'

Start 4: test_pyncepbufr_rad

4/7 Test #4: test_pyncepbufr_rad ..............***Failed 0.02 sec
Traceback (most recent call last):
File "/Users/patrick.hildreth/projects/python/venv/ncep_bufr_lib_proj/ncep_bufr_lib_root/NCEPLIBS-bufr-bufr_v12.0.0/python/test/test_rad.py", line 2, in
import ncepbufr
ModuleNotFoundError: No module named 'ncepbufr'

Start 5: test_pyncepbufr_satwnd

5/7 Test #5: test_pyncepbufr_satwnd ...........***Failed 0.03 sec
Traceback (most recent call last):
File "/Users/patrick.hildreth/projects/python/venv/ncep_bufr_lib_proj/ncep_bufr_lib_root/NCEPLIBS-bufr-bufr_v12.0.0/python/test/test_satwnd.py", line 2, in
import ncepbufr
ModuleNotFoundError: No module named 'ncepbufr'

Start 6: test_pyncepbufr_write

6/7 Test #6: test_pyncepbufr_write ............***Failed 0.03 sec
Traceback (most recent call last):
File "/Users/patrick.hildreth/projects/python/venv/ncep_bufr_lib_proj/ncep_bufr_lib_root/NCEPLIBS-bufr-bufr_v12.0.0/python/test/test_write.py", line 2, in
import ncepbufr
ModuleNotFoundError: No module named 'ncepbufr'

Start 7: test_pyncepbufr_test

7/7 Test #7: test_pyncepbufr_test .............***Failed 0.03 sec
Traceback (most recent call last):
File "/Users/patrick.hildreth/projects/python/venv/ncep_bufr_lib_proj/ncep_bufr_lib_root/NCEPLIBS-bufr-bufr_v12.0.0/python/test/test.py", line 2, in
import ncepbufr
ModuleNotFoundError: No module named 'ncepbufr'

0% tests passed, 7 tests failed out of 7

Total Test time (real) = 0.20 sec

The following tests FAILED:
1 - test_pyncepbufr_checkpoint (Failed)
2 - test_pyncepbufr_gps (Failed)
3 - test_pyncepbufr_prepbufr (Failed)
4 - test_pyncepbufr_rad (Failed)
5 - test_pyncepbufr_satwnd (Failed)
6 - test_pyncepbufr_write (Failed)
7 - test_pyncepbufr_test (Failed)
Errors while running CTest

[jbathegit]

Sorry @PatrickHildreth-NOAA that you are having this problem. As noted in #186 and #231, this isn't the first time we've seen this problem. At one point in #199 we thought we had it fixed, but apparently there's still some lingering issues.

For my part, the python wrapper is the one aspect of the library that I'm the least knowledgeable about, so I've been relying on others (@jswhit, @edwardhartnett, @aerorahul, etc.) and others to help support that part. Hopefully one of them can chime in and help.

[PatrickHildreth-NOAA]

Thanks for getting back to me Jeff. Hopefully one of the others can assist. If there's any testing I can do to help, I'd be more than happy to do that. Patrick Hildreth NOAA/Global Systems Laboratory IT Services/Data Services Group ***@***.***

…

On Tue, Aug 29, 2023 at 2:16 PM Jeff Ator ***@***.***> wrote: Sorry @PatrickHildreth-NOAA <https://github.com/PatrickHildreth-NOAA> that you are having this problem. As noted in #186 <#186> and #231 <#231>, this isn't the first time we've seen this problem. At one point in #199 <#199> we thought we had it fixed, but apparently there's still some lingering issues. For my part, the python wrapper is the one aspect of the library that I'm the least knowledgeable about, so I've been relying on others ***@***.*** <https://github.com/jswhit>, @edwardhartnett <https://github.com/edwardhartnett>, @aerorahul <https://github.com/aerorahul>, etc.) and others to help support that part. Hopefully one of them can chime in and help. — Reply to this email directly, view it on GitHub <#520 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACIX5NZFZIU37T74YB7CPCLXXZEX7ANCNFSM6AAAAAA364YGHI> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[jbathegit]

cmake -DENABLE_PYTHON=ON -DCMAKE_INSTALL_PREFIX=./ncep_libs -DMASTER_TABLE_DIR=./ncep_tables ../NCEPLIBS-bufr-bufr_v12.0.0

I also note that the directories './ncep_libs' and './ncep_tables' do not get built under 'build' Am I specifying those correctly?

Did you ever run make install? Those directories typically don't get created until the make install step, and it wasn't clear to me if you ever tried running that step, given that you had reported problems with the previous ctest step.

If that doesn't explain it, then you may also want to try using absolute pathnames instead of relative pathnames.

[jbathegit]

To clarify, relative pathnames usually aren't a problem; however, typically these are just specified as, e.g. ncep_libs, ncep_tables. In other words, including the preceding ./ notation shouldn't be necessary, and it could actually be throwing something off somewhere in one of the underlying scripts.

But again, if you never ran make install, then that's the more likely explanation for why you never saw either of those directories getting created.

[AlexanderRichert-NOAA]

I was just about to create a related issue and saw this. The python cmake config assumes that make install gets run prior to make test. One unfortunate consequence is that this means we can't run unit tests through spack, which runs test, then install. Another is that if a user forgets to run install first but has an existing installation, the test will be run but against the existing version of the library. It would be nice if the python module could be built in the regular make step like the rest of the code, that way users can run the tests before installing. @aerorahul any thoughts on how easy this would be? I can look into this further but I'm not really familiar with all the python/setup.py/etc. stuff so I don't know how deeply baked this assumption is about building the python module as part of the install step rather than at code build time.

[jbathegit]

@PatrickHildreth-NOAA hopefully this is fixed now, because the python extensions to the library are now generated during the build step and not the install step. Thanks to @AlexanderRichert-NOAA for figuring this out!

[PatrickHildreth-NOAA]

I still get all seven python tests failing before and after running 'make install' (The install is creating the custom install and tables paths though) [I tried reinstalling using the 12.0.1 release] The following tests FAILED: 1 - test_pyncepbufr_checkpoint (Failed) 2 - test_pyncepbufr_gps (Failed) 3 - test_pyncepbufr_prepbufr (Failed) 4 - test_pyncepbufr_rad (Failed) 5 - test_pyncepbufr_satwnd (Failed) 6 - test_pyncepbufr_write (Failed) 7 - test_pyncepbufr_test (Failed) Errors while running CTest Start 1: test_pyncepbufr_checkpoint 1/7 Test #1: test_pyncepbufr_checkpoint .......***Failed 0.02 sec Traceback (most recent call last): File "/Users/patrick.hildreth/projects/python/venv/ncep_bufr_lib_proj/ncep_bufr_lib_root/NCEPLIBS-bufr-12.0.1/python/test/test_checkpoint.py", line 2, in <module> import ncepbufr ModuleNotFoundError: No module named 'ncepbufr' For some reason the module can't be located. Patrick Hildreth NOAA/Global Systems Laboratory IT Services/Data Services Group ***@***.***

…

On Wed, Sep 27, 2023 at 11:06 AM Jeff Ator ***@***.***> wrote: @PatrickHildreth-NOAA <https://github.com/PatrickHildreth-NOAA> hopefully this is fixed now, because the python extensions to the library are now generated during the build step and not the install step. Thanks to @AlexanderRichert-NOAA <https://github.com/AlexanderRichert-NOAA> for figuring this out! — Reply to this email directly, view it on GitHub <#520 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACIX5N7CS4FAMNV7FPU2RUTX4RMIJANCNFSM6AAAAAA364YGHI> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[jbathegit]

Sorry for any confusion Patrick, but just to clarify, PR #524 was the first commit to the repository after the 12.0.1 release. In other words, it won't be part of an official release until whenever I can get around to doing a new future release. So if you want to test with the latest changes, then you'll need to clone the develop branch in the https://github.com/NOAA-EMC/NCEPLIBS-bufr repository, which is now post-12.0.1.

I hope that makes sense, and once you pull down that latest commit then it should hopefully work for you using the same build instructions as before. But if not, then please let us know and feel free to open another issue, and we'll take another look at this. And thanks again for your patience as we work through all of our outstanding issues!

[PatrickHildreth-NOAA]

Hi Jeff, That makes sense to me. I'll give it a go later today. Thanks for your assistance. Patrick Hildreth NOAA/Global Systems Laboratory IT Services/Data Services Group ***@***.***

…

On Wed, Sep 27, 2023 at 5:34 PM Jeff Ator ***@***.***> wrote: Sorry for any confusion Patrick, but just to clarify, PR #524 <#524> was the first commit to the repository *after* the 12.0.1 release. In other words, it won't be part of an official release until whenever I can get around to doing a new future release. So if you want to test with the latest changes, then you'll need to clone the develop branch in the https://github.com/NOAA-EMC/NCEPLIBS-bufr repository, which is now post-12.0.1. I hope that makes sense, and once you pull down that latest commit then it should hopefully work for you using the same build instructions as before. But if not, then please let us know and feel free to open another issue, and we'll take another look at this. And thanks again for your patience as we work through all of our outstanding issues! — Reply to this email directly, view it on GitHub <#520 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACIX5N6YD7ODAQ22CUFI3WLX4SZXLANCNFSM6AAAAAA364YGHI> . You are receiving this because you were mentioned.Message ID: ***@***.***>