dap4_test_test_raw not working on cmake builds?

[edhartnett]

When I check out current master, and do a cmake build, I am now seeing this failure:

158/165 Test #158: dap4_test_test_raw ....................***Failed    5.43 sec
        Start 159: C_tests_simple_xy_wr

This doesn't seem to happen with autotool builds. This seems new but I can't really say.

[WardF]

Not seeing this on our end; can you share how you are configuring with cmake, and I'll try to recreate?

[DennisHeimbigner]

Ed-
Try to locate a file in your cmake build directory named
.../Testing/Temporary/LastTest.log
It may contain more info about why it failed.

[edhartnett]

I am seeing this in every cmake build now. Perhaps there's a server down or slow?

Here's an example. I checkout the latest master and do:

cd build
CPPFLAGS='-I/usr/local_cmake/include' CFLAGS='-fPIC -g -Wall' LDFLAGS='-L/usr/local_cmake/lib' LD_LIBRARY_PATH='/usr/local_cmake/lib' cmake -D"BUILD_SHARED_LIBS=ON" -DHDF5_C_LIBRARY=/usr/local_cmake/lib/libhdf5.so -DHDF5_HL_LIBRARY=/usr/local_cmake/lib/libhdf5_hl.so -DHDF5_INCLUDE_DIR=/usr/local_cmake/include -D"CMAKE_PREFIX_PATH=/usr/local_cmake/" ..

CPPFLAGS='-I/usr/local_cmake/include' CFLAGS='-fPIC -g -Wall' LDFLAGS='-L/usr/local_cmake/lib' LD_LIBRARY_PATH='/usr/local_cmake/lib' make all test

And I get:

The following tests FAILED:
	158 - dap4_test_test_raw (Failed)
Errors while running CTest
Makefile:151: recipe for target 'test' failed
make: *** [test] Error 8

[edhartnett]

I am going to try this again later and see if I get the same result. I may be having some network issues here that are preventing a connection from being made in time, and so something's timing out.

[DennisHeimbigner]

I don't think that the dap4 tests are doing any remoter server accesses at the moment
because we have not found a home for our dap4 test server.
The term "raw" means that it is reading the raw on the wire data from a file.
I just ran locally and am not seeing any failures.
Also, you might insert this in the test_raw.sh file to get more output into the LastTest.log
file:
export SETX=1

[edhartnett]

OK, my file system is being hammered by loads of CI testing, and has been all day. I've been playing with the new valgrind testing and it's pretty freaking slow. So I will try again when all my CI tests have stopped and see if I still get this problem.

[edhartnett]

OK, total false alarm. Shared library problem. It couldn't find the hdf5 HL library.

[edhartnett]

No, it was not a shared library problem. It is something I don't understand. ncdump is segfaulting for the tests in test_raw.sh.

This is from a fresh clone of the unidata netCDF-c repo, built like this:

ed@mikado:~/unidata$ git clone git@github.com:Unidata/netcdf-c.git
Cloning into 'netcdf-c'...
remote: Counting objects: 37842, done.        
remote: Compressing objects: 100% (17/17), done.        
remote: Total 37842 (delta 10), reused 16 (delta 6), pack-reused 37819        
Receiving objects: 100% (37842/37842), 41.70 MiB | 1.16 MiB/s, done.
Resolving deltas: 100% (28586/28586), done.
ed@mikado:~/unidata$ cd netcdf-c
ed@mikado:~/unidata/netcdf-c$ mkdir build
ed@mikado:~/unidata/netcdf-c$ cd build
ed@mikado:~/unidata/netcdf-c/build$ CPPFLAGS='-I/usr/local_cmake/include' CFLAGS='-fPIC -g -Wall' LDFLAGS='-L/usr/local_cmake/lib' LD_LIBRARY_PATH='/usr/local_cmake/lib' cmake -D"BUILD_SHARED_LIBS=ON" -DHDF5_C_LIBRARY=/usr/local_cmake/lib/libhdf5.so -DHDF5_HL_LIBRARY=/usr/local_cmake/lib/libhdf5_hl.so -DHDF5_INCLUDE_DIR=/usr/local_cmake/include -D"CMAKE_PREFIX_PATH=/usr/local_cmake/" ..

-- The C compiler identification is GNU 7.2.0
-- Check for working C compiler: /usr/bin/cc
-- Check for working C compiler: /usr/bin/cc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Check if the system is big endian
-- Searching 16 bit integer
-- Looking for sys/types.h
-- Looking for sys/types.h - found
-- Looking for stdint.h
-- Looking for stdint.h - found
-- Looking for stddef.h
-- Looking for stddef.h - found
-- Check size of unsigned short
-- Check size of unsigned short - done
-- Using unsigned short
-- Check if the system is big endian - little endian
-- Performing Test LIBTOOL_HAS_NO_UNDEFINED
-- Performing Test LIBTOOL_HAS_NO_UNDEFINED - Success
-- Performing Test CC_HAS_WCONVERSION
-- Performing Test CC_HAS_WCONVERSION - Success
-- Performing Test CC_HAS_SHORTEN_64_32
-- Performing Test CC_HAS_SHORTEN_64_32 - Failed
-- Using HDF5 C Library: /usr/local_cmake/lib/libhdf5.so
-- Using HDF5 HL LIbrary: /usr/local_cmake/lib/libhdf5_hl.so
-- Looking for H5Pget_fapl_mpiposix in /usr/local_cmake/lib/libhdf5.so
-- Looking for H5Pget_fapl_mpiposix in /usr/local_cmake/lib/libhdf5.so - not found
-- Looking for H5Pget_fapl_mpio in /usr/local_cmake/lib/libhdf5.so
-- Looking for H5Pget_fapl_mpio in /usr/local_cmake/lib/libhdf5.so - not found
-- Looking for H5Pset_all_coll_metadata_ops in /usr/local_cmake/lib/libhdf5.so
-- Looking for H5Pset_all_coll_metadata_ops in /usr/local_cmake/lib/libhdf5.so - not found
-- Looking for H5Z_SZIP in /usr/local_cmake/lib/libhdf5.so
-- Looking for H5Z_SZIP in /usr/local_cmake/lib/libhdf5.so - found
-- Looking for H5Pset_libver_bounds in /usr/local_cmake/lib/libhdf5.so
-- Looking for H5Pset_libver_bounds in /usr/local_cmake/lib/libhdf5.so - found
-- Found CURL: /usr/lib/x86_64-linux-gnu/libcurl.so (found version "7.55.1") 
-- Performing Test HAVE_CURLOPT_USERNAME
-- Performing Test HAVE_CURLOPT_USERNAME - Success
-- Performing Test HAVE_CURLOPT_PASSWORD
-- Performing Test HAVE_CURLOPT_PASSWORD - Success
-- Performing Test HAVE_CURLOPT_KEYPASSWD
-- Performing Test HAVE_CURLOPT_KEYPASSWD - Success
-- Performing Test HAVE_CURLINFO_RESPONSE_CODE
-- Performing Test HAVE_CURLINFO_RESPONSE_CODE - Success
-- Performing Test HAVE_CURLINFO_HTTP_CONNECTCODE
-- Performing Test HAVE_CURLINFO_HTTP_CONNECTCODE - Success
-- Found Math library: /usr/lib/x86_64-linux-gnu/libm.so
-- Looking for math.h
-- Looking for math.h - found
-- Looking for unistd.h
-- Looking for unistd.h - found
-- Looking for alloca.h
-- Looking for alloca.h - found
-- Looking for malloc.h
-- Looking for malloc.h - found
-- Looking for fcntl.h
-- Looking for fcntl.h - found
-- Looking for getopt.h
-- Looking for getopt.h - found
-- Looking for locale.h
-- Looking for locale.h - found
-- Looking for stdio.h
-- Looking for stdio.h - found
-- Looking for stdlib.h
-- Looking for stdlib.h - found
-- Looking for stdarg.h
-- Looking for stdarg.h - found
-- Looking for strings.h
-- Looking for strings.h - found
-- Looking for signal.h
-- Looking for signal.h - found
-- Looking for sys/param.h
-- Looking for sys/param.h - found
-- Looking for sys/stat.h
-- Looking for sys/stat.h - found
-- Looking for sys/time.h
-- Looking for sys/time.h - found
-- Looking for sys/mman.h
-- Looking for sys/mman.h - found
-- Looking for sys/resource.h
-- Looking for sys/resource.h - found
-- Looking for inttypes.h
-- Looking for inttypes.h - found
-- Looking for pstdint.h
-- Looking for pstdint.h - found
-- Looking for endian.h
-- Looking for endian.h - found
-- Looking for BaseTsd.h
-- Looking for BaseTsd.h - not found
-- Looking for string.h
-- Looking for string.h - found
-- Looking for winsock2.h
-- Looking for winsock2.h - not found
-- Looking for ftw.h
-- Looking for ftw.h - found
-- Looking for libgen.h
-- Looking for libgen.h - found
-- Looking for isfinite
-- Looking for isfinite - found
-- Looking for isnan
-- Looking for isnan - found
-- Looking for isinf
-- Looking for isinf - found
-- Looking for st_blksize
-- Looking for st_blksize - not found
-- Looking for alloca
-- Looking for alloca - found
-- Looking for snprintf
-- Looking for snprintf - found
-- Check size of char
-- Check size of char - done
-- Check size of double
-- Check size of double - done
-- Check size of float
-- Check size of float - done
-- Check size of int
-- Check size of int - done
-- Check size of uint
-- Check size of uint - done
-- Check size of schar
-- Check size of schar - failed
-- Check size of long
-- Check size of long - done
-- Check size of long long
-- Check size of long long - done
-- Check size of unsigned long long
-- Check size of unsigned long long - done
-- Check size of off_t
-- Check size of off_t - done
-- Check size of off64_t
-- Check size of off64_t - done
-- Check size of short
-- Check size of short - done
-- Check size of ushort
-- Check size of ushort - done
-- Check size of _Bool
-- Check size of _Bool - done
-- Check size of size_t
-- Check size of size_t - done
-- Check size of ssize_t
-- Check size of ssize_t - done
-- Check size of __int64
-- Check size of __int64 - failed
-- Check size of int64_t
-- Check size of int64_t - done
-- Check size of uint64_t
-- Check size of uint64_t - done
-- Check size of unsigned char
-- Check size of unsigned char - done
-- Check size of unsigned short int
-- Check size of unsigned short int - done
-- Check size of unsigned int
-- Check size of unsigned int - done
-- Check size of long long
-- Check size of long long - done
-- Check size of unsigned long long
-- Check size of unsigned long long - done
-- Looking for fsync
-- Looking for fsync - found
-- Looking for strlcat
-- Looking for strlcat - not found
-- Looking for strdup
-- Looking for strdup - found
-- Looking for strndup
-- Looking for strndup - found
-- Looking for strtoll
-- Looking for strtoll - found
-- Looking for strtoull
-- Looking for strtoull - found
-- Looking for mkstemp
-- Looking for mkstemp - found
-- Looking for mktemp
-- Looking for mktemp - found
-- Looking for random
-- Looking for random - found
-- Looking for gettimeofday
-- Looking for gettimeofday - found
-- Looking for MPI_Comm_f2c
-- Looking for MPI_Comm_f2c - not found
-- Looking for memmove
-- Looking for memmove - found
-- Looking for getpagesize
-- Looking for getpagesize - found
-- Looking for sysconf
-- Looking for sysconf - found
-- Looking for getrlimit
-- Looking for getrlimit - found
-- Looking for _filelengthi64
-- Looking for _filelengthi64 - not found
-- Looking for mremap
-- Looking for mremap - found
-- Found bash: /bin/bash
-- CMAKE_SOURCE_DIR=/home/ed/unidata/netcdf-c
-- CMAKE_BINARY_DIR=/home/ed/unidata/netcdf-c/build
-- CMAKE_CURRENT_SOURCE_DIR=/home/ed/unidata/netcdf-c/nc_test
-- CMAKE_CURRENT_BINARY_DIR=/home/ed/unidata/netcdf-c/build/nc_test


Configuration Summary:

-- Building Shared Libraries:     ON
-- Building netCDF-4:             ON
-- Building DAP2 Support:         ON
-- Building DAP4 Support:         ON
-- Building Utilities:            ON
-- CMake Prefix Path:             /usr/local_cmake/

Tests Enabled:              ON
-- DAP Remote Tests:  ON
-- Extra Tests:       OFF
-- Coverage Tests:    OFF
-- Parallel Tests:    OFF
-- Large File Tests:  OFF
-- Extreme Numbers:   ON

Compiler:

-- Build Type:           DEBUG
-- CMAKE_C_COMPILER:     /usr/bin/cc
-- CMAKE_C_FLAGS:        -fPIC -g -Wall -D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=64
-- CMAKE_C_FLAGS_DEBUG:  -g -Wall -Wconversion
-- Linking against:      /usr/local_cmake/lib/libhdf5_hl.so;/usr/local_cmake/lib/libhdf5.so;/usr/lib/x86_64-linux-gnu/libm.so;/usr/local_cmake/lib/libsz.so;/usr/lib/x86_64-linux-gnu/libcurl.so

# NetCDF C Configuration Summary
==============================

# General
-------
NetCDF Version:		4.6.1-development
Configured On:		Tue Feb 27 11:43:42 MST 2018
Host System:		x86_64-Linux-4.13.0-36-generic
Build Directory: 	/home/ed/unidata/netcdf-c/build
Install Prefix:         /usr/local

# Compiling Options
-----------------
C Compiler:		/usr/bin/cc
CFLAGS:			-fPIC -g -Wall -D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=64 -g -Wall -Wconversion
CPPFLAGS:		 
LDFLAGS:		-L/usr/local_cmake/lib  -Wl,--no-undefined
AM_CFLAGS:		
AM_CPPFLAGS:		
AM_LDFLAGS:		
Shared Library:		yes
Static Library:		no
Extra libraries:	-lhdf5_hl -lhdf5 -lm -lsz -lcurl

# Features
--------
NetCDF-2 API:		yes
HDF4 Support:		no
NetCDF-4 API:		yes
NC-4 Parallel Support:	no
PNetCDF Support:	no
DAP2 Support:		yes
DAP4 Support:		yes
Diskless Support:	yes
MMap Support:		yes
JNA Support:		no
CDF5 Support:		no
ERANGE fill Support:	no

-- Configuring done
-- Generating done
-- Build files have been written to: /home/ed/unidata/netcdf-c/build
ed@mikado:~/unidata/netcdf-c/build$ ed@mikado:~/unidata/netcdf-c/build$ CPPFLAGS='-I/usr/local_cmake/include' CFLAGS='-fPIC -g -Wall' LDFLAGS='-L/usr/local_cmake/lib' LD_LIBRARY_PATH='/usr/local_cmake/lib' make all test

When I run test_raw.sh I see:

ed@mikado:~/unidata/netcdf-c/build/dap4_test$ CPPFLAGS='-I/usr/local_cmake/include' CFLAGS='-fPIC -g -Wall' LDFLAGS='-L/usr/local_cmake/lib' LD_LIBRARY_PATH='/usr/local_cmake/lib' ./test_raw.sh 
test_raw.sh:
testing: test_anon_dim.2.syn
./test_raw.sh: line 45:   726 Segmentation fault      (core dumped) ${VG} ${NCDUMP} "${URL}" > ./results_test_raw/${f}.dmp
*** Fail: [dap4]file:///home/ed/unidata/netcdf-c/build/dap4_test/daptestfiles/test_anon_dim.2.syn
1,13d0
< netcdf test_anon_dim.2 {
< dimensions:
< 	_Anonymous4 = 4 ;
< variables:
< 	int vu32(_Anonymous4) ;
< 
< // global attributes:
< 		string :_dap4.ce = "/vu32[0:3]" ;
< 		:_DAP4_Little_Endian = 1UB ;
< data:
< 
<  vu32 = -1169720286, -920699049, -2088732436, 1060190036 ;
< }
*** Fail: diff test_anon_dim.2.syn.dmp

How exactly do you guys build with cmake, specifying the directories for HDF5? That is, can you send me what you are using from the command line?

[DennisHeimbigner]

run test_raw.sh with the -x bash flag set so we can see the exact command that
is failing. Then you can use gdb on that command to see where it is crashing.

[edhartnett]

I get this:

(gdb) r '[dap4]file:///home/ed/unidata/netcdf-c/build/dap4_test/daptestfiles/test_anon_dim.2.syn'
Starting program: /home/ed/unidata/netcdf-c/build/ncdump/ncdump '[dap4]file:///home/ed/unidata/netcdf-c/build/dap4_test/daptestfiles/test_anon_dim.2.syn'
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".

Program received signal SIGSEGV, Segmentation fault.
__strlen_avx2 () at ../sysdeps/x86_64/multiarch/strlen-avx2.S:62
62	../sysdeps/x86_64/multiarch/strlen-avx2.S: No such file or directory.
(gdb) up
#1  0x00007ffff7b209e1 in set_curl_options (state=0x5555557c4c40) at /home/ed/unidata/netcdf-c/libdap4/d4curlfunctions.c:203
203	        size_t hostlen = strlen(triple->host);
(gdb) p triple->host
$1 = 0x0
(gdb) p *triple
$2 = {host = 0x0, key = 0x555555786c20 "USE_CACHE", value = 0x555555786b60 "0"}
(gdb) 

[DennisHeimbigner]

ok, I am testing a fix. I thought I fixed this, but apparently did not
do a thorough search for all occurrences of triple->host.

[WardF]

@edhartnett

How exactly do you guys build with cmake, specifying the directories for HDF5? That is, can you send me what you are using from the command line?

Hi Ed, typically I invoke as follows, from the root netcdf-c directory:

$ mkdir build && cd build
$ cmake .. -DCMAKE_PREFIX_PATH=$HOME/environments/local-serial [OTHER OPTIONS]

The path I specify via CMAKE_PREFIX_PATH contains (in this case) serial libhdf5, libhdf4, and other various and sundry installs. Doing a directory listing of local-serial shows the standard lib/, bin/, include/, etc, directory structure.

[edhartnett]

OK thanks! I will try that next time.

[WardF]

I'm working on a presentation for COSMIC tomorrow, but I'm going to try to roll a bunch of outstanding PR's into an aggregation, this afternoon, so that we can get 4.6.1 out and move on to the next.

Mentioning it here because it's as good a place as any :)

[edhartnett]

Awesome.

[DennisHeimbigner]

This issue should be fixed by pr #888

[edhartnett]

Yes, I merged #888 into HPC netCDF and this issue is now fixed. Thanks!