Add Smoke tests for ASan.

[ampandey-AMD]

No description provided.

[saipoorna]

!psdb

[ampandey-AMD]

!psdb

I am going to push the same changes into branch aomp-epsdb.

[estewart08]

Since these asan tests are supposed to fail at runtime, you need to check the pipestatus to filecheck for the return code.

[gregrodgers]

The changes you made to Makefile.defs causes changes to at least all smoke tests which now fail. Example is test/smoke/veccopy
Before your change, this is the compile command.

/work/grodgers/rocm/aomp/bin/clang -O2 -fopenmp --offload-arch=gfx906 -D__OFFLOAD_ARCH_gfx906__ veccopy.c -o veccopy

After your change, I get this with "make run"
make run
/work/grodgers/rocm/aomp/bin/clang -O2 -fopenmp --offload-arch=gfx906:xnack+ -fsanitize=address -shared-libasan -g -D__OFFLOAD_ARCH_gfx906__ veccopy.c -o veccopy
./veccopy 2>&1 | tee run.log
./veccopy: error while loading shared libraries: libclang_rt.asan-x86_64.so: cannot open shared object file: No such file or directory
make: *** [../Makefile.rules:71: run] Error 127

[gregrodgers]

Regarding your changes to Makefile.defs, I believe you want to say ifneq ($(AOMP_SANITIZER).)
which says "if AOMP_SANITIZER is NOT blank"
Then you can remove the statement AOMP_SANITIZER ?= $(AOMP_SANITIZER)
Also, you need to modify RUNENV. So your ifneq block could look like this:

ifneq ($(AOMP_SANITIZER),)
  ASAN_FLAGS = -fsanitize=address -shared-libasan -g
  # For Sanitizer below environment variable should always be enabled by default
  AOMP_TARGET_FEATURES=:xnack+
  # For Sanitizer we need to set LD_LIBRARY_PATH at runtime
  ASAN_RT_DIR := $(shell $(AOMP)/bin/clang --print-runtime-dir)
  ASAN_ENV = LD_LIBRARY_PATH=$(ASAN_RT_DIR) HSA_XNACK=1
endif

Then change the setting ofr RUNENV in the block of code about line 336 ish to this.

ifeq ($(SET_DEVICE_DEBUG),)
   RUNENV = $(ASAN_ENV) $(TKILL) $(CBL_ENV)
else
   RUNENV = env $(SET_DEVICE_DEBUG) $(ASAN_ENV) $(TKILL) $(CBL_ENV)
endif

There may be one other change to prevent invalid tests with AOMP_SANITIZER is on since it will only work for HSA_XNACK on.

[ampandey-AMD]

The changes you made to Makefile.defs causes changes to at least all smoke tests which now fail. Example is test/smoke/veccopy Before your change, this is the compile command.

/work/grodgers/rocm/aomp/bin/clang -O2 -fopenmp --offload-arch=gfx906 -D__OFFLOAD_ARCH_gfx906__ veccopy.c -o veccopy

After your change, I get this with "make run" make run /work/grodgers/rocm/aomp/bin/clang -O2 -fopenmp --offload-arch=gfx906:xnack+ -fsanitize=address -shared-libasan -g -D__OFFLOAD_ARCH_gfx906__ veccopy.c -o veccopy ./veccopy 2>&1 | tee run.log ./veccopy: error while loading shared libraries: libclang_rt.asan-x86_64.so: cannot open shared object file: No such file or directory make: *** [../Makefile.rules:71: run] Error 127

Thanks @gregrodgers. Actually previously the environment variable AOMP_SANITIZER(Created for testing smoke/smoke-asan tests with AOMP-ASan pipeline) was set with value of AOMP_BUILD_SANITIZER. However, few months back we had by default in aomp_common_vars AOMP_BUILD_SANITIZER=1. So, this dependency was causing by default the AOMP_SANITIZER=1. The current revision of the patch has removed this kind of dependency.

To enable AOMP-ASan testing for smoke/smoke-asan now the users would have to explicitly set AOMP_SANITIZER=1 which executing check_smoke_asan.sh script.

example: $$ AOMP_SANITIZER=1 ./check_smoke_asan.sh
OR
$$ export AOMP_SANITIZER=1 && ./check_smoke_asan.sh

[ampandey-AMD]

Regarding your changes to Makefile.defs, I believe you want to say ifneq ($(AOMP_SANITIZER),) which says "if AOMP_SANITIZER is NOT blank" Then you can remove the statement AOMP_SANITIZER ?= $(AOMP_SANITIZER) Also, you need to modify RUNENV. So your ifneq block could look like this:

ifneq ($(AOMP_SANITIZER),)
  ASAN_FLAGS = -fsanitize=address -shared-libasan -g
  # For Sanitizer below environment variable should always be enabled by default
  AOMP_TARGET_FEATURES=:xnack+
  # For Sanitizer we need to set LD_LIBRARY_PATH at runtime
  ASAN_RT_DIR := $(shell $(AOMP)/bin/clang --print-runtime-dir)
  ASAN_ENV = LD_LIBRARY_PATH=$(ASAN_RT_DIR) HSA_XNACK=1
endif

Hi @gregrodgers, actually I had the patch in gerrit(1030763) for amd-staging for clang-driver & for legacy-flang driver(ROCm/flang#81) both will add the rpath by default when any application is compiled with sanitizer flags. This, I actually discussed with @estewart08 since AOMP is going for rpath by default for openmp so support rpath for host-asan(libclang_rt.asan.x86_64.so) library and for openmp-asan instrumented libraries. I believe we don't need this setting.

Then change the setting ofr RUNENV in the block of code about line 336 ish to this.

ifeq ($(SET_DEVICE_DEBUG),)
   RUNENV = $(ASAN_ENV) $(TKILL) $(CBL_ENV)
else
   RUNENV = env $(SET_DEVICE_DEBUG) $(ASAN_ENV) $(TKILL) $(CBL_ENV)
endif

Don't think we need this setting also because clang-driver and flang-legacy driver would handle for asan runtime library dependencies.

[ampandey-AMD]

Since these asan tests are supposed to fail at runtime, you need to check the pipestatus to filecheck for the return code.

Thanks @estewart08. The current revision had the required changes.

[ampandey-AMD]

!psdb

Hi @saipoorna, I think this PR to merge on branch aomp-dev is enough for smoke testing of ASan. I don't require to open a duplicate PR for aomp-epsdb. I checked downstream, the npsdb CI utilizes the aomp-dev branch for aomp smoke testing.

[ampandey-AMD]

Here is the output of AOMP_SANITIZER=1 ./check_smoke_asan.sh.

/work/ampandey/aomp-toolchain/aomp-src/aomp/test/smoke-asan /work/ampandey/aomp-toolchain/aomp-src/aomp/test/smoke-asan
Script Dir Name: smoke-asan
OMP_TARGET_OFFLOAD=MANDATORY

RUNNING ALL TESTS IN: /work/ampandey/aomp-toolchain/aomp-src/aomp/test/smoke-asan

rm -f hip-global-buffer-overflow hip-global-buffer-overflow.a llbin sbin obin *.i *.ii *.bc *.lk a.out-* *.ll *.s *.o *.log *.mod verify_output *.stb *.ilm *.cmod *.cmdx *.so hip-global-buffer-overflow_og11 make-log.txt TEST_STATUS FILECHECK_STATUS
/opt/rocm-6.1.0-staging/llvm/bin/clang++  -x hip -std=c++11 -L/opt/rocm-6.1.0-staging/llvm/../../lib/asan -lamdhip64 -Wl,-rpath,/opt/rocm-6.1.0-staging/llvm/../../lib/asan    -fopenmp --offload-arch=gfx90a:xnack+ -fsanitize=address -shared-libasan -g     -D__OFFLOAD_ARCH_gfx90a__ hip-global-buffer-overflow.cpp -o hip-global-buffer-overflow

rm -f hip-heap-buffer-overflow hip-heap-buffer-overflow.a llbin sbin obin *.i *.ii *.bc *.lk a.out-* *.ll *.s *.o *.log *.mod verify_output *.stb *.ilm *.cmod *.cmdx *.so hip-heap-buffer-overflow_og11 make-log.txt TEST_STATUS FILECHECK_STATUS
/opt/rocm-6.1.0-staging/llvm/bin/clang++  -x hip -std=c++11 -L/opt/rocm-6.1.0-staging/llvm/../../lib/asan -lamdhip64 -Wl,-rpath,/opt/rocm-6.1.0-staging/llvm/../../lib/asan    -fopenmp --offload-arch=gfx90a:xnack+ -fsanitize=address -shared-libasan -g     -D__OFFLOAD_ARCH_gfx90a__ hip-heap-buffer-overflow.cpp -o hip-heap-buffer-overflow

rm -f hip-use-after-free hip-use-after-free.a llbin sbin obin *.i *.ii *.bc *.lk a.out-* *.ll *.s *.o *.log *.mod verify_output *.stb *.ilm *.cmod *.cmdx *.so hip-use-after-free_og11 make-log.txt TEST_STATUS FILECHECK_STATUS
/opt/rocm-6.1.0-staging/llvm/bin/clang++  -x hip -std=c++11 -L/opt/rocm-6.1.0-staging/llvm/../../lib/asan -lamdhip64 -Wl,-rpath,/opt/rocm-6.1.0-staging/llvm/../../lib/asan    -fopenmp --offload-arch=gfx90a:xnack+ -fsanitize=address -shared-libasan -g     -D__OFFLOAD_ARCH_gfx90a__ hip-use-after-free.cpp -o hip-use-after-free

rm -f omp-global-buffer-overflow omp-global-buffer-overflow.a llbin sbin obin *.i *.ii *.bc *.lk a.out-* *.ll *.s *.o *.log *.mod verify_output *.stb *.ilm *.cmod *.cmdx *.so omp-global-buffer-overflow_og11 make-log.txt TEST_STATUS FILECHECK_STATUS
/opt/rocm-6.1.0-staging/llvm/bin/clang++  -O2    -fopenmp --offload-arch=gfx90a:xnack+ -fsanitize=address -shared-libasan -g      -D__OFFLOAD_ARCH_gfx90a__ omp-global-buffer-overflow.cpp -o omp-global-buffer-overflow

rm -f omp-heap-buffer-overflow omp-heap-buffer-overflow.a llbin sbin obin *.i *.ii *.bc *.lk a.out-* *.ll *.s *.o *.log *.mod verify_output *.stb *.ilm *.cmod *.cmdx *.so omp-heap-buffer-overflow_og11 make-log.txt TEST_STATUS FILECHECK_STATUS
/opt/rocm-6.1.0-staging/llvm/bin/clang++  -O2    -fopenmp --offload-arch=gfx90a:xnack+ -fsanitize=address -shared-libasan -g      -D__OFFLOAD_ARCH_gfx90a__ omp-heap-buffer-overflow.cpp -o omp-heap-buffer-overflow


************************************************************************************
                   A non-zero exit code means a failure occured.
Tests that need to be visually inspected: devices, stream
***********************************************************************************

-------------------- Results --------------------
Number of tests: 5

Passing tests: 5/5

[estewart08]

LGTM, I did have some comments on other asan related aomp patches though. This should be ok to land as it is isolated to the tests rather than the build.

[ampandey-AMD]

Hi Ethan,

I don't have anything more on this PR to be added, unless I write more tests. Can you please review again this PR.?

Thanks & Regards
Amit

[estewart08]

LGTM, asan enabled testing will be enabled with run_rocm_test.sh -a

[ampandey-AMD]

LGTM, asan enabled testing will be enabled with run_rocm_test.sh -a

Thanks. Merging this PR now.