deepvariant 1.6.0 with singularity gpu support #761
Comments
|
Hi @melop , I'll be helping you with this, but I'm a bit behind on my tasks. I'll assign this myself and I'll try to take a look soon (hopefully by end of tomorrow!) |
|
Hi @melop , sorry that it took me a while. I have a question for you: When call_variants step is running, is your GPU being utilized? |
|
And @melop , the most recent time I tried to set up a GPU machine was using these steps: I don't think I've seen the Based on that error message I did a bit internet search. I wonder if this is relevant: https://stackoverflow.com/a/48715413 Specifically , try setting CUDA_VISIBLE_DEVICES to 0, by running |
|
Hi @melop , let me know if you have a chance to try and follow up. I'll close this because it has been 2 weeks. Please feel free to follow up and reopen if you have more questions. |
|
I get the same error. There appears to be Cuda 12 modules being used even though Cuda 11 is installed? Adding '--env LD_LIBRARY_PATH=/usr/local/cuda/lib64:/usr/local/nvidia/lib:/usr/local/lib/python3.8/dist-packages/nvidia/cublas/lib:/usr/local/lib/python3.8/dist-packages/nvidia/cuda_nvrtc/lib:/usr/local/lib/python3.8/dist-packages/nvidia/cuda_runtime/lib:/usr/local/lib/python3.8/dist-packages/nvidia/cudnn/lib' lets the image see all these libraries and stops the "Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libcublas.so.12: cannot open shared object file" error. But now during the call_variants step, I get a new error --> "2024-03-11 23:23:22.756430: E tensorflow/compiler/xla/stream_executor/cuda/cuda_dnn.cc:433] Could not create cudnn handle: CUDNN_STATUS_NOT_INITIALIZED Just doing '--env LD_LIBRARY_PATH=/usr/local/cuda/lib64:/usr/local/nvidia/lib:/usr/local/lib/python3.8/dist-packages/nvidia/cublas/lib' also seems to get past the "Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libcublas.so.12:" error, but now I get "could not retrieve CUDA device count: CUDA_ERROR_NOT_INITIALIZED: initialization error" during the call_variants step The 1.6.0-gpu image appears to have some mixed cuda 11 & cuda 12 module library conflict issues. Some of these issues may be suppressed if you have a sufficient nvidia driver for both cuda 11 and cuda 12 (I'm still waiting to test that though). |
Dear all. I am new to deepvariant. We are trying to use deepvariant on a HPC cluster with singularity.
We installed nvidia and cuda drivers through conda, and tested it with other python programs that used gpu with success.
I also managed to run the CPU version with deepvariant with singularity with success.
However when running deepvariant on a gpu node with the following command, deepvariant complained that certain libraries are not found which prevented it from using the GPU:
apptainer run --nv -B /public:/public,/public3:/public3,/public2:/public2,/fast3:/fast3 \ /public/software/deepvariants/1.6.0/gpuver/deepvariant_1.6.0-gpu.sif \ /opt/deepvariant/bin/run_deepvariant \ --model_type=WGS \ --ref=$REF \ --reads="/public2/courses/ec3121/shareddata/Pomacea_canaliculata/wgs/FSL10-M.bam" \ --regions "NC_037590.1:200,000-950,000" \ --output_vcf=${OUTPUT_DIR}/output.vcf.gz \ --output_gvcf=${OUTPUT_DIR}/output.g.vcf.gz \ --num_shards=2Error messages:
`==========
== CUDA ==
CUDA Version 11.3.1
Container image Copyright (c) 2016-2023, NVIDIA CORPORATION & AFFILIATES. All rights reserved.
This container image and its contents are governed by the NVIDIA Deep Learning Container License.
By pulling and using the container, you accept the terms and conditions of this license:
https://developer.nvidia.com/ngc/nvidia-deep-learning-container-license
A copy of this license is made available in this container at /NGC-DL-CONTAINER-LICENSE for your convenience.
WARNING: The NVIDIA Driver was not detected. GPU functionality will not be available.
Use the NVIDIA Container Toolkit to start this container with GPU support; see
https://docs.nvidia.com/datacenter/cloud-native/ .
2024-01-05 15:52:56.748367: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX2 AVX512F AVX512_VNNI FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2024-01-05 15:52:57.864310: I tensorflow/core/util/port.cc:104] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable
TF_ENABLE_ONEDNN_OPTS=0.2024-01-05 15:53:10.688853: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libcublas.so.12: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/local/nvidia/lib:/usr/local/nvidia/lib64:/.singularity.d/libs
2024-01-05 15:53:10.692890: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.
2024-01-05 15:53:26.990784: E tensorflow/compiler/xla/stream_executor/cuda/cuda_driver.cc:267] failed call to cuInit: CUDA_ERROR_NO_DEVICE: no CUDA-capable device is detected
I0105 15:53:27.004992 140619855705920 run_deepvariant.py:519] Re-using the directory for intermediate results in /public3/group_crf/home/cuirf/.tmp/tmp3vf8mpw9
***** Intermediate results will be written to /public3/group_crf/home/cuirf/.tmp/tmp3vf8mpw9 in docker. ****
***** Running the command:*****
time seq 0 1 | parallel -q --halt 2 --line-buffer /opt/deepvariant/bin/make_examples --mode calling --ref "/public2/courses/ec3121/shareddata/Pomacea_canaliculata/refgenome/GCF_003073045.1_ASM307304v1_genomic.fna" --reads "/public2/courses/ec3121/shareddata/Pomacea_canaliculata/wgs/FSL10-M.bam" --examples "/public3/group_crf/home/cuirf/.tmp/tmp3vf8mpw9/make_examples.tfrecord@2.gz" --channels "insert_size" --gvcf "/public3/group_crf/home/cuirf/.tmp/tmp3vf8mpw9/gvcf.tfrecord@2.gz" --regions "NC_037590.1:200,000-950,000" --task {}
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "C.UTF-8",
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to the standard locale ("C").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "C.UTF-8",
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to the standard locale ("C").
2024-01-05 15:53:39.096475: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libcublas.so.12: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/local/nvidia/lib:/usr/local/nvidia/lib64:/.singularity.d/libs
2024-01-05 15:53:39.096611: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.
2024-01-05 15:53:39.226747: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libcublas.so.12: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/local/nvidia/lib:/usr/local/nvidia/lib64:/.singularity.d/libs
2024-01-05 15:53:39.226871: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.
2024-01-05 15:53:49.941043: E tensorflow/compiler/xla/stream_executor/cuda/cuda_driver.cc:267] failed call to cuInit: CUDA_ERROR_NO_DEVICE: no CUDA-capable device is detected
I0105 15:53:49.987410 140173517489984 genomics_reader.py:222] Reading /public2/courses/ec3121/shareddata/Pomacea_canaliculata/wgs/FSL10-M.bam with NativeSamReader
W0105 15:53:49.988560 140173517489984 make_examples_core.py:344] No non-empty sample name found in the input reads. DeepVariant will use default as the sample name. You can also provide a sample name with the --sample_name argument.
I0105 15:53:50.021419 140173517489984 make_examples_core.py:301] Task 0/2: Preparing inputs
I0105 15:53:50.036767 140173517489984 genomics_reader.py:222] Reading /public2/courses/ec3121/shareddata/Pomacea_canaliculata/wgs/FSL10-M.bam with NativeSamReader
I0105 15:53:50.054040 140173517489984 make_examples_core.py:301] Task 0/2: Common contigs are ['NC_037590.1', 'NC_037591.1', 'NC_037592.1', 'NC_037593.1', 'NC_037594.1', 'NC_037595.1', 'NC_037596.1', 'NC_037597.1', 'NC_037598.1', 'NC_037599.1', 'NC_037600.1', 'NC_037601.1', 'NC_037602.1', 'NC_037603.1', 'NW_020229205.1', 'NW_020229206.1', 'NW_020229207.1', 'NW_020229208.1', 'NW_020229209.1', 'NW_020229210.1', 'NW_020229211.1', 'NW_020229212.1', 'NW_020229213.1', 'NC_024586.1']
I0105 15:53:50.067565 140173517489984 make_examples_core.py:301] Task 0/2: Starting from v0.9.0, --use_ref_for_cram is default to true. If you are using CRAM input, note that we will decode CRAM using the reference you passed in with --ref
2024-01-05 15:53:49.942446: E tensorflow/compiler/xla/stream_executor/cuda/cuda_driver.cc:267] failed call to cuInit: CUDA_ERROR_NO_DEVICE: no CUDA-capable device is detected
I0105 15:53:49.983960 140329169033024 genomics_reader.py:222] Reading /public2/courses/ec3121/shareddata/Pomacea_canaliculata/wgs/FSL10-M.bam with NativeSamReader
W0105 15:53:49.992453 140329169033024 make_examples_core.py:344] No non-empty sample name found in the input reads. DeepVariant will use default as the sample name. You can also provide a sample name with the --sample_name argument.
I0105 15:53:50.050559 140329169033024 make_examples_core.py:301] Task 1/2: Preparing inputs
I0105 15:53:50.080640 140329169033024 genomics_reader.py:222] Reading /public2/courses/ec3121/shareddata/Pomacea_canaliculata/wgs/FSL10-M.bam with NativeSamReader
I0105 15:53:50.128940 140329169033024 make_examples_core.py:301] Task 1/2: Common contigs are ['NC_037590.1', 'NC_037591.1', 'NC_037592.1', 'NC_037593.1', 'NC_037594.1', 'NC_037595.1', 'NC_037596.1', 'NC_037597.1', 'NC_037598.1', 'NC_037599.1', 'NC_037600.1', 'NC_037601.1', 'NC_037602.1', 'NC_037603.1', 'NW_020229205.1', 'NW_020229206.1', 'NW_020229207.1', 'NW_020229208.1', 'NW_020229209.1', 'NW_020229210.1', 'NW_020229211.1', 'NW_020229212.1', 'NW_020229213.1', 'NC_024586.1']
I0105 15:53:50.174469 140329169033024 make_examples_core.py:301] Task 1/2: Starting from v0.9.0, --use_ref_for_cram is default to true. If you are using CRAM input, note that we will decode CRAM using the reference you passed in with --ref
I0105 15:53:50.325648 140173517489984 genomics_reader.py:222] Reading /public2/courses/ec3121/shareddata/Pomacea_canaliculata/wgs/FSL10-M.bam with NativeSamReader
I0105 15:53:50.668989 140173517489984 genomics_reader.py:222] Reading /public2/courses/ec3121/shareddata/Pomacea_canaliculata/wgs/FSL10-M.bam with NativeSamReader
I0105 15:53:50.671522 140173517489984 make_examples_core.py:301] Task 0/2: Writing gvcf records to /public3/group_crf/home/cuirf/.tmp/tmp3vf8mpw9/gvcf.tfrecord-00000-of-00002.gz
I0105 15:53:50.685908 140173517489984 make_examples_core.py:301] Task 0/2: Writing examples to /public3/group_crf/home/cuirf/.tmp/tmp3vf8mpw9/make_examples.tfrecord-00000-of-00002.gz
I0105 15:53:50.686180 140173517489984 make_examples_core.py:301] Task 0/2: Overhead for preparing inputs: 0 seconds
I0105 15:53:50.897492 140173517489984 make_examples_core.py:301] Task 0/2: 15 candidates (15 examples) [0.21s elapsed]
I0105 15:53:50.507650 140329169033024 genomics_reader.py:222] Reading /public2/courses/ec3121/shareddata/Pomacea_canaliculata/wgs/FSL10-M.bam with NativeSamReader
I0105 15:53:50.743297 140329169033024 genomics_reader.py:222] Reading /public2/courses/ec3121/shareddata/Pomacea_canaliculata/wgs/FSL10-M.bam with NativeSamReader
I0105 15:53:50.748420 140329169033024 make_examples_core.py:301] Task 1/2: Writing gvcf records to /public3/group_crf/home/cuirf/.tmp/tmp3vf8mpw9/gvcf.tfrecord-00001-of-00002.gz
I0105 15:53:50.752661 140329169033024 make_examples_core.py:301] Task 1/2: Writing examples to /public3/group_crf/home/cuirf/.tmp/tmp3vf8mpw9/make_examples.tfrecord-00001-of-00002.gz
I0105 15:53:50.753379 140329169033024 make_examples_core.py:301] Task 1/2: Overhead for preparing inputs: 0 seconds
I0105 15:53:50.999809 140329169033024 make_examples_core.py:301] Task 1/2: 6 candidates (8 examples) [0.25s elapsed]
I0105 15:54:39.302088 140329169033024 make_examples_core.py:301] Task 1/2: 2003 candidates (2147 examples) [48.30s elapsed]
I0105 15:54:41.885760 140173517489984 make_examples_core.py:301] Task 0/2: 2000 candidates (2124 examples) [50.99s elapsed]
I0105 15:55:13.311895 140173517489984 make_examples_core.py:301] Task 0/2: Writing example info to /public3/group_crf/home/cuirf/.tmp/tmp3vf8mpw9/make_examples.tfrecord-00000-of-00002.gz.example_info.json
I0105 15:55:13.312177 140173517489984 make_examples_core.py:2958] example_shape = [100, 221, 7]
I0105 15:55:13.312256 140173517489984 make_examples_core.py:2959] example_channels = [1, 2, 3, 4, 5, 6, 19]
I0105 15:55:13.315201 140173517489984 make_examples_core.py:301] Task 0/2: Found 3287 candidate variants
I0105 15:55:13.315361 140173517489984 make_examples_core.py:301] Task 0/2: Created 3569 examples
I0105 15:55:21.254777 140329169033024 make_examples_core.py:301] Task 1/2: Writing example info to /public3/group_crf/home/cuirf/.tmp/tmp3vf8mpw9/make_examples.tfrecord-00001-of-00002.gz.example_info.json
I0105 15:55:21.255679 140329169033024 make_examples_core.py:2958] example_shape = [100, 221, 7]
I0105 15:55:21.255904 140329169033024 make_examples_core.py:2959] example_channels = [1, 2, 3, 4, 5, 6, 19]
I0105 15:55:21.262568 140329169033024 make_examples_core.py:301] Task 1/2: Found 3672 candidate variants
I0105 15:55:21.263317 140329169033024 make_examples_core.py:301] Task 1/2: Created 3944 examples
real 1m56.796s
user 3m3.813s
sys 0m4.710s
***** Running the command:*****
time /opt/deepvariant/bin/call_variants --outfile "/public3/group_crf/home/cuirf/.tmp/tmp3vf8mpw9/call_variants_output.tfrecord.gz" --examples "/public3/group_crf/home/cuirf/.tmp/tmp3vf8mpw9/make_examples.tfrecord@2.gz" --checkpoint "/opt/models/wgs"
2024-01-05 15:55:31.140705: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libcublas.so.12: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/local/nvidia/lib:/usr/local/nvidia/lib64:/.singularity.d/libs
2024-01-05 15:55:31.140953: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.
/usr/local/lib/python3.8/dist-packages/tensorflow_addons/utils/tfa_eol_msg.py:23: UserWarning:
TensorFlow Addons (TFA) has ended development and introduction of new features.
TFA has entered a minimal maintenance and release mode until a planned end of life in May 2024.
Please modify downstream libraries to take dependencies from other repositories in our TensorFlow community (e.g. Keras, Keras-CV, and Keras-NLP).
For more information see: tensorflow/addons#2807
warnings.warn(
2024-01-05 15:55:38.664328: E tensorflow/compiler/xla/stream_executor/cuda/cuda_driver.cc:267] failed call to cuInit: CUDA_ERROR_NO_DEVICE: no CUDA-capable device is detected
I0105 15:55:38.709242 140372734228288 call_variants.py:471] Total 1 writing processes started.
I0105 15:55:38.765925 140372734228288 dv_utils.py:365] From /public3/group_crf/home/cuirf/.tmp/tmp3vf8mpw9/make_examples.tfrecord-00000-of-00002.gz.example_info.json: Shape of input examples: [100, 221, 7], Channels of input examples: [1, 2, 3, 4, 5, 6, 19].
I0105 15:55:38.766286 140372734228288 call_variants.py:506] Shape of input examples: [100, 221, 7]
I0105 15:55:38.768594 140372734228288 call_variants.py:510] Use saved model: True
I0105 15:56:02.220975 140372734228288 dv_utils.py:365] From /opt/models/wgs/example_info.json: Shape of input examples: [100, 221, 7], Channels of input examples: [1, 2, 3, 4, 5, 6, 19].
I0105 15:56:02.221645 140372734228288 dv_utils.py:365] From /public3/group_crf/home/cuirf/.tmp/tmp3vf8mpw9/make_examples.tfrecord-00000-of-00002.gz.example_info.json: Shape of input examples: [100, 221, 7], Channels of input examples: [1, 2, 3, 4, 5, 6, 19].
I0105 15:56:51.296850 140372734228288 call_variants.py:583] Predicted 1024 examples in 1 batches [4.670 sec per 100].
I0105 16:00:45.139408 140372734228288 call_variants.py:623] Complete: call_variants.
real 5m27.431s
user 6m58.490s
sys 0m19.033s
***** Running the command:*****
time /opt/deepvariant/bin/postprocess_variants --ref "/public2/courses/ec3121/shareddata/Pomacea_canaliculata/refgenome/GCF_003073045.1_ASM307304v1_genomic.fna" --infile "/public3/group_crf/home/cuirf/.tmp/tmp3vf8mpw9/call_variants_output.tfrecord.gz" --outfile "./outputgpu/output.vcf.gz" --cpus "2" --gvcf_outfile "./outputgpu/output.g.vcf.gz" --nonvariant_site_tfrecord_path "/public3/group_crf/home/cuirf/.tmp/tmp3vf8mpw9/gvcf.tfrecord@2.gz"
2024-01-05 16:00:59.661436: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libcublas.so.12: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/local/nvidia/lib:/usr/local/nvidia/lib64:/.singularity.d/libs
2024-01-05 16:00:59.661893: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.
2024-01-05 16:01:06.236791: E tensorflow/compiler/xla/stream_executor/cuda/cuda_driver.cc:267] failed call to cuInit: CUDA_ERROR_NO_DEVICE: no CUDA-capable device is detected
I0105 16:01:06.304423 140416700553024 postprocess_variants.py:1211] Using sample name from call_variants output. Sample name: default
I0105 16:01:06.676597 140416700553024 postprocess_variants.py:1313] CVO sorting took 0.006136405467987061 minutes
I0105 16:01:06.677379 140416700553024 postprocess_variants.py:1316] Transforming call_variants_output to variants.
I0105 16:01:06.677495 140416700553024 postprocess_variants.py:1318] Using 2 CPUs for parallelization of variant transformation.
I0105 16:01:06.808352 140416700553024 postprocess_variants.py:1211] Using sample name from call_variants output. Sample name: default
I0105 16:01:08.209710 140416700553024 postprocess_variants.py:1386] Processing variants (and writing to temporary file) took 0.01743464469909668 minutes
I0105 16:01:10.258949 140416700553024 postprocess_variants.py:1407] Finished writing VCF and gVCF in 0.03414338032404582 minutes.
real 0m21.740s
user 0m13.473s
sys 0m2.305s
***** Running the command:*****
time /opt/deepvariant/bin/vcf_stats_report --input_vcf "./outputgpu/output.vcf.gz" --outfile_base "./outputgpu/output"
2024-01-05 16:01:21.188421: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libcublas.so.12: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/local/nvidia/lib:/usr/local/nvidia/lib64:/.singularity.d/libs
2024-01-05 16:01:21.188700: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.
2024-01-05 16:01:28.513759: E tensorflow/compiler/xla/stream_executor/cuda/cuda_driver.cc:267] failed call to cuInit: CUDA_ERROR_NO_DEVICE: no CUDA-capable device is detected
I0105 16:01:28.547411 140591583876928 genomics_reader.py:222] Reading ./outputgpu/output.vcf.gz with NativeVcfReader
real 0m18.513s
user 0m11.281s
sys 0m1.577s
`
The text was updated successfully, but these errors were encountered: