/opt/deepvariant/bin/deeptrio/run_deeptrio: No such file or directory

[alanlamsiu]

I tried to run deeptrio using the suggested command:

singularity run --nv -B /usr/lib/locale/:/usr/lib/locale/
deepvariant_deeptrio-1.6.0rc2-gpu.sif
/opt/deepvariant/bin/deeptrio/run_deeptrio

However, there was an error showing that

/opt/nvidia/nvidia_entrypoint.sh: line 67: /opt/deepvariant/bin/run_deeptrio: No such file or directory

I shelled into the sif file and couldn't find /opt/deepvariant/bin/run_deeptrio. Is this file missing in container or shall I use another executable?

Thanks.

[pichuan]

Hi @alanlamsiu ,
Can you provide the full context? Like, where did your deepvariant_deeptrio-1.6.0rc2-gpu.sif came from?

You can also check out https://github.com/google/deepvariant/blob/r1.6/docs/deeptrio-quick-start.md#notes-on-singularity to see if that works for you.

[alanlamsiu]

Hi @pichuan,

I pulled the image using docker pull google/deepvariant:deeptrio-1.6.0-gpu, which got deepvariant_deeptrio-1.6.0rc2-gpu.sif.

Then I did singularity shell --nv -B /usr/lib/locale/:/usr/lib/locale/ /sc/arion/work/linx19/utilities/deepvariant_deeptrio-1.6.0rc2-gpu.sif and shelled into the container.

When I checked ls /opt/deepvariant/bin/, I saw the following files.

call_variants labeled_examples_to_vcf make_examples_somatic postprocess_variants run_deepsomatic.py runtime_by_region_vis.zip train
call_variants.zip labeled_examples_to_vcf.zip make_examples_somatic.zip postprocess_variants.zip run_deepvariant settings.sh train.zip
call_variants_slim make_examples multisample_make_examples run-prereq.sh run_deepvariant.py show_examples vcf_stats_report
call_variants_slim.zip make_examples.zip multisample_make_examples.zip run_deepsomatic runtime_by_region_vis show_examples.zip vcf_stats_report.zip

Do you think the run_deeptrio file is missing here?

Thanks.

[pichuan]

Hi @alanlamsiu

Can you try the steps in https://github.com/google/deepvariant/blob/r1.6/docs/deeptrio-quick-start.md#notes-on-singularity and see if that works for you?

Hi @pichuan,

I pulled the image using docker pull google/deepvariant:deeptrio-1.6.0-gpu, which got deepvariant_deeptrio-1.6.0rc2-gpu.sif.

I don't think I understand this part.
Can you share the exact commands you ran here?

[alanlamsiu]

HI @pichuan,

Please check this screenshot out and see if it is more clear.

What I did was I first shelled into the container. Then I checked the files in /opt/deepvariant/bin/ in the container because I tried to look for the run_deeptrio file, which could not be found when I followed steps in https://github.com/google/deepvariant/blob/r1.6/docs/deeptrio-quick-start.md#notes-on-singularity and gave the error message as in the title of this issue. Was it possible that I missed anything when running Deeptrio?

Thanks.

[pichuan]

Hi @alanlamsiu ,

If I understand correctly, your .sif file was previously built from our Docker version?

I'm going to walk through what I've tested so far. Maybe you can check which step is different from your experience.

---

Just to make sure I try it myself, here is what I did:

Get a GPU machine to test with:

gcloud compute instances create "${USER}-gpu" \
  --scopes "compute-rw,storage-full,cloud-platform" \
  --maintenance-policy "TERMINATE" \
  --accelerator=type=nvidia-tesla-p100,count=1 \
  --image-family "ubuntu-2004-lts" \
  --image-project "ubuntu-os-cloud" \
  --machine-type "n1-standard-16" \
  --boot-disk-size "300" \
  --zone "us-west1-b" \
  --min-cpu-platform "Intel Skylake"

I ssh to the machine gcloud compute ssh pichuan-gpu --zone us-west1-b

Because my machine doesn't have Nvidia driver installed, I used:

wget https://github.com/raw/google/deepvariant/r1.6/scripts/install_nvidia_docker.sh
sudo bash -x install_nvidia_docker.sh 

(the docker part is probably not necessary. I just need the driver)

And then because I want to test Singularity, I install that with:

wget https://github.com/raw/google/deepvariant/r1.6/scripts/install_singularity.sh
sudo bash -x install_singularity.sh 

Now, my machine has Singularity. I'll start following https://github.com/google/deepvariant/blob/r1.6/docs/deeptrio-quick-start.md#notes-on-singularity

I ran:

BIN_VERSION=1.6.0
singularity pull docker://google/deepvariant:deeptrio-"${BIN_VERSION}-gpu"

This created the file deepvariant_deeptrio-1.6.0-gpu.sif on my machine. I checked its size:

pichuan@pichuan-gpu:~$ ls -lh deepvariant_deeptrio-1.6.0-gpu.sif 
-rwxrwxr-x 1 pichuan pichuan 12G Dec  5 07:38 deepvariant_deeptrio-1.6.0-gpu.sif

( @alanlamsiu , This is one thing I'd like you to double check. If you're converting from the 1.6.0 version, I don't think you should see 1.6.0rc2 in your .sif filename. Which is why I asked what command you used to that get that .sif file. You might be pulling a previous, unofficial Docker file. If so, please remake your .sif file with 1.6.0. )

Because @alanlamsiu used the .sif file, I'll also do something similar:
So, then I ran:

singularity run --nv -B /usr/lib/locale/:/usr/lib/locale/ \
  deepvariant_deeptrio-1.6.0-gpu.sif \
  /opt/deepvariant/bin/deeptrio/run_deeptrio

The command above gave me:

==========
== CUDA ==
==========

CUDA Version 11.3.1

Container image Copyright (c) 2016-2023, NVIDIA CORPORATION & AFFILIATES. All rights reserved.

This container image and its contents are governed by the NVIDIA Deep Learning Container License.
By pulling and using the container, you accept the terms and conditions of this license:
https://developer.nvidia.com/ngc/nvidia-deep-learning-container-license

A copy of this license is made available in this container at /NGC-DL-CONTAINER-LICENSE for your convenience.

2023-12-05 07:43:20.303963: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 AVX512F FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2023-12-05 07:43:24.030774: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libcublas.so.12: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/local/nvidia/lib:/usr/local/nvidia/lib64:/.singularity.d/libs
2023-12-05 07:43:24.033082: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.
--model_type is required.
Pass --helpshort or --helpfull to see help on flags.

Which is expected because I didn't pass in more flags for run_deeptrio. The most relevant message is the ones at the bottom:

--model_type is required.
Pass --helpshort or --helpfull to see help on flags.

From here, I should be able to pass in the arguments from Quick Start and have it working. (If you want, I can continue the test)

---

@alanlamsiu ,
Now I got here, please check these two things on your side:

1. Can you confirm that your .sif is indeed made from 1.6.0? (Which is our latest version. Please don't use the rc2 one).
2. Can you try /opt/deepvariant/bin/deeptrio/run_deeptrio? You seem to be using a different path. I'm also not sure where that came from -- if we have any GitHub documentation that's inconsistent, please let me know and I can update.

[alanlamsiu]

Hi @pichuan ,

Thanks for checking it out. I later found out that I did use the rc2 version before. Now I have the one without rc2 as well.

Now I see that /opt/deepvariant/bin/deeptrio/run_deeptrio is in this container.

Thanks for pointing it out and I will use the 1.6.0-gpu version instead of 1.6.0rc2-gpu.