apex installation failures

[momo1986]

command "pip install -v --disable-pip-version-check --no-cache-dir --global-option="--cpp_ext" --global-option="--cuda_ext" ./"

error log:
"DEPRECATION: --build-option and --global-option are deprecated. pip 24.0 will enforce this behaviour change. A possible replacement is to use --config-settings. Discussion can be found at pypa/pip#11859
WARNING: Implying --no-binary=:all: due to the presence of --build-option / --global-option.
Processing /root/junyan/apex
Running command python setup.py egg_info

torch.version = 1.9.0+cu111

running egg_info
creating /tmp/pip-pip-egg-info-eip1b_ig/apex.egg-info
writing /tmp/pip-pip-egg-info-eip1b_ig/apex.egg-info/PKG-INFO
writing dependency_links to /tmp/pip-pip-egg-info-eip1b_ig/apex.egg-info/dependency_links.txt
writing top-level names to /tmp/pip-pip-egg-info-eip1b_ig/apex.egg-info/top_level.txt
writing manifest file '/tmp/pip-pip-egg-info-eip1b_ig/apex.egg-info/SOURCES.txt'
reading manifest file '/tmp/pip-pip-egg-info-eip1b_ig/apex.egg-info/SOURCES.txt'
adding license file 'LICENSE'
writing manifest file '/tmp/pip-pip-egg-info-eip1b_ig/apex.egg-info/SOURCES.txt'
/root/junyan/apex/setup.py:111: UserWarning: Option --pyprof not specified. Not installing PyProf dependencies!
warnings.warn("Option --pyprof not specified. Not installing PyProf dependencies!")
Preparing metadata (setup.py) ... done
Building wheels for collected packages: apex
Running command python setup.py bdist_wheel

torch.version = 1.9.0+cu111

/root/junyan/apex/setup.py:111: UserWarning: Option --pyprof not specified. Not installing PyProf dependencies!
warnings.warn("Option --pyprof not specified. Not installing PyProf dependencies!")

Compiling cuda extensions with
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2020 NVIDIA Corporation
Built on Tue_Sep_15_19:10:02_PDT_2020
Cuda compilation tools, release 11.1, V11.1.74
Build cuda_11.1.TC455_06.29069683_0
from /usr/local/cuda/bin

running bdist_wheel
running build
running build_py
creating build/lib.linux-x86_64-cpython-37
creating build/lib.linux-x86_64-cpython-37/apex
copying apex/init.py -> build/lib.linux-x86_64-cpython-37/apex
copying apex/_autocast_utils.py -> build/lib.linux-x86_64-cpython-37/apex
creating build/lib.linux-x86_64-cpython-37/apex/RNN
copying apex/RNN/init.py -> build/lib.linux-x86_64-cpython-37/apex/RNN
copying apex/RNN/RNNBackend.py -> build/lib.linux-x86_64-cpython-37/apex/RNN
copying apex/RNN/cells.py -> build/lib.linux-x86_64-cpython-37/apex/RNN
copying apex/RNN/models.py -> build/lib.linux-x86_64-cpython-37/apex/RNN
creating build/lib.linux-x86_64-cpython-37/apex/amp
copying apex/amp/init.py -> build/lib.linux-x86_64-cpython-37/apex/amp
copying apex/amp/version.py -> build/lib.linux-x86_64-cpython-37/apex/amp
copying apex/amp/_process_optimizer.py -> build/lib.linux-x86_64-cpython-37/apex/amp
copying apex/amp/opt.py -> build/lib.linux-x86_64-cpython-37/apex/amp
copying apex/amp/rnn_compat.py -> build/lib.linux-x86_64-cpython-37/apex/amp
copying apex/amp/scaler.py -> build/lib.linux-x86_64-cpython-37/apex/amp
copying apex/amp/utils.py -> build/lib.linux-x86_64-cpython-37/apex/amp
copying apex/amp/wrap.py -> build/lib.linux-x86_64-cpython-37/apex/amp
copying apex/amp/amp.py -> build/lib.linux-x86_64-cpython-37/apex/amp
copying apex/amp/frontend.py -> build/lib.linux-x86_64-cpython-37/apex/amp
copying apex/amp/_amp_state.py -> build/lib.linux-x86_64-cpython-37/apex/amp
copying apex/amp/_initialize.py -> build/lib.linux-x86_64-cpython-37/apex/amp
copying apex/amp/compat.py -> build/lib.linux-x86_64-cpython-37/apex/amp
copying apex/amp/handle.py -> build/lib.linux-x86_64-cpython-37/apex/amp
creating build/lib.linux-x86_64-cpython-37/apex/contrib
copying apex/contrib/init.py -> build/lib.linux-x86_64-cpython-37/apex/contrib
creating build/lib.linux-x86_64-cpython-37/apex/fp16_utils
copying apex/fp16_utils/init.py -> build/lib.linux-x86_64-cpython-37/apex/fp16_utils
copying apex/fp16_utils/fp16_optimizer.py -> build/lib.linux-x86_64-cpython-37/apex/fp16_utils
copying apex/fp16_utils/fp16util.py -> build/lib.linux-x86_64-cpython-37/apex/fp16_utils
copying apex/fp16_utils/loss_scaler.py -> build/lib.linux-x86_64-cpython-37/apex/fp16_utils
creating build/lib.linux-x86_64-cpython-37/apex/multi_tensor_apply
copying apex/multi_tensor_apply/init.py -> build/lib.linux-x86_64-cpython-37/apex/multi_tensor_apply
copying apex/multi_tensor_apply/multi_tensor_apply.py -> build/lib.linux-x86_64-cpython-37/apex/multi_tensor_apply
creating build/lib.linux-x86_64-cpython-37/apex/normalization
copying apex/normalization/init.py -> build/lib.linux-x86_64-cpython-37/apex/normalization
copying apex/normalization/fused_layer_norm.py -> build/lib.linux-x86_64-cpython-37/apex/normalization
copying apex/normalization/instance_norm.py -> build/lib.linux-x86_64-cpython-37/apex/normalization
creating build/lib.linux-x86_64-cpython-37/apex/optimizers
copying apex/optimizers/init.py -> build/lib.linux-x86_64-cpython-37/apex/optimizers
copying apex/optimizers/fused_adagrad.py -> build/lib.linux-x86_64-cpython-37/apex/optimizers
copying apex/optimizers/fused_adam.py -> build/lib.linux-x86_64-cpython-37/apex/optimizers
copying apex/optimizers/fused_lamb.py -> build/lib.linux-x86_64-cpython-37/apex/optimizers
copying apex/optimizers/fused_mixed_precision_lamb.py -> build/lib.linux-x86_64-cpython-37/apex/optimizers
copying apex/optimizers/fused_novograd.py -> build/lib.linux-x86_64-cpython-37/apex/optimizers
copying apex/optimizers/fused_sgd.py -> build/lib.linux-x86_64-cpython-37/apex/optimizers
creating build/lib.linux-x86_64-cpython-37/apex/parallel
copying apex/parallel/multiproc.py -> build/lib.linux-x86_64-cpython-37/apex/parallel
copying apex/parallel/optimized_sync_batchnorm.py -> build/lib.linux-x86_64-cpython-37/apex/parallel
copying apex/parallel/sync_batchnorm_kernel.py -> build/lib.linux-x86_64-cpython-37/apex/parallel
copying apex/parallel/init.py -> build/lib.linux-x86_64-cpython-37/apex/parallel
copying apex/parallel/distributed.py -> build/lib.linux-x86_64-cpython-37/apex/parallel
copying apex/parallel/sync_batchnorm.py -> build/lib.linux-x86_64-cpython-37/apex/parallel
copying apex/parallel/LARC.py -> build/lib.linux-x86_64-cpython-37/apex/parallel
copying apex/parallel/optimized_sync_batchnorm_kernel.py -> build/lib.linux-x86_64-cpython-37/apex/parallel
creating build/lib.linux-x86_64-cpython-37/apex/pyprof
copying apex/pyprof/init.py -> build/lib.linux-x86_64-cpython-37/apex/pyprof
creating build/lib.linux-x86_64-cpython-37/apex/reparameterization
copying apex/reparameterization/reparameterization.py -> build/lib.linux-x86_64-cpython-37/apex/reparameterization
copying apex/reparameterization/init.py -> build/lib.linux-x86_64-cpython-37/apex/reparameterization
copying apex/reparameterization/weight_norm.py -> build/lib.linux-x86_64-cpython-37/apex/reparameterization
creating build/lib.linux-x86_64-cpython-37/apex/fused_dense
copying apex/fused_dense/init.py -> build/lib.linux-x86_64-cpython-37/apex/fused_dense
copying apex/fused_dense/fused_dense.py -> build/lib.linux-x86_64-cpython-37/apex/fused_dense
creating build/lib.linux-x86_64-cpython-37/apex/mlp
copying apex/mlp/init.py -> build/lib.linux-x86_64-cpython-37/apex/mlp
copying apex/mlp/mlp.py -> build/lib.linux-x86_64-cpython-37/apex/mlp
creating build/lib.linux-x86_64-cpython-37/apex/transformer
copying apex/transformer/init.py -> build/lib.linux-x86_64-cpython-37/apex/transformer
copying apex/transformer/enums.py -> build/lib.linux-x86_64-cpython-37/apex/transformer
copying apex/transformer/log_util.py -> build/lib.linux-x86_64-cpython-37/apex/transformer
copying apex/transformer/microbatches.py -> build/lib.linux-x86_64-cpython-37/apex/transformer
copying apex/transformer/parallel_state.py -> build/lib.linux-x86_64-cpython-37/apex/transformer
copying apex/transformer/utils.py -> build/lib.linux-x86_64-cpython-37/apex/transformer
creating build/lib.linux-x86_64-cpython-37/apex/amp/lists
copying apex/amp/lists/init.py -> build/lib.linux-x86_64-cpython-37/apex/amp/lists
copying apex/amp/lists/functional_overrides.py -> build/lib.linux-x86_64-cpython-37/apex/amp/lists
copying apex/amp/lists/torch_overrides.py -> build/lib.linux-x86_64-cpython-37/apex/amp/lists
copying apex/amp/lists/tensor_overrides.py -> build/lib.linux-x86_64-cpython-37/apex/amp/lists
creating build/lib.linux-x86_64-cpython-37/apex/contrib/groupbn
copying apex/contrib/groupbn/init.py -> build/lib.linux-x86_64-cpython-37/apex/contrib/groupbn
copying apex/contrib/groupbn/batch_norm.py -> build/lib.linux-x86_64-cpython-37/apex/contrib/groupbn
creating build/lib.linux-x86_64-cpython-37/apex/contrib/multihead_attn
copying apex/contrib/multihead_attn/init.py -> build/lib.linux-x86_64-cpython-37/apex/contrib/multihead_attn
copying apex/contrib/multihead_attn/encdec_multihead_attn.py -> build/lib.linux-x86_64-cpython-37/apex/contrib/multihead_attn
copying apex/contrib/multihead_attn/encdec_multihead_attn_func.py -> build/lib.linux-x86_64-cpython-37/apex/contrib/multihead_attn
copying apex/contrib/multihead_attn/fast_encdec_multihead_attn_func.py -> build/lib.linux-x86_64-cpython-37/apex/contrib/multihead_attn
copying apex/contrib/multihead_attn/fast_encdec_multihead_attn_norm_add_func.py -> build/lib.linux-x86_64-cpython-37/apex/contrib/multihead_attn
copying apex/contrib/multihead_attn/fast_self_multihead_attn_func.py -> build/lib.linux-x86_64-cpython-37/apex/contrib/multihead_attn
copying apex/contrib/multihead_attn/fast_self_multihead_attn_norm_add_func.py -> build/lib.linux-x86_64-cpython-37/apex/contrib/multihead_attn
copying apex/contrib/multihead_attn/mask_softmax_dropout_func.py -> build/lib.linux-x86_64-cpython-37/apex/contrib/multihead_attn
copying apex/contrib/multihead_attn/self_multihead_attn.py -> build/lib.linux-x86_64-cpython-37/apex/contrib/multihead_attn
copying apex/contrib/multihead_attn/self_multihead_attn_func.py -> build/lib.linux-x86_64-cpython-37/apex/contrib/multihead_attn
creating build/lib.linux-x86_64-cpython-37/apex/contrib/optimizers
copying apex/contrib/optimizers/init.py -> build/lib.linux-x86_64-cpython-37/apex/contrib/optimizers
copying apex/contrib/optimizers/fp16_optimizer.py -> build/lib.linux-x86_64-cpython-37/apex/contrib/optimizers
copying apex/contrib/optimizers/fused_sgd.py -> build/lib.linux-x86_64-cpython-37/apex/contrib/optimizers
copying apex/contrib/optimizers/distributed_fused_adam.py -> build/lib.linux-x86_64-cpython-37/apex/contrib/optimizers
copying apex/contrib/optimizers/distributed_fused_adam_v2.py -> build/lib.linux-x86_64-cpython-37/apex/contrib/optimizers
copying apex/contrib/optimizers/distributed_fused_adam_v3.py -> build/lib.linux-x86_64-cpython-37/apex/contrib/optimizers
copying apex/contrib/optimizers/distributed_fused_lamb.py -> build/lib.linux-x86_64-cpython-37/apex/contrib/optimizers
copying apex/contrib/optimizers/fused_adam.py -> build/lib.linux-x86_64-cpython-37/apex/contrib/optimizers
copying apex/contrib/optimizers/fused_lamb.py -> build/lib.linux-x86_64-cpython-37/apex/contrib/optimizers
creating build/lib.linux-x86_64-cpython-37/apex/contrib/xentropy
copying apex/contrib/xentropy/init.py -> build/lib.linux-x86_64-cpython-37/apex/contrib/xentropy
copying apex/contrib/xentropy/softmax_xentropy.py -> build/lib.linux-x86_64-cpython-37/apex/contrib/xentropy
creating build/lib.linux-x86_64-cpython-37/apex/contrib/bottleneck
copying apex/contrib/bottleneck/init.py -> build/lib.linux-x86_64-cpython-37/apex/contrib/bottleneck
copying apex/contrib/bottleneck/bottleneck.py -> build/lib.linux-x86_64-cpython-37/apex/contrib/bottleneck
copying apex/contrib/bottleneck/bottleneck_module_test.py -> build/lib.linux-x86_64-cpython-37/apex/contrib/bottleneck
copying apex/contrib/bottleneck/halo_exchangers.py -> build/lib.linux-x86_64-cpython-37/apex/contrib/bottleneck
copying apex/contrib/bottleneck/test.py -> build/lib.linux-x86_64-cpython-37/apex/contrib/bottleneck
creating build/lib.linux-x86_64-cpython-37/apex/contrib/conv_bias_relu
copying apex/contrib/conv_bias_relu/init.py -> build/lib.linux-x86_64-cpython-37/apex/contrib/conv_bias_relu
copying apex/contrib/conv_bias_relu/conv_bias_relu.py -> build/lib.linux-x86_64-cpython-37/apex/contrib/conv_bias_relu
creating build/lib.linux-x86_64-cpython-37/apex/contrib/fmha
copying apex/contrib/fmha/init.py -> build/lib.linux-x86_64-cpython-37/apex/contrib/fmha
copying apex/contrib/fmha/fmha.py -> build/lib.linux-x86_64-cpython-37/apex/contrib/fmha
creating build/lib.linux-x86_64-cpython-37/apex/contrib/focal_loss
copying apex/contrib/focal_loss/init.py -> build/lib.linux-x86_64-cpython-37/apex/contrib/focal_loss
copying apex/contrib/focal_loss/focal_loss.py -> build/lib.linux-x86_64-cpython-37/apex/contrib/focal_loss
creating build/lib.linux-x86_64-cpython-37/apex/contrib/layer_norm
copying apex/contrib/layer_norm/init.py -> build/lib.linux-x86_64-cpython-37/apex/contrib/layer_norm
copying apex/contrib/layer_norm/layer_norm.py -> build/lib.linux-x86_64-cpython-37/apex/contrib/layer_norm
creating build/lib.linux-x86_64-cpython-37/apex/contrib/peer_memory
copying apex/contrib/peer_memory/init.py -> build/lib.linux-x86_64-cpython-37/apex/contrib/peer_memory
copying apex/contrib/peer_memory/peer_halo_exchange_module_tests.py -> build/lib.linux-x86_64-cpython-37/apex/contrib/peer_memory
copying apex/contrib/peer_memory/peer_halo_exchanger_1d.py -> build/lib.linux-x86_64-cpython-37/apex/contrib/peer_memory
copying apex/contrib/peer_memory/peer_memory.py -> build/lib.linux-x86_64-cpython-37/apex/contrib/peer_memory
creating build/lib.linux-x86_64-cpython-37/apex/contrib/sparsity
copying apex/contrib/sparsity/init.py -> build/lib.linux-x86_64-cpython-37/apex/contrib/sparsity
copying apex/contrib/sparsity/asp.py -> build/lib.linux-x86_64-cpython-37/apex/contrib/sparsity
copying apex/contrib/sparsity/permutation_lib.py -> build/lib.linux-x86_64-cpython-37/apex/contrib/sparsity
copying apex/contrib/sparsity/sparse_masklib.py -> build/lib.linux-x86_64-cpython-37/apex/contrib/sparsity
creating build/lib.linux-x86_64-cpython-37/apex/contrib/transducer
copying apex/contrib/transducer/init.py -> build/lib.linux-x86_64-cpython-37/apex/contrib/transducer
copying apex/contrib/transducer/transducer.py -> build/lib.linux-x86_64-cpython-37/apex/contrib/transducer
creating build/lib.linux-x86_64-cpython-37/apex/contrib/sparsity/permutation_search_kernels
copying apex/contrib/sparsity/permutation_search_kernels/init.py -> build/lib.linux-x86_64-cpython-37/apex/contrib/sparsity/permutation_search_kernels
copying apex/contrib/sparsity/permutation_search_kernels/call_permutation_search_kernels.py -> build/lib.linux-x86_64-cpython-37/apex/contrib/sparsity/permutation_search_kernels
copying apex/contrib/sparsity/permutation_search_kernels/exhaustive_search.py -> build/lib.linux-x86_64-cpython-37/apex/contrib/sparsity/permutation_search_kernels
copying apex/contrib/sparsity/permutation_search_kernels/permutation_utilities.py -> build/lib.linux-x86_64-cpython-37/apex/contrib/sparsity/permutation_search_kernels
creating build/lib.linux-x86_64-cpython-37/apex/pyprof/nvtx
copying apex/pyprof/nvtx/init.py -> build/lib.linux-x86_64-cpython-37/apex/pyprof/nvtx
copying apex/pyprof/nvtx/nvmarker.py -> build/lib.linux-x86_64-cpython-37/apex/pyprof/nvtx
creating build/lib.linux-x86_64-cpython-37/apex/pyprof/parse
copying apex/pyprof/parse/init.py -> build/lib.linux-x86_64-cpython-37/apex/pyprof/parse
copying apex/pyprof/parse/main.py -> build/lib.linux-x86_64-cpython-37/apex/pyprof/parse
copying apex/pyprof/parse/db.py -> build/lib.linux-x86_64-cpython-37/apex/pyprof/parse
copying apex/pyprof/parse/kernel.py -> build/lib.linux-x86_64-cpython-37/apex/pyprof/parse
copying apex/pyprof/parse/nvvp.py -> build/lib.linux-x86_64-cpython-37/apex/pyprof/parse
copying apex/pyprof/parse/parse.py -> build/lib.linux-x86_64-cpython-37/apex/pyprof/parse
creating build/lib.linux-x86_64-cpython-37/apex/pyprof/prof
copying apex/pyprof/prof/main.py -> build/lib.linux-x86_64-cpython-37/apex/pyprof/prof
copying apex/pyprof/prof/activation.py -> build/lib.linux-x86_64-cpython-37/apex/pyprof/prof
copying apex/pyprof/prof/base.py -> build/lib.linux-x86_64-cpython-37/apex/pyprof/prof
copying apex/pyprof/prof/blas.py -> build/lib.linux-x86_64-cpython-37/apex/pyprof/prof
copying apex/pyprof/prof/conv.py -> build/lib.linux-x86_64-cpython-37/apex/pyprof/prof
copying apex/pyprof/prof/convert.py -> build/lib.linux-x86_64-cpython-37/apex/pyprof/prof
copying apex/pyprof/prof/data.py -> build/lib.linux-x86_64-cpython-37/apex/pyprof/prof
copying apex/pyprof/prof/dropout.py -> build/lib.linux-x86_64-cpython-37/apex/pyprof/prof
copying apex/pyprof/prof/embedding.py -> build/lib.linux-x86_64-cpython-37/apex/pyprof/prof
copying apex/pyprof/prof/index_slice_join_mutate.py -> build/lib.linux-x86_64-cpython-37/apex/pyprof/prof
copying apex/pyprof/prof/linear.py -> build/lib.linux-x86_64-cpython-37/apex/pyprof/prof
copying apex/pyprof/prof/loss.py -> build/lib.linux-x86_64-cpython-37/apex/pyprof/prof
copying apex/pyprof/prof/misc.py -> build/lib.linux-x86_64-cpython-37/apex/pyprof/prof
copying apex/pyprof/prof/normalization.py -> build/lib.linux-x86_64-cpython-37/apex/pyprof/prof
copying apex/pyprof/prof/optim.py -> build/lib.linux-x86_64-cpython-37/apex/pyprof/prof
copying apex/pyprof/prof/output.py -> build/lib.linux-x86_64-cpython-37/apex/pyprof/prof
copying apex/pyprof/prof/pointwise.py -> build/lib.linux-x86_64-cpython-37/apex/pyprof/prof
copying apex/pyprof/prof/pooling.py -> build/lib.linux-x86_64-cpython-37/apex/pyprof/prof
copying apex/pyprof/prof/prof.py -> build/lib.linux-x86_64-cpython-37/apex/pyprof/prof
copying apex/pyprof/prof/randomSample.py -> build/lib.linux-x86_64-cpython-37/apex/pyprof/prof
copying apex/pyprof/prof/recurrentCell.py -> build/lib.linux-x86_64-cpython-37/apex/pyprof/prof
copying apex/pyprof/prof/reduction.py -> build/lib.linux-x86_64-cpython-37/apex/pyprof/prof
copying apex/pyprof/prof/softmax.py -> build/lib.linux-x86_64-cpython-37/apex/pyprof/prof
copying apex/pyprof/prof/usage.py -> build/lib.linux-x86_64-cpython-37/apex/pyprof/prof
copying apex/pyprof/prof/init.py -> build/lib.linux-x86_64-cpython-37/apex/pyprof/prof
copying apex/pyprof/prof/utility.py -> build/lib.linux-x86_64-cpython-37/apex/pyprof/prof
creating build/lib.linux-x86_64-cpython-37/apex/transformer/_data
copying apex/transformer/_data/init.py -> build/lib.linux-x86_64-cpython-37/apex/transformer/_data
copying apex/transformer/_data/_batchsampler.py -> build/lib.linux-x86_64-cpython-37/apex/transformer/_data
creating build/lib.linux-x86_64-cpython-37/apex/transformer/amp
copying apex/transformer/amp/init.py -> build/lib.linux-x86_64-cpython-37/apex/transformer/amp
copying apex/transformer/amp/grad_scaler.py -> build/lib.linux-x86_64-cpython-37/apex/transformer/amp
creating build/lib.linux-x86_64-cpython-37/apex/transformer/functional
copying apex/transformer/functional/init.py -> build/lib.linux-x86_64-cpython-37/apex/transformer/functional
copying apex/transformer/functional/fused_softmax.py -> build/lib.linux-x86_64-cpython-37/apex/transformer/functional
creating build/lib.linux-x86_64-cpython-37/apex/transformer/pipeline_parallel
copying apex/transformer/pipeline_parallel/init.py -> build/lib.linux-x86_64-cpython-37/apex/transformer/pipeline_parallel
copying apex/transformer/pipeline_parallel/_timers.py -> build/lib.linux-x86_64-cpython-37/apex/transformer/pipeline_parallel
copying apex/transformer/pipeline_parallel/p2p_communication.py -> build/lib.linux-x86_64-cpython-37/apex/transformer/pipeline_parallel
copying apex/transformer/pipeline_parallel/utils.py -> build/lib.linux-x86_64-cpython-37/apex/transformer/pipeline_parallel
creating build/lib.linux-x86_64-cpython-37/apex/transformer/tensor_parallel
copying apex/transformer/tensor_parallel/init.py -> build/lib.linux-x86_64-cpython-37/apex/transformer/tensor_parallel
copying apex/transformer/tensor_parallel/cross_entropy.py -> build/lib.linux-x86_64-cpython-37/apex/transformer/tensor_parallel
copying apex/transformer/tensor_parallel/data.py -> build/lib.linux-x86_64-cpython-37/apex/transformer/tensor_parallel
copying apex/transformer/tensor_parallel/layers.py -> build/lib.linux-x86_64-cpython-37/apex/transformer/tensor_parallel
copying apex/transformer/tensor_parallel/mappings.py -> build/lib.linux-x86_64-cpython-37/apex/transformer/tensor_parallel
copying apex/transformer/tensor_parallel/memory.py -> build/lib.linux-x86_64-cpython-37/apex/transformer/tensor_parallel
copying apex/transformer/tensor_parallel/random.py -> build/lib.linux-x86_64-cpython-37/apex/transformer/tensor_parallel
copying apex/transformer/tensor_parallel/utils.py -> build/lib.linux-x86_64-cpython-37/apex/transformer/tensor_parallel
creating build/lib.linux-x86_64-cpython-37/apex/transformer/testing
copying apex/transformer/testing/init.py -> build/lib.linux-x86_64-cpython-37/apex/transformer/testing
copying apex/transformer/testing/arguments.py -> build/lib.linux-x86_64-cpython-37/apex/transformer/testing
copying apex/transformer/testing/commons.py -> build/lib.linux-x86_64-cpython-37/apex/transformer/testing
copying apex/transformer/testing/distributed_test_base.py -> build/lib.linux-x86_64-cpython-37/apex/transformer/testing
copying apex/transformer/testing/global_vars.py -> build/lib.linux-x86_64-cpython-37/apex/transformer/testing
copying apex/transformer/testing/standalone_bert.py -> build/lib.linux-x86_64-cpython-37/apex/transformer/testing
copying apex/transformer/testing/standalone_gpt.py -> build/lib.linux-x86_64-cpython-37/apex/transformer/testing
creating build/lib.linux-x86_64-cpython-37/apex/transformer/pipeline_parallel/schedules
copying apex/transformer/pipeline_parallel/schedules/init.py -> build/lib.linux-x86_64-cpython-37/apex/transformer/pipeline_parallel/schedules
copying apex/transformer/pipeline_parallel/schedules/common.py -> build/lib.linux-x86_64-cpython-37/apex/transformer/pipeline_parallel/schedules
copying apex/transformer/pipeline_parallel/schedules/fwd_bwd_no_pipelining.py -> build/lib.linux-x86_64-cpython-37/apex/transformer/pipeline_parallel/schedules
copying apex/transformer/pipeline_parallel/schedules/fwd_bwd_pipelining_with_interleaving.py -> build/lib.linux-x86_64-cpython-37/apex/transformer/pipeline_parallel/schedules
copying apex/transformer/pipeline_parallel/schedules/fwd_bwd_pipelining_without_interleaving.py -> build/lib.linux-x86_64-cpython-37/apex/transformer/pipeline_parallel/schedules
running build_ext
building 'apex_C' extension
creating /root/junyan/apex/build/temp.linux-x86_64-cpython-37
creating /root/junyan/apex/build/temp.linux-x86_64-cpython-37/csrc
Traceback (most recent call last):
File "", line 36, in
File "", line 34, in
File "/root/junyan/apex/setup.py", line 712, in
extras_require=extras,
File "/opt/conda/lib/python3.7/site-packages/setuptools/init.py", line 107, in setup
return distutils.core.setup(**attrs)
File "/opt/conda/lib/python3.7/site-packages/setuptools/_distutils/core.py", line 185, in setup
return run_commands(dist)
File "/opt/conda/lib/python3.7/site-packages/setuptools/_distutils/core.py", line 201, in run_commands
dist.run_commands()
File "/opt/conda/lib/python3.7/site-packages/setuptools/_distutils/dist.py", line 969, in run_commands
self.run_command(cmd)
File "/opt/conda/lib/python3.7/site-packages/setuptools/dist.py", line 1234, in run_command
super().run_command(command)
File "/opt/conda/lib/python3.7/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
cmd_obj.run()
File "/opt/conda/lib/python3.7/site-packages/wheel/bdist_wheel.py", line 325, in run
self.run_command("build")
File "/opt/conda/lib/python3.7/site-packages/setuptools/_distutils/cmd.py", line 318, in run_command
self.distribution.run_command(command)
File "/opt/conda/lib/python3.7/site-packages/setuptools/dist.py", line 1234, in run_command
super().run_command(command)
File "/opt/conda/lib/python3.7/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
cmd_obj.run()
File "/opt/conda/lib/python3.7/site-packages/setuptools/_distutils/command/build.py", line 131, in run
self.run_command(cmd_name)
File "/opt/conda/lib/python3.7/site-packages/setuptools/_distutils/cmd.py", line 318, in run_command
self.distribution.run_command(command)
File "/opt/conda/lib/python3.7/site-packages/setuptools/dist.py", line 1234, in run_command
super().run_command(command)
File "/opt/conda/lib/python3.7/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
cmd_obj.run()
File "/opt/conda/lib/python3.7/site-packages/setuptools/command/build_ext.py", line 84, in run
_build_ext.run(self)
File "/opt/conda/lib/python3.7/site-packages/setuptools/_distutils/command/build_ext.py", line 345, in run
self.build_extensions()
File "/opt/conda/lib/python3.7/site-packages/torch/utils/cpp_extension.py", line 709, in build_extensions
build_ext.build_extensions(self)
File "/opt/conda/lib/python3.7/site-packages/setuptools/_distutils/command/build_ext.py", line 467, in build_extensions
self._build_extensions_serial()
File "/opt/conda/lib/python3.7/site-packages/setuptools/_distutils/command/build_ext.py", line 493, in _build_extensions_serial
self.build_extension(ext)
File "/opt/conda/lib/python3.7/site-packages/setuptools/command/build_ext.py", line 246, in build_extension
_build_ext.build_extension(self, ext)
File "/opt/conda/lib/python3.7/site-packages/setuptools/_distutils/command/build_ext.py", line 555, in build_extension
depends=ext.depends,
File "/opt/conda/lib/python3.7/site-packages/torch/utils/cpp_extension.py", line 539, in unix_wrap_ninja_compile
with_cuda=with_cuda)
File "/opt/conda/lib/python3.7/site-packages/torch/utils/cpp_extension.py", line 1336, in _write_ninja_file_and_compile_objects
check_compiler_abi_compatibility(compiler)
File "/opt/conda/lib/python3.7/site-packages/torch/utils/cpp_extension.py", line 283, in check_compiler_abi_compatibility
if not check_compiler_ok_for_platform(compiler):
File "/opt/conda/lib/python3.7/site-packages/torch/utils/cpp_extension.py", line 243, in check_compiler_ok_for_platform
which = subprocess.check_output(['which', compiler], stderr=subprocess.STDOUT)
File "/opt/conda/lib/python3.7/subprocess.py", line 411, in check_output
**kwargs).stdout
File "/opt/conda/lib/python3.7/subprocess.py", line 512, in run
output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '['which', 'c++']' returned non-zero exit status 1.
error: subprocess-exited-with-error

× python setup.py bdist_wheel did not run successfully.
│ exit code: 1
╰─> See above for output.

note: This error originates from a subprocess, and is likely not a problem with pip.

except ImportError as error:
print(
"ERROR: Can not execute setup.py since setuptools is not available in "
"the build environment.",
file=sys.stderr,
)
sys.exit(1)

file = %r
sys.argv[0] = file

if os.path.exists(file):
filename = file
with tokenize.open(file) as f:
setup_py_code = f.read()
else:
filename = ""
setup_py_code = "from setuptools import setup; setup()"

exec(compile(setup_py_code, filename, "exec"))
'"'"''"'"''"'"' % ('"'"'/root/junyan/apex/setup.py'"'"',), "", "exec"))' --cpp_ext --cuda_ext bdist_wheel -d /tmp/pip-wheel-53qg6he5
cwd: /root/junyan/apex/
Building wheel for apex (setup.py) ... error
ERROR: Failed building wheel for apex
Running setup.py clean for apex
Running command python setup.py clean

torch.version = 1.9.0+cu111

/root/junyan/apex/setup.py:111: UserWarning: Option --pyprof not specified. Not installing PyProf dependencies!
warnings.warn("Option --pyprof not specified. Not installing PyProf dependencies!")

Compiling cuda extensions with
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2020 NVIDIA Corporation
Built on Tue_Sep_15_19:10:02_PDT_2020
Cuda compilation tools, release 11.1, V11.1.74
Build cuda_11.1.TC455_06.29069683_0
from /usr/local/cuda/bin

running clean
removing 'build/temp.linux-x86_64-cpython-37' (and everything under it)
removing 'build/lib.linux-x86_64-cpython-37' (and everything under it)
'build/bdist.linux-x86_64' does not exist -- can't clean it
'build/scripts-3.7' does not exist -- can't clean it
Failed to build apex
ERROR: Could not build wheels for apex, which is required to install pyproject.toml-based projects
".
It is very weird, but the installation process is blocked.

Could you help me?

Best Regards!
Momo

[Zakria96]

The problem is that the --build-option and --global-option are deprecated.
A possible replacement is to use --config-settings.

See this pypa/pip#11859