libstdc++.so.6: version GLIBCXX_3.4.30' not found`

[stas00]

After doing pip install deepspeed the JIT build was failing to load async_io lib:

tests/deepspeed/test_deepspeed.py:390: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
src/transformers/trainer.py:1631: in train
    return inner_training_loop(
src/transformers/trainer.py:1700: in _inner_training_loop
    deepspeed_engine, optimizer, lr_scheduler = deepspeed_init(
src/transformers/deepspeed.py:378: in deepspeed_init
    deepspeed_engine, optimizer, _, lr_scheduler = deepspeed.initialize(**kwargs)
/home/stas/anaconda3/envs/py38-pt113/lib/python3.8/site-packages/deepspeed/__init__.py:125: in initialize
    engine = DeepSpeedEngine(args=args,
/home/stas/anaconda3/envs/py38-pt113/lib/python3.8/site-packages/deepspeed/runtime/engine.py:336: in __init__
    self._configure_optimizer(optimizer, model_parameters)
/home/stas/anaconda3/envs/py38-pt113/lib/python3.8/site-packages/deepspeed/runtime/engine.py:1292: in _configure_optimizer
    self.optimizer = self._configure_zero_optimizer(basic_optimizer)
/home/stas/anaconda3/envs/py38-pt113/lib/python3.8/site-packages/deepspeed/runtime/engine.py:1594: in _configure_zero_optimizer
    optimizer = DeepSpeedZeroOptimizer_Stage3(
/home/stas/anaconda3/envs/py38-pt113/lib/python3.8/site-packages/deepspeed/runtime/zero/stage3.py:151: in __init__
    self.parameter_offload = DeepSpeedZeRoOffload(
/home/stas/anaconda3/envs/py38-pt113/lib/python3.8/site-packages/deepspeed/runtime/zero/parameter_offload.py:209: in __init__
    self._convert_to_zero_parameters(ds_config, module, mpu)
/home/stas/anaconda3/envs/py38-pt113/lib/python3.8/site-packages/deepspeed/runtime/zero/parameter_offload.py:273: in _convert_to_zero_parameters
    Init(module=module,
/home/stas/anaconda3/envs/py38-pt113/lib/python3.8/site-packages/deepspeed/runtime/zero/partition_parameters.py:699: in __init__
    self.param_swapper = AsyncPartitionedParameterSwapper(_ds_config, self.dtype)
/home/stas/anaconda3/envs/py38-pt113/lib/python3.8/site-packages/deepspeed/runtime/swap_tensor/partitioned_param_swapper.py:38: in __init__
    aio_op = AsyncIOBuilder().load(verbose=False)
/home/stas/anaconda3/envs/py38-pt113/lib/python3.8/site-packages/deepspeed/ops/op_builder/builder.py:485: in load
    return self.jit_load(verbose)
/home/stas/anaconda3/envs/py38-pt113/lib/python3.8/site-packages/deepspeed/ops/op_builder/builder.py:520: in jit_load
    op_module = load(
/home/stas/anaconda3/envs/py38-pt113/lib/python3.8/site-packages/torch/utils/cpp_extension.py:1284: in load
    return _jit_compile(
/home/stas/anaconda3/envs/py38-pt113/lib/python3.8/site-packages/torch/utils/cpp_extension.py:1534: in _jit_compile
    return _import_module_from_library(name, build_directory, is_python_module)
/home/stas/anaconda3/envs/py38-pt113/lib/python3.8/site-packages/torch/utils/cpp_extension.py:1936: in _import_module_from_library
    module = importlib.util.module_from_spec(spec)
<frozen importlib._bootstrap>:556: in module_from_spec
    ???
<frozen importlib._bootstrap_external>:1166: in create_module
    ???
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

f = <built-in function create_dynamic>
args = (ModuleSpec(name='async_io', loader=<_frozen_importlib_external.ExtensionFileLoader object at 0x7fa944fc0430>, origin='/home/stas/.cache/torch_extensions/py38_cu117/async_io/async_io.so'),)
kwds = {}

>   ???
E   ImportError: /home/stas/anaconda3/envs/py38-pt113/lib/python3.8/site-packages/torch/lib/../../../../libstdc++.so.6: version `GLIBCXX_3.4.30' not found (required by /home/stas/.cache/torch_extensions/py38_cu117/async_io/async_io.so)

<frozen importlib._bootstrap>:219: ImportError

It looks like conda env has a borked libstdc++.so.6 library or some dynamically linked object isn't found, even though ldd seems to be ok

but we can tell the symbol isn't there (this is what it is complaining about)

nm /home/stas/anaconda3/envs/py38-pt113/lib/python3.8/site-packages/torch/lib/../../../../libstdc++.so.6  | grep GLIBCXX
0000000000000000 A GLIBCXX_3.4
0000000000000000 A GLIBCXX_3.4.1
0000000000000000 A GLIBCXX_3.4.10
0000000000000000 A GLIBCXX_3.4.11
0000000000000000 A GLIBCXX_3.4.12
0000000000000000 A GLIBCXX_3.4.13
0000000000000000 A GLIBCXX_3.4.14
0000000000000000 A GLIBCXX_3.4.15
0000000000000000 A GLIBCXX_3.4.16
0000000000000000 A GLIBCXX_3.4.17
0000000000000000 A GLIBCXX_3.4.18
0000000000000000 A GLIBCXX_3.4.19
0000000000000000 A GLIBCXX_3.4.2
0000000000000000 A GLIBCXX_3.4.20
0000000000000000 A GLIBCXX_3.4.21
0000000000000000 A GLIBCXX_3.4.22
0000000000000000 A GLIBCXX_3.4.23
0000000000000000 A GLIBCXX_3.4.24
0000000000000000 A GLIBCXX_3.4.25
0000000000000000 A GLIBCXX_3.4.26
0000000000000000 A GLIBCXX_3.4.27
0000000000000000 A GLIBCXX_3.4.28
0000000000000000 A GLIBCXX_3.4.29
0000000000000000 A GLIBCXX_3.4.3
0000000000000000 A GLIBCXX_3.4.4
0000000000000000 A GLIBCXX_3.4.5
0000000000000000 A GLIBCXX_3.4.6
0000000000000000 A GLIBCXX_3.4.7
0000000000000000 A GLIBCXX_3.4.8
0000000000000000 A GLIBCXX_3.4.9
00000000000aec70 T _ZN10__gnu_norm15_List_node_base4hookEPS0_@@GLIBCXX_3.4
00000000000aeb70 T _ZN10__gnu_norm15_List_node_base4swapERS0_S1_@@GLIBCXX_3.4
00000000000aec90 T _ZN10__gnu_norm15_List_node_base6unhookEv@@GLIBCXX_3.4
00000000000aec40 T _ZN10__gnu_norm15_List_node_base7reverseEv@@GLIBCXX_3.4

[stas00]

The solution that worked for me was

conda install -c conda-forge libstdcxx-ng=12

Found here: https://stackoverflow.com/a/74533050/9201239

Though after the update nm /home/stas/anaconda3/envs/py38-pt113/lib/libstdc++.so.6.0.30 | grep GLIBCXX_3 still shows unresolved symbols, but I didn't dig deeper as it worked and I moved on.

[aixuedegege]

apt-get install libstdc++6 worked for me

[104-wonohfor]

sudo apt-get update
sudo apt-get upgrade
sudo apt-get dist-upgrade

Then, check "GLIBCXX_3.4.30" by:
strings /usr/lib/x86_64-linux-gnu/libstdc++.so.6 | grep GLIBCXX
If it has, then:

import os
os.environ['LD_LIBRARY_PATH'] = '/usr/lib/x86_64-linux-gnu'

This worked for me.

[xushijie]

The strings /usr/lib/x86_64-linux-gnu/libstdc++.so.6 | grep GLIBCXX list GLIBCXX_3.4.30 version, why the strings /path/to/anacode/.../libstdc++.so.6 | grep GLIBCXX does not. It is easy to resolve this by placing the LD_LIBRARY_PATH ahead of /path/to/anacode/.../lib. But the question is why a newly built pytorch would result to this issue.