[BUG-REPORT] Vaex uses more threads than it should / documentation is incomplete

[vladmihaisima]

Description
The page https://vaex.readthedocs.io/en/latest/conf.html#thread-count specifies that by setting vaex.settings.main.thread_count, you set the thread count, while this in practice this happens only when setting the environment variable VAEX_NUM_THREADS.

This was observed during run on server that had large differences between actual number of processors (96) and number of threads allowed for the particular vaex program (via a batch system) which resulted in the process being killed, even though the program was setting vaex.settings.main.thread_count to a small number of threads.

This seems indeed to be the case by checking the code, thread_count_default is set at

vaex/packages/vaex-core/vaex/multithreading.py

Line 21 in 98b9f79

thread_count_default = vaex.utils.get_env_type(int, 'VAEX_NUM_THREADS', multiprocessing.cpu_count())

but as far as I can tell that does not check at all the settings.

Software information

• Vaex version (import vaex; vaex.__version__): {'vaex': '4.12.0', 'vaex-core': '4.12.0', 'vaex-viz': '0.5.3', 'vaex-hdf5': '0.12.3', 'vaex-server': '0.8.1', 'vaex-astro': '0.9.1', 'vaex-jupyter': '0.8.0', 'vaex-ml': '0.18.0'}
• Vaex was installed via: pip
• OS: Arch (similar behavior on Ubuntu)

Additional information
Code to reproduce:

import vaex
vaex.settings.main.thread_count = 2
vaex.settings.main.thread_count_io = 2

import vaex.dataframe
df = vaex.example()
print(f"vaex.dataframe.main_executor.thread_pool.nthreads={vaex.dataframe.main_executor.thread_pool.nthreads}")

results in (incorrect)

vaex.dataframe.main_executor.thread_pool.nthreads=12

While if you run

VAEX_NUM_THREADS=2 python3 test.py

you get (the correct):

vaex.dataframe.main_executor.thread_pool.nthreads=2

[maartenbreddels]

Thanks, this is fixed in #2268 and should be out in the next release (4.15)