[dask] Dask training fails using LocalCluster on Windows

[philip-2020]

I am receiving the Warning as bellow:

C:\Users\pwang\PycharmProjects\01_PL_Home\venv\lib\site-packages\lightgbm\dask.py:526: UserWarning: Parameter n_jobs will be ignored.
  _log_warning(f"Parameter {param_alias} will be ignored.")
Finding random open ports for workers
[LightGBM] [Fatal] Machine list file doesn't contain the local machine
2023-06-08 15:22:57,309 - distributed.worker - WARNING - Compute Failed
Key:       _train_part-5abf29b6-2c80-45a7-83fd-3532f41ccffa

import dask.array as da
from distributed import Client, LocalCluster
from sklearn.datasets import make_regression

import lightgbm as lgb

workspace = "C:/Users/pwang/PycharmProjects/Temp/"


# ----------------------------------------------------------------
# https://github.com/microsoft/LightGBM/blob/master/examples/python-guide/dask/regression.py
# ----------------------------------------------------------------
if __name__ == "__main__":
    print("loading data")

    X, y = make_regression(n_samples=1000, n_features=50)

    print("initializing a Dask cluster")

    cluster = LocalCluster(n_workers=2, local_directory=workspace)
    client = Client(cluster)

    print("created a Dask LocalCluster")

    print("distributing training data on the Dask cluster")

    dX = da.from_array(X, chunks=(100, 50))
    dy = da.from_array(y, chunks=(100,))

    print("beginning training")

    dask_model = lgb.DaskLGBMRegressor(n_estimators=10)
    dask_model.fit(dX, dy)
    assert dask_model.fitted_

    print("done training")

Dask workers look good on dashboard, each worker has 15.9GiB memory and 6 threads.
The laptop has 6 cores 2 threads per core. OS is Windows 10 Enterprise.
Ligbm 3.3.5, Dask 2023.5.1, distributed 2023.5.1, Python 3.10

Thank you for any help!

Originally posted by @philip-2020 in #4942 (comment)

[jameslamb]

Thanks for your questions and for using LightGBM. And thanks for providing a minimal, reproducible example! I've reformatted the text a bit and modified the description to more closely match the issue you're reporting.

Please see https://docs.github.com/en/get-started/writing-on-github/getting-started-with-writing-and-formatting-on-github/basic-writing-and-formatting-syntax to learn how to format code and other text on GitHub.

Unfortunately, lightgbm.dask is not currently supported on Windows, as noted at https://lightgbm.readthedocs.io/en/v3.3.5/Parallel-Learning-Guide.html#dask.

And as can be seen in the test configuration for the Python package:

LightGBM/tests/python_package_test/test_dask.py

Lines 20 to 22 in fe69fa9

	if not platform.startswith('linux'):
	pytest.skip('lightgbm.dask is currently supported in Linux environments', allow_module_level=True)
	if machine() != 'x86_64':

We also already have a feature request in the project backlog to add Dask support on macOS and Windows: #3782.

If you're interested in trying to contribute Windows support we would welcome a contribution! Otherwise, you can subscribe to that issue to be notified if it's addressed by some future contribution.

[jameslamb]

@jmoralez Do you think we should add a warning in the Dask interface like the following?

*WARN: lightgbm.dask is only supported on Linux systems. See #3782 for the latest developments on supporting other platforms.

I think it should be a warning and not an error, but that in general something printed by code is more likely to be seen by users than documentation on the docs site (and definitely more prominent than configuration in a test file).

[philip-2020]

Thank you so much! I am a starter in this field and trying to learn. If I find anything I will update. I will try on my personal Ubuntu linux machine.

[jameslamb]

Ok no problem. Sorry for the inconvenience.

For now, I'll close this issue and let's use #3782 to discuss progress towards supporting macOS and Windows in lightgbm.dask.

[jmoralez]

Yes, I think printing that would be useful.

[github-actions]

This issue has been automatically locked since there has not been any recent activity since it was closed. To start a new related discussion, open a new issue at https://github.com/microsoft/LightGBM/issues including a reference to this.