Irreproducible Segfault with LightGBM

[vruvora]

We are experiencing cases of irreproducible Segfault with LightGBM. Does anyone else have this issue?

Environment info

Operating System:

CPU/GPU model:

C++/Python/R version:

Error message

Reproducible examples

Steps to reproduce

[henry0312]

Did you install LightGBM from PyPI?
The distributions on PyPI may be broken.
(I guess compile optimization may be a cause)
So, can you try to compile from source?

[vruvora]

We are installing with pip install. It just hangs in a middle of an optimization. We can try compiling from source but is there a reason why PyPI may be broken?

[henry0312]

I guess there are something wrong with manylinux and -O3.
Maybe, we should compile without -O3 on manylinux.

[vruvora]

Interesting. Have you or anyone else experienced this because we have been experiencing this a fair amount but I have not seen any big reported use cases? This seems like it would break the package from being used in production.

[guolinke]

@vruvora
Did the Segfault occur often?

[vruvora]

Yes. It has been happening a quite a bit as of late. We have automated statistical unit tests with sample datasets which may be a degenerate optimization problem. However, we use them to make sure our pipeline is in order when we have big changes. Generally, ~1/5 times it will segfault and it will stop once we do it again.

[henry0312]

I saw uncertain crash many times in my main job.

[vruvora]

@henry0312 Interesting. Have you been able to fix it?

[henry0312]

@vruvora you should be sure to uninstall the package from PyPI and install manually.

[henry0312]

@vruvora Yeah, installing from source have solved the problem.

[vruvora]

@henry0312 I will try this out and update accordingly. Thanks. @guolinke Do you know why source isn't up to date with PyPI?

[henry0312]

No, please see https://github.com/Microsoft/LightGBM/blob/master/docs/Installation-Guide.rst.

[guolinke]

@vruvora you can use

pip install --no-binary :all: lightgbm

as well.

[guolinke]

@henry0312
building without O3 may slow-down the running speed.

I am not very sure why Segfault happened, and why building from source can fix it.

[henry0312]

@guolinke Yes, -O3 makes cumpting perfomance better, but there may be something wrong with tunes for manylinux.

[vruvora]

@guolinke Will this be slower pip install --no-binary :all: lightgbm than regular pip? Can you elaborate on the differences?

[henry0312]

@vruvora have you solved your issue?

[vruvora]

@henry0312 Yes. As far as we know. No more segafults.

[guolinke]

should we add this problem and solution to the document?

[StrikerRUS]

ping @henry0312

[vruvora]

Yes. We should.

[henry0312]

yeah, I also think so.

[guolinke]

@vruvora

@guolinke Will this be slower pip install --no-binary :all: lightgbm than regular pip? Can you elaborate on the differences?

The install speed will be slower. But the running speed isn't affected.

[StrikerRUS]

@vruvora A bit simplistic, the difference is in that regular pip simply downloads .whl file with prebuilt library file and copies its content in the appropriate directory.
In opposite, pip install --no-binary :all: compiles library file on your machine and performs copying.

@henry0312 Would you mind creating a PR with new FAQ entry about this?