[ci] speed up the linux builds

[guolinke]

No description provided.

[guolinke]

before:

after:

[guolinke]

remaining bottlenecks:

1. GPU builds, cost about 12 mins.
2. Windows builds, cost about 10 mins. The main costs in Windows build are the conda install. Maybe we can use docker to speed up it.

[StrikerRUS]

Impressive speedup!

We can overcome bottlenecks by the ordering of builds.
We have 12 build tasks but are limited by 10 concurrent agents.
At present Windows builds, which are the longest ones, start after first finished ones. So the best case is 5 min (Linux) + 10 min (Windows).

If there is a way to control the order of the builds, then we can place Windows builds at first place, and the whole CI time will be max(Linux + Linux, Windows) which is smaller than the present Linux + Windows time.

[guolinke]

@StrikerRUS
maybe I can add some self-hosted agents, for gpu and windows bulids.

[StrikerRUS]

@guolinke In my opinion, it doesn't value to complicate CI for the 2-3 min speedup (in case we are able to reorder the builds).

[guolinke]

@StrikerRUS can we merge this first?

[chivee]

@StrikerRUS , the build time varies from time to time. it depends on whether CI/AppVeyor is busy or not. and the O(Build) was same if there are few PR was build at the same time.

I think another opinion is that could the build logic of Push & PR was trigger at different phase?

[chivee]

I'm not sure whether CI will optimize for the same source code build, considering there will be two times build (appveyor/branch v.s appveyor/pr and travis-ci/pr v.s travis-ci/push) for every code commit.

[StrikerRUS]

@chivee As I know, the only way to optimize this is creating PR at later time, when all code changes have been done 😢 .

Also, there will be useful to enable the option on Azure (if exists) similar to Appveyor's Rolling Builds and Travis' Auto-Cancellation.

[StrikerRUS]

@guolinke Please take a look: I left some comments/questions below.

[StrikerRUS]

Do you want to leave this commented?

[StrikerRUS]

Maybe move this to the shell script:

if [[ $TRAVIS == "true" ]]; then
    sudo add-apt-repository ppa:kzemek/boost -y
fi

somewhere here
https://github.com/Microsoft/LightGBM/blob/d852716c21e52fda47d7eccee7e6b674c1e9e35d/.ci/setup.sh#L44
for not bothering other tasks, except gpu?