Buildkite CI improvements

[markov00]

Future enhancements

• Create foundation of buildkite CI ci(buildkite): create foundation for ci and triggers #1733
• Remove obsolete code and configurations ci: remove obsolete configurations #1803
• git clone with depth 1 to avoid downloading the full repo see here
• Enable better caching see link
• Configure our own agents. Currently we are using a small instance from kibana that is suitable to all our jobs. But the wait times for these are between 1m and >3m for the job to even start, even with the job priority set very high. Not using our own agents also limits our ability to cache things specific to charts. I don't think this is a huge issue for the time being but should be considered later.
• Optimize the buildkite upload script (see example run). This script, like all others, takes several minutes (~3) to wait, start and finish running. This is creates incredible flexibility for dynamic pipelines but blocks all other jobs from even starting. Having our own agent pool would help this.
• Verify the reason why every job has a installing npm modules as part of their last command. In particular the Firebase deploy has a very long 1.4 minutes node module install process https://buildkite.com/elastic/datavis-ci/builds/646#_
• expose logs at github level or link directly the status of a PR to the buildkite without passing for github
• show the total CI time testing time on the PR message
• When requesting to update the screenshot from a GH comment, just rerun the minimal required tasks

[markov00]

Closing this because it's not planned to be resolved in the foreseeable future. It will be tracked in our Icebox and will be re-opened if our priorities change. Feel free to re-open if you think it should be melted sooner.