Run examples in cmake CI

[mostafaelhoushi]

Addresses #596

[ChrisCummins]

For example_compiler_gym_service/demo_without_bazel.py, I wrote a simple "smoke test" that runs the main entry point:

CompilerGym/examples/example_compiler_gym_service/demo_without_bazel_test.py

Lines 5 to 12 in 2e3b176

	"""Smoke test for examples/example_compiler_gym_service/demo_without_bazel.py"""
	from example_compiler_gym_service.demo_without_bazel import main
	from flaky import flaky
	@flaky
	def test_demo_without_bazel():
	main()

I believes this should be run by the CI's example test jobs. Would this cover your needs for the other files?

Cheers,
Chris

[codecov-commenter]

Codecov Report

Merging #695 (75d0968) into development (9ad5fbb) will decrease coverage by 0.08%.
The diff coverage is n/a.

@@               Coverage Diff               @@
##           development     #695      +/-   ##
===============================================
- Coverage        88.64%   88.55%   -0.09%     
===============================================
  Files              131      131              
  Lines             7926     7926              
===============================================
- Hits              7026     7019       -7     
- Misses             900      907       +7     

Impacted Files	Coverage Δ
...er_gym/third_party/inst2vec/inst2vec_preprocess.py	69.73% <0.00%> (-4.22%)	⬇️
compiler_gym/util/commands.py	83.33% <0.00%> (-2.39%)	⬇️
compiler_gym/envs/llvm/datasets/cbench.py	79.42% <0.00%> (-1.09%)	⬇️
compiler_gym/envs/gcc/service/gcc_service.py	97.06% <0.00%> (+0.36%)	⬆️
compiler_gym/service/connection.py	78.33% <0.00%> (+0.66%)	⬆️
...loop_tool/service/loop_tool_compilation_session.py	90.54% <0.00%> (+0.67%)	⬆️
compiler_gym/spaces/reward.py	80.70% <0.00%> (+7.01%)	⬆️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 9ad5fbb...75d0968. Read the comment docs.

[mostafaelhoushi]

I believes this should be run by the CI's example test jobs. Would this cover your needs for the other files?

OK. Got it. I will update the CI accordingly...
How can we ensure that the new tests I create will only run in the cmake job?

The point is that the other 2 tests (loop_unroller and opt_loops) require building some executables. I have tried in this CI adding the other tests: #642 but I found myself opening a can of worms trying to get those test files working for different jobs that use bazel, CMake, or pip install.

If I can get those examples to work with CMake only, and then look into getting them to work in other jobs that would be great.

[ChrisCummins]

The point is that the other 2 tests (loop_unroller and opt_loops) require building some executables.

Ah, sorry, I see now.

If I can get those examples to work with CMake only, and then look into getting them to work in other jobs that would be great.

I think you need to use CMake's testing ctest program to run the tests:

https://cmake.org/cmake/help/book/mastering-cmake/chapter/Testing%20With%20CMake%20and%20CTest.html

TBH I'm not super familiar with it, but you could maybe write a test job that depends on the cmake build and runs ctest on the examples dir?

Cheers,
Chris

[mostafaelhoushi]

So far, this is working:

• running the example_unrolling_service and loop_optimizations_service python examples inside the build-linux-cmake job
• uploading the loop_unroller and opt_loops binaries inside the build-linux-cmake job
• downloading the loop_unroller and opt_loops binaries inside the test-linux-cmake job

However, this is not working:

• running the example_unrolling_service and loop_optimizations_service python examples inside the test-linux-cmake job
  It returns an error saying that it can't find the loop_unroller binary, although I made sure it's moved to its correct directory.

[mostafaelhoushi]

OK. I just noticed that about GitHub's upload/download artifcats: https://github.com/actions/download-artifact#permission-loss:

❗ File permissions are not maintained during artifact upload ❗ 

So now I have chmod on the binaries after downloading them. Hopefully this time it will work.

[ChrisCummins]

Good progress @mostafaelhoushi!

[ChrisCummins]

@mostafaelhoushi, these shouldn't be necessary:

Do you see the UI button to re-run jobs?

[mostafaelhoushi]

Do you see the UI button to re-run jobs?

For some reason that "re-run jobs" button doesn't always appear to me. Most of the time it is not there :(

[ChrisCummins]

Oh weird, but sometimes it is? So not a permission issue?

[ChrisCummins]

here is what the "details" screen for the current running CI job looks like:

If I were to hit Cancel, I'd have to wait a few seconds/minutes for the job to finish cancelling before the Re-try button shows up.

Cheers,
Chris

[mostafaelhoushi]

here is what the "details" screen for the current running CI job looks like:
If I were to hit Cancel, I'd have to wait a few seconds/minutes for the job to finish cancelling before the Re-try button shows up.

Actually, sometimes I would see the "re-run job" button in that "Details" screen and sometimes I don't.

To make things even more confusing, now when I click on the "Checks" tab of the PR, I do see the "re-run jobs" button, but earlier today I couldn't see it:

[mostafaelhoushi]

Only one job is remaining and it says "Waiting for a runner to pick up this job..." is this expected?

[ChrisCummins]

Hmm, guess we're stuck in a queue somewhere

[ChrisCummins]

Bit just be one of those "grab a cup of coffee and come back later" messages

[mostafaelhoushi]

Never mind... the test ran... the problem was solved... but now hit another error:

NotImplementedError: [Errno 2] No such file or directory: '/home/runner/work/CompilerGym/CompilerGym/examples/loop_optimizations_service/service_py/./../../../compiler_gym/third_party/programl/compute_programl'

I think that should be easy to solve: upload that compute_programl binary and download it.
However, I believe in the future that binary should be part of the *.whl file.

[ChrisCummins]

Is there a difference between compute_programl and running compute_observation Programl /path/to/bitcode.bc?

The compute_observation binary is in the wheel

[mostafaelhoushi]

Is there a difference between compute_programl and running compute_observation Programl /path/to/bitcode.bc?

The compute_observation binary is in the wheel

OK. I wasn't aware of compute_observation. I created an Issue to replace compute_programl with compute_observation : #702

[mostafaelhoushi]

build-linux-cmake and test-linux-cmake are now green. We can merge it now.

In upcoming PRs I will remove the calls to the examples from ci.yaml and ensure that pytest runs them instead.

[mostafaelhoushi]

build-linux-cmake and test-linux-cmake are now green. We can merge it now.

In upcoming PRs I will remove the calls to the examples from ci.yaml and ensure that pytest runs them instead.