Add Python wheel building + Publishing workflow #70
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
A docker image is made in necessary129/manylinux2014-LLVM with prebuilt LLVM + cmake + ninja cibuildwheel is being used to generate the wheels. Which uses `auditwheel` to bundle linked libraries. So the ELF patching in `preparation.py` is not needed. The includes can be be found by Python by itself. Only `utils.h` needs to be specifically included.
A new musl docker image was made. Which would make it possible to build musl wheels also.
Will publish to pypi on release
necessary129
force-pushed
the
python-packaging-3
branch
4 times, most recently
from
0173bab
to
1d34c00
Compare
Also removed the prebuild wheels as wheel as CI autogenerated now
necessary129
force-pushed
the
python-packaging-3
branch
from
1d34c00
to
3f59245
Compare
svkeerthy
requested review from
svkeerthy and
ShikharJain97
and removed request for
svkeerthy
|
Hi @necessary129, |
Will also publish to testpypi first on release, and a manual workflow dispatch will have the option to deploy to actual pypi
necessary129
force-pushed
the
python-packaging-3
branch
from
38d9741
to
b374381
Compare
necessary129
force-pushed
the
python-packaging-3
branch
from
d780901
to
c01d516
Compare
necessary129
force-pushed
the
python-packaging-3
branch
from
c01d516
to
4739ca3
Compare
Added tests. They only test the program vectors in oracle at the moment because the function names in oracle and those produced by the Python binding is different. |
necessary129
force-pushed
the
python-packaging-3
branch
2 times, most recently
from
de5e1af
to
a7d5393
Compare
Compare the python module output to the oracle
necessary129
force-pushed
the
python-packaging-3
branch
from
a7d5393
to
7671950
Compare
necessary129
force-pushed
the
python-packaging-3
branch
from
2f12304
to
ffa53fc
Compare
necessary129
force-pushed
the
python-packaging-3
branch
3 times, most recently
from
21397c8
to
f109e22
Compare
Will test each FA and SYM flevel embeddings also
Also fix a memory leak caused by not freeing the `__cxx::demangle` return pointer
Python wheels are built in GLIBc ~ 2.17 so some functions are not demangled properly. So we rebuild them using the ir2vec binary and test the Python output against it. The ir2vec binary output is tested against the actual oracle in another workflow. So, this should be fine.
clang doesn't compile on musllinux directly. Need to figure that out. in the meantime, don't build it. musl is not common anyway.
necessary129
force-pushed
the
python-packaging-3
branch
from
f109e22
to
104f2a4
Compare
necessary129
force-pushed
the
python-packaging-3
branch
from
e54f2c5
to
5f66ba8
Compare
necessary129
force-pushed
the
python-packaging-3
branch
from
5f66ba8
to
38c3e51
Compare
svkeerthy
approved these changes
Sep 24, 2023
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
A docker image is made in necessary129/manylinux2014-LLVM
with prebuilt LLVM + cmake + ninja
cibuildwheel is being used to generate the wheels. Which
uses
auditwheelto bundle linked libraries. So the ELFpatching in
preparation.pyis not needed.The includes can be be found by Python by itself. Only
utils.hneeds to be specifically included.
wheels are automatically made for all linux x86_64 Python versions
from 3.6 to 3.11
On publishing a release, the Python package is also published to
testpypi (as a proof of concept) -- This should be changed to actual pypi
on the main branch.
Each wheel is also down to ~5MB from ~60MB
Will close #52
The testpypi package: https://test.pypi.org/project/IR2Vec/