Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(workflows): cibuildwheel integration #45

Merged
merged 7 commits into from
Aug 4, 2022
Merged

feat(workflows): cibuildwheel integration #45

merged 7 commits into from
Aug 4, 2022

Conversation

XuehaiPan
Copy link
Member

Description

Describe your changes in detail.

Motivation and Context

Use cibuildwheel to build manylinux wheels in GitHub Actions and update the contribution guide.

Types of changes

What types of changes does your code introduce? Put an x in all the boxes that apply:

  • New feature (non-breaking change which adds core functionality)

Checklist

Go over all the following points, and put an x in all the boxes that apply.
If you are unsure about any of these, don't hesitate to ask. We are here to help!

  • I have read the CONTRIBUTION guide (required)
  • My change requires a change to the documentation.
  • I have updated the tests accordingly (required for a bug fix or a new feature).
  • I have updated the documentation accordingly.
  • I have reformatted the code using make format (required)
  • I have checked the code using make lint (required)
  • I have ensured make test pass. (required)

@XuehaiPan XuehaiPan force-pushed the cibuildwheel branch 5 times, most recently from 0ad231b to 214813b Compare July 27, 2022 16:37
@XuehaiPan XuehaiPan added the enhancement New feature or request label Jul 27, 2022
@XuehaiPan XuehaiPan force-pushed the cibuildwheel branch 2 times, most recently from de7b5c6 to 2aab0d8 Compare July 28, 2022 06:49
@XuehaiPan XuehaiPan linked an issue Aug 3, 2022 that may be closed by this pull request
3 tasks
@@ -144,8 +144,7 @@ endif()
unset(TORCH_LIBRARIES)

foreach(VAR_PATH ${TORCH_LIBRARY_PATH})
file(GLOB TORCH_LIBRARY "${VAR_PATH}/*.so")
list(APPEND TORCH_LIBRARIES "${TORCH_LIBRARY}")
list(APPEND TORCH_LIBRARIES "${VAR_PATH}/libtorch_python.so")
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

See #46 (comment).

We only need to link libtorch_python.so instead of globbing for all shared libraries.

@XuehaiPan XuehaiPan mentioned this pull request Aug 3, 2022
3 tasks
@XuehaiPan
Copy link
Member Author

I think this is ready to merge now @Benjamin-eecs @JieRen98. I can build py310 wheels on my local machine (inside docker). The build errors may be resolved in the next PyTorch minor release (1.12.1).

@@ -243,7 +243,7 @@ cd TorchOpt
CONDA_OVERRIDE_CUDA=11.7 conda env create --file conda-recipe.yaml

conda activate torchopt
pip3 install -e .
pip3 install --no-build-isolation --editable .
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Wondering what is no-build-isolation, seem a bit complicated for new users or ones who didn't go through readme.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Wondering what is no-build-isolation

We list torch == 1.12.0 as our build dependency to get torch libs and includes at build time. Now the default behavior for pip install -e will build a wheel in an isolated environment before installation. If the user does not set environment variable PIP_EXTRA_INDEX_URL=https://download.pytorch.org/whl/cu116", pip will install torch from PyPI, which is torch == 1.12.0+cu102, for building our C++ extension.

seem a bit complicated for new users or ones who didn't go through readme.

We only link libtorch_python.so now, the user can use pip3 install -e . without any problem.

The case to add the --no-build-isolation option is to reduce unnecessary downloads for torch wheels from PyPI. The wheel builder can use the already installed torch for compilation.

@XuehaiPan XuehaiPan merged commit 3023bc9 into main Aug 4, 2022
@XuehaiPan XuehaiPan deleted the cibuildwheel branch August 4, 2022 08:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[BUG] Separate CPU / CUDA wheels
2 participants