Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add PyTorch macosx-arm64 cross-compiled build #1463

Merged
merged 7 commits into from
Jan 25, 2024
Merged

Add PyTorch macosx-arm64 cross-compiled build #1463

Show file tree
Hide file tree
Changes from 3 commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
a3bd375
Add PyTorch macosx-arm64 cross-compiled build
sbrunk Jan 13, 2024
4ca07ee
Improve PyTorch cross-compilation and add Maven profile for CI build
sbrunk Jan 21, 2024
66d9543
Make PyTorch cross-compilation more consistent with numpy and scipy
sbrunk Jan 23, 2024
1673bbb
Fix builds for PyTorch
saudet Jan 24, 2024
bda4475
Fix builds for PyTorch some more
saudet Jan 24, 2024
bd9d136
Update changelog
sbrunk Jan 24, 2024
6f5cfb0
Merge remote-tracking branch 'upstream/master' into pytorch-macosx-arm64
saudet Jan 24, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 5 additions & 1 deletion .github/workflows/pytorch.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -32,6 +32,10 @@ jobs:
path: /home/runner/.ccache
- uses: bytedeco/javacpp-presets/.github/actions/deploy-ubuntu@actions
timeout-minutes: 350
macosx-arm64:
runs-on: macos-11
steps:
- uses: bytedeco/javacpp-presets/.github/actions/deploy-macosx@actions
macosx-x86_64:
runs-on: macos-11
# strategy:
Expand Down Expand Up @@ -61,7 +65,7 @@ jobs:
- uses: bytedeco/javacpp-presets/.github/actions/deploy-windows@actions
timeout-minutes: 350
redeploy:
needs: [linux-x86_64, macosx-x86_64, windows-x86_64]
needs: [linux-x86_64, macosx-arm64, macosx-x86_64, windows-x86_64]
runs-on: ubuntu-20.04
steps:
- uses: bytedeco/javacpp-presets/.github/actions/redeploy@actions
1 change: 1 addition & 0 deletions pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1466,6 +1466,7 @@
<module>libpostal</module>
<module>libraw</module>
<module>leptonica</module>
<module>pytorch</module>
<module>tesseract</module>
<module>sentencepiece</module>
</modules>
Expand Down
78 changes: 51 additions & 27 deletions pytorch/cppbuild.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -56,6 +56,7 @@ git submodule foreach --recursive 'git reset --hard'
# https://github.com/pytorch/pytorch/pull/66219
#patch -Np1 < ../../../pytorch.patch

CPYTHON_HOST_PATH="$INSTALL_PATH/../../../cpython/cppbuild/$PLATFORM/host/"
CPYTHON_PATH="$INSTALL_PATH/../../../cpython/cppbuild/$PLATFORM/"
OPENBLAS_PATH="$INSTALL_PATH/../../../openblas/cppbuild/$PLATFORM/"
NUMPY_PATH="$INSTALL_PATH/../../../numpy/cppbuild/$PLATFORM/"
Expand All @@ -65,7 +66,12 @@ if [[ -n "${BUILD_PATH:-}" ]]; then
IFS="$BUILD_PATH_SEPARATOR"
for P in $BUILD_PATH; do
if [[ $(find "$P" -name Python.h) ]]; then
CPYTHON_PATH="$P"
if [[ "$(basename $P)" == "$PLATFORM_HOST" ]]; then
CPYTHON_HOST_PATH="$P"
fi
if [[ "$(basename $P)" == "$PLATFORM" ]]; then
CPYTHON_PATH="$P"
fi
elif [[ -f "$P/include/openblas_config.h" ]]; then
OPENBLAS_PATH="$P"
elif [[ -f "$P/python/numpy/core/include/numpy/numpyconfig.h" ]]; then
Expand All @@ -75,36 +81,46 @@ if [[ -n "${BUILD_PATH:-}" ]]; then
IFS="$PREVIFS"
fi

CPYTHON_HOST_PATH="${CPYTHON_HOST_PATH//\\//}"
CPYTHON_PATH="${CPYTHON_PATH//\\//}"
OPENBLAS_PATH="${OPENBLAS_PATH//\\//}"
NUMPY_PATH="${NUMPY_PATH//\\//}"

if [[ -f "$CPYTHON_PATH/include/python3.12/Python.h" ]]; then
# setup.py won't pick up the right libgfortran.so without this
export LD_LIBRARY_PATH="$OPENBLAS_PATH/lib/:$CPYTHON_PATH/lib/:$NUMPY_PATH/lib/"
export PYTHON_BIN_PATH="$CPYTHON_PATH/bin/python3.12"
export PYTHON_INCLUDE_PATH="$CPYTHON_PATH/include/python3.12/"
export PYTHON_LIB_PATH="$CPYTHON_PATH/lib/python3.12/"
export PYTHON_INSTALL_PATH="$INSTALL_PATH/lib/python3.12/site-packages/"
export SSL_CERT_FILE="$CPYTHON_PATH/lib/python3.12/site-packages/pip/_vendor/certifi/cacert.pem"
chmod +x "$PYTHON_BIN_PATH"
elif [[ -f "$CPYTHON_PATH/include/Python.h" ]]; then
CPYTHON_PATH=$(cygpath $CPYTHON_PATH)
OPENBLAS_PATH=$(cygpath $OPENBLAS_PATH)
NUMPY_PATH=$(cygpath $NUMPY_PATH)
export PATH="$OPENBLAS_PATH:$CPYTHON_PATH:$NUMPY_PATH:$PATH"
export PYTHON_BIN_PATH="$CPYTHON_PATH/bin/python.exe"
export PYTHON_INCLUDE_PATH="$CPYTHON_PATH/include/"
export PYTHON_LIB_PATH="$CPYTHON_PATH/lib/"
export PYTHON_INSTALL_PATH="$INSTALL_PATH/lib/site-packages/"
export SSL_CERT_FILE="$CPYTHON_PATH/lib/pip/_vendor/certifi/cacert.pem"
fi
export PYTHONPATH="$PYTHON_INSTALL_PATH:$NUMPY_PATH/python/"
mkdir -p "$PYTHON_INSTALL_PATH"

export CFLAGS="-I$CPYTHON_PATH/include/ -I$PYTHON_LIB_PATH/include/python/ -L$CPYTHON_PATH/lib/ -L$CPYTHON_PATH/libs/"
export PYTHONNOUSERSITE=1
$PYTHON_BIN_PATH -m pip install --target=$PYTHON_LIB_PATH setuptools==67.6.1 pyyaml==6.0.1 typing_extensions==4.8.0

TOOLS="setuptools==67.6.1 pyyaml==6.0.1 typing_extensions==4.8.0"

if [[ $PLATFORM == $PLATFORM_HOST ]]; then
if [[ -f "$CPYTHON_PATH/include/python3.12/Python.h" ]]; then
# setup.py won't pick up the right libgfortran.so without this
export LD_LIBRARY_PATH="$OPENBLAS_PATH/lib/:$CPYTHON_PATH/lib/:$NUMPY_PATH/lib/"
export PYTHON_BIN_PATH="$CPYTHON_PATH/bin/python3.12"
export PYTHON_INCLUDE_PATH="$CPYTHON_PATH/include/python3.12/"
export PYTHON_LIB_PATH="$CPYTHON_PATH/lib/python3.12/"
export PYTHON_INSTALL_PATH="$INSTALL_PATH/lib/python3.12/site-packages/"
export SSL_CERT_FILE="$CPYTHON_PATH/lib/python3.12/site-packages/pip/_vendor/certifi/cacert.pem"
chmod +x "$PYTHON_BIN_PATH"
elif [[ -f "$CPYTHON_PATH/include/Python.h" ]]; then
CPYTHON_PATH=$(cygpath $CPYTHON_PATH)
OPENBLAS_PATH=$(cygpath $OPENBLAS_PATH)
NUMPY_PATH=$(cygpath $NUMPY_PATH)
export PATH="$OPENBLAS_PATH:$CPYTHON_PATH:$NUMPY_PATH:$PATH"
export PYTHON_BIN_PATH="$CPYTHON_PATH/bin/python.exe"
export PYTHON_INCLUDE_PATH="$CPYTHON_PATH/include/"
export PYTHON_LIB_PATH="$CPYTHON_PATH/lib/"
export PYTHON_INSTALL_PATH="$INSTALL_PATH/lib/site-packages/"
export SSL_CERT_FILE="$CPYTHON_PATH/lib/pip/_vendor/certifi/cacert.pem"
fi
export PYTHONPATH="$PYTHON_INSTALL_PATH:$NUMPY_PATH/python/"
mkdir -p "$PYTHON_INSTALL_PATH"

export CFLAGS="-I$CPYTHON_PATH/include/ -I$PYTHON_LIB_PATH/include/python/ -L$CPYTHON_PATH/lib/ -L$CPYTHON_PATH/libs/"
$PYTHON_BIN_PATH -m pip install --target=$PYTHON_LIB_PATH $TOOLS
else # cross-compile
export PYTHON_BIN_PATH="$CPYTHON_HOST_PATH/bin/python3.12"
chmod +x $PYTHON_BIN_PATH
$PYTHON_BIN_PATH -m pip install --target="$CPYTHON_HOST_PATH/lib/python3.12/" $TOOLS
fi
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please remove the if [[ $PLATFORM == $PLATFORM_HOST ]]; then and do something like this instead:

if ! $PYTHON_BIN_PATH -m pip install --target=$PYTHON_LIB_PATH $TOOLS; then
    chmod +x "$CPYTHON_HOST_PATH/bin/python3.12"
    export LD_LIBRARY_PATH="$LD_LIBRARY_PATH:$CPYTHON_HOST_PATH/lib/:$CPYTHON_HOST_PATH"
    chmod +x $CPYTHON_HOST_PATH/lib/python3.12/bin/*
    export PATH="$CPYTHON_HOST_PATH/lib/python3.12/bin/:$PATH"
    export PYTHON_BIN_PATH="python"
fi

Copy link
Contributor Author

@sbrunk sbrunk Jan 24, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more. 

if ! $PYTHON_BIN_PATH -m pip install --target=$PYTHON_LIB_PATH $TOOLS; then

fails (stopping the script) in the cross-compilation setting since PYTHON_LIB_PATH is not set at all in that case right now without arm cpython, that's why I put the whole block in a conditional.

The same is true i.e. for

export PYTHONPATH="$PYTHON_INSTALL_PATH:$NUMPY_PATH/python/"

since the PYTHON_INSTALL_PATH is not set.

But I think see the issue, it would break if we wanted to use both host and target cpython right?

What if we test for PYTHON_BIN_PATH to exist before running pip install?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Right, so instead of all that, let's simply put CPYTHON_PATH=CPYTHON_HOST_PATH before
if [[ -f "$CPYTHON_PATH/include/python3.12/Python.h" ]]; then


case $PLATFORM in
linux-x86)
Expand All @@ -115,9 +131,17 @@ case $PLATFORM in
export CC="gcc -m64"
export CXX="g++ -m64"
;;
macosx-*)
macosx-x86_64)
export CC="clang"
export CXX="clang++"
;;
macosx-arm64)
export CC="clang"
export CXX="clang++"
export CMAKE_OSX_ARCHITECTURES=arm64 # enable cross-compilation on a x86_64 host machine
export USE_MKLDNN=OFF
export USE_QNNPACK=OFF # not compatible with arm64 as of PyTorch 2.1.2
export CMAKE_OSX_DEPLOYMENT_TARGET=11.00 # minimum needed for arm64 support
;;
windows-x86_64)
if which ccache.exe; then
Expand Down
43 changes: 43 additions & 0 deletions pytorch/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -91,6 +91,8 @@
<buildResource>/${javacpp.platform.library.path}/</buildResource>
<buildResource>/org/bytedeco/openblas/${javacpp.platform}/</buildResource>
<buildResource>/org/bytedeco/cpython/${javacpp.platform}/</buildResource>
<!-- add host cpython for running setup.py when cross-compiling -->
<buildResource>/org/bytedeco/cpython/${os.name}-${os.arch}/</buildResource>
<buildResource>/org/bytedeco/mkldnn/${javacpp.platform}/</buildResource>
<buildResource>/org/bytedeco/numpy/${javacpp.platform}/</buildResource>
</buildResources>
Expand Down Expand Up @@ -135,4 +137,45 @@
</plugins>
</build>

<!-- cross-compilation profile for macosx-arm64 builds on macosx-x86_64 CI machines -->
<profiles>
<profile>
<id>pytorch-cross-compile</id>
<activation>
<property>
<name>javacpp.platform</name>
<value>macosx-arm64</value>
</property>
</activation>
<build>
<plugins>
<plugin>
<groupId>org.bytedeco</groupId>
<artifactId>javacpp</artifactId>
<dependencies>
<dependency>
<groupId>org.bytedeco</groupId>
<artifactId>cpython</artifactId>
<version>3.12.1-${project.parent.version}</version>
<classifier>${os.name}-${os.arch}</classifier>
</dependency>
<!-- we need to exclude target cpython until we have a macosx-arm64 build -->
<dependency>
<groupId>org.bytedeco</groupId>
<artifactId>numpy-platform</artifactId>
<version>1.26.3-${project.parent.version}</version>
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I suppose we need to do something similar in platform/pom.xml:

2024-01-24T00:31:14.1394220Z [ERROR] Failed to execute goal on project pytorch-platform: Could not resolve dependencies for project org.bytedeco:pytorch-platform:jar:2.1.2-1.5.10-SNAPSHOT: Could not find artifact org.bytedeco:cpython:jar:macosx-arm64:3.12.1-1.5.10-SNAPSHOT in sonatype-nexus-snapshots (https://oss.sonatype.org/content/repositories/snapshots) -> [Help 1]
2024-01-24T00:31:14.1403910Z org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal on project pytorch-platform: Could not resolve dependencies for project org.bytedeco:pytorch-platform:jar:2.1.2-1.5.10-SNAPSHOT: Could not find artifact org.bytedeco:cpython:jar:macosx-arm64:3.12.1-1.5.10-SNAPSHOT in sonatype-nexus-snapshots (https://oss.sonatype.org/content/repositories/snapshots)

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actually, I don't think we need to include those optional dependencies at all...

<exclusions>
<exclusion>
<groupId>org.bytedeco</groupId>
<artifactId>cpython</artifactId>
</exclusion>
</exclusions>
</dependency>
</dependencies>
</plugin>
</plugins>
</build>
</profile>
</profiles>

</project>
Loading