Create CMake option `onnxruntime_USE_VCPKG` #21348

luncliff · 2024-07-13T18:15:41Z

Changes

CMake option onnxruntime_USE_VCPKG. It will be used in the vcpkg port
- Unit test may fail because this option leads to a mixture of unexpected external library versions.
  Especially ONNX, Protobuf, and Flatbuffers version can be different
Overhaul of onnxruntime_external_deps.cmake
- Make FetchContent_Declare to try find_package.
  See https://cmake.org/cmake/help/latest/guide/using-dependencies/index.html
- Relocated FetchContent_Declare and FetchContent_MakeAvailable(or onnxruntime_fetchcontent_makeavailable) to closer lines.
  It was too hard to navigate the entire file to search related sections...
- Alias IMPORTED targets like build targets (e.g. ONNX::onnx --> onnx)

# The script uses `find_package` with the changes.
# In this case, use vcpkg to search dependencies
# See https://cmake.org/cmake/help/latest/guide/using-dependencies/index.html
include(external/onnxruntime_external_deps.cmake)

Create CMakePresets.json and presets to run vcpkg in manifest mode
- Currently, it's NOT for training build
- Main triplets are x64-windows and x64-osx

Push-Location "cmake"
    cmake --preset "x64-windows-vcpkg"
    cmake --build --preset "x64-windows-vcpkg-debug"
Pop-Location

pushd "cmake"
    cmake --preset "x64-osx-vcpkg"
    cmake --build --preset "x64-osx-vcpkg-debug"
popd

Updated tools/ci_build/build.py
- --use_vcpkg option: it needs CMAKE_TOOLCHAIN_FILE with vcpkg.cmake toolchain script
- --compile_no_warning_as_error is recommended because library version differences will cause unexpected compiler warnings

python ./tools/ci_build/build.py \
    --compile_no_warning_as_error \
    --use_vcpkg \
    --cmake_extra_defines "CMAKE_TOOLCHAIN_FILE:FILEPATH=${VCPKG_ROOT}/scripts/buildsystems/vcpkg.cmake" \
    --cmake_extra_defines "VCPKG_TARGET_TRIPLET=..."

Created Job Vcpkg for Windows and macOS
- Show how to setup and use vcpkg.
  Similar to the CMakePresets.json usage

Motivation and Context

Future Works?

More feature coverage with the vcpkg supported libraries

CUDA feature support
Training feature support

snnn · 2024-07-15T21:25:59Z

I think cmake is a great choice.
Which cmake triplet this PR is going to support?

Besides, we might decouple the work. You may create a vcpkg config file, then install all the packages via "vcpkg install", then let cmake search installed packages there when running ONNX Runtime's build.py script. Then in theory you don't need to modify any ONNX Runtime's cmake file. No matter which way you choose to go, we always greatly appreciate your contribution.

luncliff · 2024-07-21T13:29:02Z

36919b0 New GitHub Actions jobs for vcpkg build

6227f5f Trying to merge onnxruntime_vcpkg_deps.cmake into onnxruntime_external_deps.cmake using FIND_PACKAGE_ARGS .
Currently I have some problems with pb.h,pb.cc generation

cmake/vcpkg-configuration.json

* microsoft/vcpkg#39881

* Detect protoc path for build.py * Run vcpkg for each build.py run Reorganized vcpkg install folder and CMake build directory

* CUDA build will be tried next time... * Remove onnxruntime_vcpkg_deps.cmake

luncliff · 2024-07-24T18:00:11Z

Updated note and title.

Dropped CUDA feature support in this PR.
I felt like making too huge changes here. I would make another PR for other features

* add dropped changes

snnn · 2024-07-24T18:14:07Z

/azp run Big Models, Linux Android Emulator QNN CI Pipeline, Linux CPU CI Pipeline, Linux CPU Minimal Build E2E CI Pipeline, Linux GPU CI Pipeline, Linux GPU TensorRT CI Pipeline, Linux OpenVINO CI Pipeline, Linux QNN CI Pipeline, MacOS CI Pipeline

snnn · 2024-07-24T18:14:16Z

/azp run Windows ARM64 QNN CI Pipeline, Windows CPU CI Pipeline, Windows GPU CI Pipeline, Windows GPU TensorRT CI Pipeline, Windows x64 QNN CI Pipeline, onnxruntime-binary-size-checks-ci-pipeline, orttraining-linux-ci-pipeline, orttraining-linux-gpu-ci-pipeline, orttraining-ortmodule-distributed

azure-pipelines · 2024-07-24T18:14:49Z

Azure Pipelines successfully started running 9 pipeline(s).

azure-pipelines · 2024-07-24T18:14:51Z

Azure Pipelines successfully started running 9 pipeline(s).

snnn · 2024-07-24T18:30:40Z

We probably need to wait for the next absl release.

luncliff · 2024-07-24T18:37:25Z

absl::demangle_rust was not intended one. I saw absl::demangle_internal in my fork, but not the one in the log. I will check again

snnn · 2024-08-15T14:54:10Z

/azp run Big Models, Linux Android Emulator QNN CI Pipeline, Linux CPU CI Pipeline, Linux CPU Minimal Build E2E CI Pipeline, Linux GPU CI Pipeline, Linux GPU TensorRT CI Pipeline, Linux OpenVINO CI Pipeline, Linux QNN CI Pipeline, MacOS CI Pipeline

snnn · 2024-08-15T14:54:43Z

/azp run Windows ARM64 QNN CI Pipeline, Windows CPU CI Pipeline, Windows GPU TensorRT CI Pipeline, Windows x64 QNN CI Pipeline, onnxruntime-binary-size-checks-ci-pipeline, orttraining-linux-ci-pipeline, orttraining-linux-gpu-ci-pipeline, orttraining-ortmodule-distributed

azure-pipelines · 2024-08-15T14:54:53Z

Azure Pipelines successfully started running 9 pipeline(s).

azure-pipelines · 2024-08-15T14:55:19Z

Azure Pipelines successfully started running 8 pipeline(s).

luncliff · 2024-09-06T04:50:18Z

Hi, is there a plan for this PR?
I'm going to restart discussion in vcpkg, so it will be helpful if I know the milestone of the project.

Any feedback will be welcomed. Thanks 😊

snnn · 2024-09-09T17:51:06Z

Sorry I was on vacation last month. Just came back.

luncliff · 2024-09-09T17:55:04Z

Probably with the latest vcpkg release. I will check it within days

…

On Tue, Sep 10, 2024, 02:52 Changming Sun ***@***.***> wrote: ***@***.**** commented on this pull request. ------------------------------ In cmake/vcpkg.json <#21348 (comment)> : > + "license": "MIT", + "dependencies": [ + "abseil", + { + "name": "boost-config", + "version>=": "1.82.0" + }, + { + "name": "boost-mp11", + "version>=": "1.82.0" + }, + "cpuinfo", + "cxxopts", + "date", + "dlpack", + "eigen3", How can we avoid depending on your personal repos? — Reply to this email directly, view it on GitHub <#21348 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ADLSYE5XMRBU5ENOG5QU3DDZVXN6LAVCNFSM6AAAAABK2O3XMGVHI2DSMVQWIX3LMV43YUDVNRWFEZLROVSXG5CSMV3GSZLXHMZDEOJQGU3TOMZYGE> . You are receiving this because you authored the thread.Message ID: ***@***.***>

cmake/external/onnxruntime_external_deps.cmake

* https://github.com/microsoft/vcpkg/releases/tag/2024.08.23 * remove abseil, eigen3 in configuration

* cmake: use eigen from FetchContent * ONNX will be used with FetchContent Current vcpkg upstream doesn't support onnx 1.16.0+. Here, we will mix FetchContent and vcpkg-supported find_package

cmake/vcpkg-configuration.json

snnn · 2024-09-10T17:05:45Z

/azp run Big Models, Linux Android Emulator QNN CI Pipeline, Linux CPU CI Pipeline, Linux CPU Minimal Build E2E CI Pipeline, Linux GPU CI Pipeline, Linux GPU TensorRT CI Pipeline

snnn · 2024-09-10T17:05:52Z

/azp run Linux OpenVINO CI Pipeline, Linux QNN CI Pipeline, MacOS CI Pipeline, ONNX Runtime Web CI Pipeline, Windows ARM64 QNN CI Pipeline, Windows CPU CI Pipeline, Windows GPU CUDA CI Pipeline

snnn · 2024-09-10T17:06:00Z

/azp run Linux OpenVINO CI Pipeline, Linux QNN CI Pipeline, MacOS CI Pipeline, ONNX Runtime Web CI Pipeline, Windows ARM64 QNN CI Pipeline, Windows CPU CI Pipeline, Windows GPU CUDA CI Pipeline

azure-pipelines · 2024-09-10T17:06:14Z

Azure Pipelines successfully started running 6 pipeline(s).

azure-pipelines · 2024-09-10T17:06:25Z

Azure Pipelines successfully started running 7 pipeline(s).

azure-pipelines · 2024-09-10T17:06:26Z

Azure Pipelines successfully started running 7 pipeline(s).

snnn · 2024-09-10T20:59:31Z

/azp run Windows GPU DML CI Pipeline, Windows GPU Doc Gen CI Pipeline, Windows GPU TensorRT CI Pipeline, Windows x64 QNN CI Pipeline, onnxruntime-binary-size-checks-ci-pipeline, orttraining-linux-ci-pipeline, orttraining-linux-gpu-ci-pipeline

azure-pipelines · 2024-09-10T21:00:03Z

Azure Pipelines successfully started running 7 pipeline(s).

snnn · 2024-09-10T23:39:34Z

Thank you!

luncliff · 2024-09-11T01:11:10Z

Thank you, @snnn. I will submit another PR if microsoft/vcpkg needs it.

luncliff added 3 commits July 14, 2024 01:51

build: support vcpkg in cmake-presets

d0e440b

use std::string for abseil::raw_hash_map find

c41c244

fix case mismatch

6223414

luncliff marked this pull request as draft July 13, 2024 18:15

luncliff changed the title ~~Create files and CMake option to support vcpkg and CMakePresets.json to test it~~ Create files and CMake option to support vcpkg Jul 13, 2024

luncliff added 2 commits July 14, 2024 03:24

use later version of vcpkg

ff97583

cmake: check option in CUDA provider file

fabe7d9

luncliff added 4 commits July 21, 2024 21:31

build.py: create --use_vcpkg option and workflow for Windows

4b8bc45

gh-actions: jobs for each vcpkg triplet

36919b0

vcpkg: update manifest

310aaa6

cmake: try merge of onnxruntime_external_*.cmake

6227f5f

snnn reviewed Jul 22, 2024

View reviewed changes

cmake/vcpkg-configuration.json Outdated Show resolved Hide resolved

luncliff added 3 commits July 24, 2024 22:44

python: apply lintrunner suggestions

091ea11

ci: update vcpkg-configuration

ba5bb5c

* microsoft/vcpkg#39881

ci: use vcpkg installed flatc, protoc

e37f9d6

* Detect protoc path for build.py * Run vcpkg for each build.py run Reorganized vcpkg install folder and CMake build directory

luncliff changed the title ~~Create files and CMake option to support vcpkg~~ Create CMake option onnxruntime_USE_VCPKG Jul 24, 2024

cmake: remove unused parts

68920d9

* CUDA build will be tried next time... * Remove onnxruntime_vcpkg_deps.cmake

luncliff marked this pull request as ready for review July 24, 2024 17:56

luncliff requested a review from a team as a code owner July 24, 2024 17:56

cmake: use vcpkg 2024.07.12

d97ec19

* add dropped changes

luncliff marked this pull request as draft July 24, 2024 18:37

cmake: fix preset cacheVariables

4faeeee

luncliff mentioned this pull request Aug 9, 2024

[onnxruntime] Update to v1.18.1, redesign portfile luncliff/vcpkg-registry#227

Merged

luncliff requested a review from snnn August 15, 2024 13:49

luncliff mentioned this pull request Aug 25, 2024

[onnxruntime] Create features tensorrt, directml, openvino luncliff/vcpkg-registry#237

Merged

snnn reviewed Sep 9, 2024

View reviewed changes

cmake/external/onnxruntime_external_deps.cmake Show resolved Hide resolved

luncliff added 2 commits September 10, 2024 21:37

vcpkg: use 2024.08.23

dba1862

* https://github.com/microsoft/vcpkg/releases/tag/2024.08.23 * remove abseil, eigen3 in configuration

vcpkg: remove eigen3, onnx in manifest

a67dd2d

* cmake: use eigen from FetchContent * ONNX will be used with FetchContent Current vcpkg upstream doesn't support onnx 1.16.0+. Here, we will mix FetchContent and vcpkg-supported find_package

snnn reviewed Sep 10, 2024

View reviewed changes

cmake/vcpkg-configuration.json Show resolved Hide resolved

snnn approved these changes Sep 10, 2024

View reviewed changes

snnn merged commit f633caa into microsoft:main Sep 10, 2024
71 checks passed

luncliff deleted the support/vcpkg branch September 11, 2024 01:11

cloudhan mentioned this pull request Sep 11, 2024

fix: respect onnxruntime_USE_PREINSTALLED_EIGEN option #22057

Open

jordanozang mentioned this pull request Sep 23, 2024

CMake exports for static onnxruntime #22173

Draft

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Create CMake option `onnxruntime_USE_VCPKG` #21348

Create CMake option `onnxruntime_USE_VCPKG` #21348

luncliff commented Jul 13, 2024 •

edited

Loading

snnn commented Jul 15, 2024

luncliff commented Jul 21, 2024

luncliff commented Jul 24, 2024

snnn commented Jul 24, 2024

snnn commented Jul 24, 2024

azure-pipelines bot commented Jul 24, 2024

azure-pipelines bot commented Jul 24, 2024

snnn commented Jul 24, 2024

luncliff commented Jul 24, 2024

snnn commented Aug 15, 2024

snnn commented Aug 15, 2024

azure-pipelines bot commented Aug 15, 2024

azure-pipelines bot commented Aug 15, 2024

luncliff commented Sep 6, 2024

snnn commented Sep 9, 2024

luncliff commented Sep 9, 2024 via email

snnn commented Sep 10, 2024

snnn commented Sep 10, 2024

snnn commented Sep 10, 2024

azure-pipelines bot commented Sep 10, 2024

azure-pipelines bot commented Sep 10, 2024

azure-pipelines bot commented Sep 10, 2024

snnn commented Sep 10, 2024

azure-pipelines bot commented Sep 10, 2024

snnn commented Sep 10, 2024

luncliff commented Sep 11, 2024

Create CMake option onnxruntime_USE_VCPKG #21348

Create CMake option onnxruntime_USE_VCPKG #21348

Conversation

luncliff commented Jul 13, 2024 • edited Loading

Changes

Motivation and Context

Future Works?

snnn commented Jul 15, 2024

luncliff commented Jul 21, 2024

luncliff commented Jul 24, 2024

snnn commented Jul 24, 2024

snnn commented Jul 24, 2024

azure-pipelines bot commented Jul 24, 2024

azure-pipelines bot commented Jul 24, 2024

snnn commented Jul 24, 2024

luncliff commented Jul 24, 2024

snnn commented Aug 15, 2024

snnn commented Aug 15, 2024

azure-pipelines bot commented Aug 15, 2024

azure-pipelines bot commented Aug 15, 2024

luncliff commented Sep 6, 2024

snnn commented Sep 9, 2024

luncliff commented Sep 9, 2024 via email

snnn commented Sep 10, 2024

snnn commented Sep 10, 2024

snnn commented Sep 10, 2024

azure-pipelines bot commented Sep 10, 2024

azure-pipelines bot commented Sep 10, 2024

azure-pipelines bot commented Sep 10, 2024

snnn commented Sep 10, 2024

azure-pipelines bot commented Sep 10, 2024

snnn commented Sep 10, 2024

luncliff commented Sep 11, 2024

Create CMake option `onnxruntime_USE_VCPKG` #21348

Create CMake option `onnxruntime_USE_VCPKG` #21348

luncliff commented Jul 13, 2024 •

edited

Loading