Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

WIP: Prevent path conflict in builds #1508

Closed
wants to merge 11 commits into from

Conversation

AyodeAwe
Copy link
Contributor

@AyodeAwe AyodeAwe commented Mar 25, 2024

Fixes #1528.

Contributes to rapidsai/build-planning#54 and rapidsai/build-planning#56.

Related to rapidsai/rapids-cmake#592

Notes for Reviewers

This is not ready for review yet.

Related conversations:

@AyodeAwe AyodeAwe requested a review from a team as a code owner March 25, 2024 19:07
@AyodeAwe AyodeAwe added non-breaking Non-breaking change improvement Improvement / enhancement to an existing function ci labels Mar 25, 2024
@jameslamb
Copy link
Member

jameslamb commented Apr 19, 2024

This is currently failing with the following conflicts.

(I've included just 1 example of each type below)

(CUDA 11.8 build) (CUDA 12.2 build)

  1. fmt headers in include/fmt (conflicting packages: conda-forge/fmt, librmm)
This transaction has incompatible packages due to a shared path.
  packages: conda-forge/linux-aarch64::fmt-10.2.1-h2a328a1_0, file:///tmp/conda-bld-output/linux-aarch64::librmm-24.06.00a16-cuda12_240419_g9dfd9070_16
  path: 'include/fmt/chrono.h'
  1. fmt builds scripts in lib/cmake/fmt/ *(conflicting packages: conda-forge/fmt, librmm`)*
This transaction has incompatible packages due to a shared path.
  packages: conda-forge/linux-aarch64::fmt-10.2.1-h2a328a1_0, file:///tmp/conda-bld-output/linux-aarch64::librmm-24.06.00a16-cuda12_240419_g9dfd9070_16
  path: 'lib/cmake/fmt/fmt-targets.cmake'
  1. fmt pkgconfig script (conflicting packages: conda-forge/fmt, librmm)
This transaction has incompatible packages due to a shared path.
  packages: conda-forge/linux-aarch64::fmt-10.2.1-h2a328a1_0, file:///tmp/conda-bld-output/linux-aarch64::librmm-
  path: 'lib/pkgconfig/fmt.pc'
  1. spdlog headers (conflicting packages: conda-forge/fmt, librmm)
This transaction has incompatible packages due to a shared path.
   packages: conda-forge/linux-aarch64::spdlog-1.12.0-h6b8df57_2, file:///tmp/conda-bld-output/linux-aarch64::librmm-24.06.00a16-cuda12_240419_g9dfd9070_16
   path: 'include/spdlog/async.h'
  1. spdlog build scripts (conflicting packages: conda-forge/fmt, librmm)
This transaction has incompatible packages due to a shared path.
  packages: conda-forge/linux-aarch64::spdlog-1.12.0-h6b8df57_2, file:///tmp/conda-bld-output/linux-aarch64::librmm-24.06.00a16-cuda12_240419_g9dfd9070_16
  path: 'lib/cmake/spdlog/spdlogConfig.cmake'
  1. spdlog pkgconfig script (conflicting packages: conda-forge/fmt, librmm)
This transaction has incompatible packages due to a shared path.
  packages: conda-forge/linux-aarch64::spdlog-1.12.0-h6b8df57_2, file:///tmp/conda-bld-output/linux-aarch64::librmm-24.06.00a16-cuda12_240419_g9dfd9070_16
  path: 'lib/pkgconfig/spdlog.pc'

@github-actions github-actions bot added the conda label Apr 22, 2024
@jameslamb jameslamb marked this pull request as draft April 22, 2024 21:21
@jameslamb jameslamb changed the title Prevent path conflict in builds WIP: Prevent path conflict in builds Apr 22, 2024
@jameslamb jameslamb added 2 - In Progress Currently a work in progress 5 - DO NOT MERGE Hold off on merging; see PR for details labels Apr 22, 2024
@github-actions github-actions bot removed the conda label Apr 23, 2024
@harrism
Copy link
Member

harrism commented Apr 23, 2024

I can now see this clobbering blocking RMM PRs such as #1537. Can't build RMM C++ in CI due to path conflicts for spdlog and fmt. e.g.

ClobberWarning: This transaction has incompatible packages due to a shared path.
  packages: conda-forge/linux-64::fmt-10.2.1-h00ab1b0_0, file:///tmp/conda-bld-output/linux-64::librmm-24.06.00a20-cuda11_240423_ga4d6c965_20
  path: 'include/fmt/args.h'

@jameslamb
Copy link
Member

@harrism I don't think the clobbering stuff is what's causing that PR to fail (although it is generating thousands of lines of scary-looking logs 😅 ).

#1537 is failing because it removes a file but not the corresponding test in the conda recipe.

+ test -f /opt/conda/conda-bld/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_pl/include/rmm/thrust_rmm_allocator.h
WARNING: Tests failed for librmm-24.06.00a20-cuda12_240423_ga4d6c965_20.tar.bz2 - moving package to /opt/conda/conda-bld/broken

Remove this test there:

- test -f $PREFIX/include/rmm/thrust_rmm_allocator.h

@harrism
Copy link
Member

harrism commented Apr 23, 2024

Oh how did you even find that? I just saw all the ClobberWarnings.

@harrism
Copy link
Member

harrism commented Apr 23, 2024

Any way to make those failures fail with the text "error:"? This is usually what I search for in the logs.

@jameslamb
Copy link
Member

Oh how did you even find that? I just saw all the ClobberWarnings.

I went straight to the end of the log and read back up from there until I saw something problematic. A lot of the CI scripts across RAPIDS have set -e -u -o pipefail set, so they tend to fail at the first place where something goes wrong.

I also had just looked at these tests in the conda recipe today, in the process of testing for this PR, so had some pattern recognition for what it looked like when they failed.

Any way to make those failures fail with the text "error:"? This is usually what I search for in the logs.

Not that I'm aware of. That comes from within conda itself, I don't think we can control it.

It does say "fail" which I usually search for along with "error".

...
WARNING: Tests failed for librmm-24.06.00a20-cuda11_240423_ga4d6c965_20.tar.bz2 - moving package to /opt/conda/conda-bld/broken
...
TESTS FAILED: librmm-24.06.00a20-cuda11_240423_ga4d6c965_20.tar.bz2
[rapids-conda-retry] conda returned exit code: 1

(build link)

@github-actions github-actions bot added the CMake label Apr 24, 2024
@harrism
Copy link
Member

harrism commented May 9, 2024

Will this make it into 24.06?

@jameslamb
Copy link
Member

Will this make it into 24.06?

short answer

Not unless we decide that there's an urgent need for it.

long answer

The root cause of these fmt and spdlog clobbering issues across RAPIDS is "RAPIDS is carrying around patches to those libraries, so rapids-cmake always downloads them, and it places them at likely-to-cause-conflicts paths like include/fmt".

I'd started pursuing a short-term fix (upgrade to newer versions of fmt and spdlog that don't need the patches), described in rapidsai/build-planning#56 and tested over in #1544.

Stopped short of trying to roll that out across all of RAPIDS conda packages, because doing it might lead to RAPIDS packages conflicting with conda itself and other packages from conda-forge. @bdice summarized that well here: rapidsai/build-planning#56 (comment)

At that point, we paused on this to work towards other packaging priorities for this release: rapidsai/build-planning#54 (comment)

I'd like to pick up a more permanent solution (RAPIDS redistributing these things when necessary, via its own conda package built from rapids-cmake) in the next release cycle.

cc @mmccarty for visibility

@jameslamb jameslamb changed the base branch from branch-24.06 to branch-24.08 May 21, 2024 15:48
@harrism
Copy link
Member

harrism commented May 22, 2024

Thanks. Moving to backlog.

@jameslamb
Copy link
Member

This work is paused, in favor of pursuing a better long-term solution in the future. Closing this PR for now.

Subscribe to rapidsai/build-planning#54 and rapidsai/build-planning#56 for updates.

@jameslamb jameslamb closed this Jul 18, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
2 - In Progress Currently a work in progress 5 - DO NOT MERGE Hold off on merging; see PR for details ci CMake conda improvement Improvement / enhancement to an existing function non-breaking Non-breaking change
Projects
Status: Done
Development

Successfully merging this pull request may close these issues.

[BUG] Conda package for rmm 24.04.00 clobbering files from fmt and spdlog
3 participants