Dependency TODOs: enablements & clean-ups

[h-vetinari]

From looking at the list of third-party deps, we could still do better with:

• azure bindings
  • unix
  • windows: [C++] Error when building Arrow-CPP w/ Azure Filesystem for Conda-Forge apache/arrow#41990
• jemalloc (not possible?)
• opentelemetry
  • Build shared libs, add run-export, register in global pinning cpp-opentelemetry-sdk-feedstock#38
  • WIP: Build with opentelemetry support #1048
• orc for windows
  • Build for windows orc-feedstock#54
  • enabled for libarrow (as of db46b2c)
  • [Python] BUG: Reading ORC segfaults on windows (if TZDIR isn't set) apache/arrow#36026
  • ORC-1684: [C++] Find tzdb without TZDIR when in conda-environments apache/orc#1882
  • [15.x] Rebuild for orc 2.0.1; enable orc-on-windows also for pyarrow #1086
  • don't fail tzdb availability check prematurely #1424
  • enable orc-on-windows also for pyarrow pyarrow-feedstock#122
  • GH-41755: [C++][ORC] Ensure setting detected ORC version apache/arrow#41767
• nlohmann_json (as of c548fc2)
• substrait (not yet in conda-forge)
• ucx (as of c548fc2)

Switch to shared builds for (if/once available):

• google-cloud-cpp (win; not yet available in cf)
• gflags (win; available, but apparently symbols aren't needed; clarified as of bda7682)
• glog (win; available, but apparently symbols aren't needed; clarified as of bda7682)
• grpc (win; available as of 1.55, which requires protobuf >=4.23; fixed as of 025b77a)
• thrift (needed fixes for CMake metadata; fully cleaned up as of 2af3d60)

Bindings:

• skyhook (needs librados)
• tensorflow (though we should use libtensorflow, if at all, because it's a very heavy dependency to pull in)

[xhochy]

We cannot unvendor jemalloc as the build is with special options. Unverdoring it would have an impact on the default allocator.

[jakirkham]

Would it be possible to just build a variant of the jemalloc package with options needed?

[jakirkham]

Would not count on ucx being supported on other architectures OSes. Would also like to understand why that is relevant

[h-vetinari]

Would also like to understand why that is relevant

I have a maximalist approach to packaging the bindings offered by a given library -- if the library offers integration for it, why shouldn't we package it?¹

Users rely on conda-forge because a lot of these packages are really hard to build from source, so if a binding is not supported by us, it's effectively unusuable (even moreso if the distributed wheels also turn it off).

Footnotes

1. whether to break a package into separate outputs is a separate question ↩

[jakirkham]

Sorry I should clarify, what I mean is why are other OSes relevant? Asking since ucx likely will remain Linux only (possibly macOS at some point and basically no chance of Windows support)

[h-vetinari]

Sorry I should clarify, what I mean is why are other OSes relevant? Asking since ucx likely will remain Linux only (possibly macOS at some point and basically no chance of Windows support)

Not relevant indeed, unless they become: a) available for the respective platform & b) are supported by arrow on that platform.