[bug] Impossible to build latest Arrow with parquet?

[kevin-mcvey]

Environment details

• Operating System+version: Ubuntu 22.04
• Compiler+version: gcc 11.4.0
• Conan version: 2.0.9
• Python version: 3.10.12

Steps to reproduce

First of all, thank you all very much for your hard work on Conan. I love the tool and get a ton of use out of it. :)

Onto my issue: I'm trying to install Arrow with the with_parquet=True option on the latest version of Conan. However, this appears to be impossible due to mismatched requirements. To demonstrate this, I've got a minimal Conanfile in a fresh Ubuntu 22.04 installation:

from conan import ConanFile

class MyConan(ConanFile):
    name = 'MyProject'
    version = '1.0.0'
    description = 'A test project'

    generators = ('CMakeDeps')
    build_policy = 'missing'

    requires = (
        'arrow/12.0.1',
        'boost/1.81.0',
        'thrift/0.13.0',
    )

    def configure(self):
        self.options['arrow'].parquet = True
        self.options['arrow'].with_boost = True

I then attempt to install: conan install conanfile.py --build=missing

Note that Boost and Thrift are added as requirements. This is for the following reasons:

• If I do not include Boost (and I leave off with_boost = True), Arrow will fail to build with a message of Could NOT find Boost (missing: Boost_INCLUDE_DIR) (Required is at least version "1.58")
• If I do not include Thrift, Arrow will fail to build with a message of Could NOT find ThriftAlt: Found unsuitable version "", but required is at least "0.11.0" (found ThriftAlt_LIB-NOTFOUND)

Unfortunately, adding Thrift does not appear to resolve the latter issue. I'm hoping there's something obvious I'm missing. Thank you so much in advance for your support!

Logs

-- Building without OpenSSL support. Minimum OpenSSL version 1.0.2 required.
CMake Warning at cmake_modules/FindThriftAlt.cmake:56 (find_package):
  By not providing "FindThrift.cmake" in CMAKE_MODULE_PATH this project has
  asked CMake to find a package configuration file provided by "Thrift", but
  CMake did not find one.

  Could not find a package configuration file provided by "Thrift" (requested
  version 0.11.0) with any of the following names:

    ThriftConfig.cmake
    thrift-config.cmake

  Add the installation prefix of "Thrift" to CMAKE_PREFIX_PATH or set
  "Thrift_DIR" to a directory containing one of the above files.  If "Thrift"
  provides a separate development package or SDK, be sure it has been
  installed.
Call Stack (most recent call first):
  cmake_modules/ThirdpartyToolchain.cmake:286 (find_package)
  cmake_modules/ThirdpartyToolchain.cmake:1646 (resolve_dependency)
  CMakeLists.txt:506 (include)


CMake Error at /usr/share/cmake-3.22/Modules/FindPackageHandleStandardArgs.cmake:230 (message):
  Could NOT find ThriftAlt: Found unsuitable version "", but required is at
  least "0.11.0" (found ThriftAlt_LIB-NOTFOUND)
Call Stack (most recent call first):
  /usr/share/cmake-3.22/Modules/FindPackageHandleStandardArgs.cmake:592 (_FPHSA_FAILURE_MESSAGE)
  cmake_modules/FindThriftAlt.cmake:164 (find_package_handle_standard_args)
  cmake_modules/ThirdpartyToolchain.cmake:286 (find_package)
  cmake_modules/ThirdpartyToolchain.cmake:1646 (resolve_dependency)
  CMakeLists.txt:506 (include)


-- Configuring incomplete, errors occurred!
See also "/root/.conan2/p/b/arrow7126f5c93982a/b/build/Release/CMakeFiles/CMakeOutput.log".
See also "/root/.conan2/p/b/arrow7126f5c93982a/b/build/Release/CMakeFiles/CMakeError.log".

arrow/12.0.1: ERROR:
Package '462c3467d99b24bd66999e639bd9e8a51f67fa22' build failed

(I can attach more complete logs if needed!)

[memsharded]

Hi @kevin-mcvey

First of all, thank you all very much for your hard work on Conan. I love the tool and get a ton of use out of it. :)

Thanks for your feedback and your kind words! :) Keep the feedback coming and we will keep trying our best to further improve and support it.

Onto my issue: I'm trying to install Arrow with the with_parquet=True option on the latest version of Conan.

I think the first step could be not adding the dependencies directly. The arrow recipe contains:

def requirements(self):
        if self.options.with_thrift:
            self.requires("thrift/0.17.0")
        if self.options.with_protobuf:
            self.requires("protobuf/3.21.9")
        if self.options.with_jemalloc:
            self.requires("jemalloc/5.3.0")
        if self.options.with_mimalloc:
            self.requires("mimalloc/1.7.6")
        if self.options.with_boost:
            self.requires("boost/1.81.0")

That means, it has options to control its conditional dependencies, as long as the options are activated, it shouldn't be necessary to add the dependencies yourself. If this is not working, it is possible that the recipe is not working correctly. ConanCenter can only build the default configurations, so other options values different to the default are not really tested in ConanCenter, and relies on user testing.

I am trying:

class CoralCConan(ConanFile):
    name = 'MyProject'
    version = '1.0.0'
    description = 'A test project'

    generators = ('CMakeDeps')

    requires = ('arrow/12.0.1')

    def configure(self):
        self.options['arrow'].parquet = True
        self.options["arrow"].with_thrift = True

And what I get is a conflict:

ERROR: Version conflict: thrift/0.17.0->boost/1.82.0, arrow/12.0.1->boost/1.81.0.

Was this the reason that made you add the requires? Is it possible that it is needed to fix ConanCenter to align the versions used.

[kevin-mcvey]

Hi @memsharded , thank you for the lightning-fast reply!

Was this the reason that made you add the requires?

Yes, that's exactly why I started adding packages directly to the requires list. I followed four steps in order to arrive at this particular conanfile:

Step 1:
I first tried to install arrow/12.0.1 directly with the with_parquet option set to true:

# Same name, version, generators, and build policy as before

    requires = (
        'arrow/12.0.1',
    )

    def configure(self):
        self.options['arrow'].parquet = True

With this configuration, conan install conanfile.py --build=missing fails with the following log:

CMake Error at /usr/share/cmake-3.22/Modules/FindPackageHandleStandardArgs.cmake:230 (message):
  Could NOT find Boost (missing: Boost_INCLUDE_DIR) (Required is at least
  version "1.58")
Call Stack (most recent call first):
  /usr/share/cmake-3.22/Modules/FindPackageHandleStandardArgs.cmake:594 (_FPHSA_FAILURE_MESSAGE)
  /usr/share/cmake-3.22/Modules/FindBoost.cmake:2360 (find_package_handle_standard_args)
  cmake_modules/ThirdpartyToolchain.cmake:286 (find_package)
  cmake_modules/ThirdpartyToolchain.cmake:1190 (resolve_dependency)
  CMakeLists.txt:506 (include)

Step 2:
I then tried adding the with_boost option:

# Same name, version, generators, and build policy as before

    requires = (
        'arrow/12.0.1',
    )

    def configure(self):
        self.options['arrow'].parquet = True
        self.options['arrow'].with_boost = True

However that yields the following error log:

CMake Error at /usr/share/cmake-3.22/Modules/FindPackageHandleStandardArgs.cmake:230 (message):
  Could NOT find ThriftAlt: Found unsuitable version "", but required is at
  least "0.11.0" (found ThriftAlt_LIB-NOTFOUND)
Call Stack (most recent call first):
  /usr/share/cmake-3.22/Modules/FindPackageHandleStandardArgs.cmake:592 (_FPHSA_FAILURE_MESSAGE)
  cmake_modules/FindThriftAlt.cmake:164 (find_package_handle_standard_args)
  cmake_modules/ThirdpartyToolchain.cmake:286 (find_package)
  cmake_modules/ThirdpartyToolchain.cmake:1646 (resolve_dependency)
  CMakeLists.txt:506 (include)

Step 3:
So I then tried adding with_thrift:

# Same name, version, generators, and build policy as before

    requires = (
        'arrow/12.0.1',
    )

    def configure(self):
        self.options['arrow'].parquet = True
        self.options['arrow'].with_boost = True
        self.options['arrow'].with_thrift = True

However that yields the version conflict you mentioned above.

ERROR: Version conflict: thrift/0.17.0->boost/1.82.0, arrow/12.0.1->boost/1.81.0.

Step 4:
In response, I started trying to manually set version numbers until I could get something to click. No such luck I'm afraid. :)

[memsharded]

It seems arrow got some very recent changes, it might be worth trying again. @danimtb will be having a look, he did some of these changes.

[danimtb]

Hi @kevin-mcvey,

I worked recently updating the arrow recipe on conan center at conan-io/conan-center-index#19380 and it should be working fine with conan v2.

I have tested your consumer recipe and got to the same version conflict of boost you pointed out.
Although this is not desirable, we didn't spot this issue because we only build the default options configurations in conan center (with_boost: True, parquet: False, with_thrift: False).

Anyway, since this is a pretty common scenario, here you can see how to overcome the version conflict of boost for your case.

conanfile.py

from conan import ConanFile

class CoralCConan(ConanFile):
    name = 'MyProject'
    version = '1.0.0'
    description = 'A test project'

    generators = 'CMakeDeps'

    def requirements(self):
        self.requires('arrow/12.0.1')
        self.requires('boost/1.82.0', override=True)

    def configure(self):
        self.options['arrow'].with_boost = True
        self.options['arrow'].parquet = True
        self.options['arrow'].with_thrift = True

Using override, you can force the version of boost, see: https://docs.conan.io/2/reference/conanfile/methods/requirements.html#override

Also, since you require a package that has different option values than the default ones, you will need to build it from sources (this configuration is not available as package in conan center):

conan install conanfile --build=arrow/12.0.1

---

I have also opened a new PR on conan-center-index to update the version of the boost requirement in the arrow recipe conan-io/conan-center-index#19465

It will avoid adding the override=True in the boost requirement eventually when the PR is merged 😄

Thank you for reporting and hope it helps!

[kevin-mcvey]

Wow, thank you so much for the assistance @danimtb ! Confirmed that using the override flag on my Boost dependency does resolve the version conflict. Additionally, re-confirming that adding the with_thrift flag (as @memsharded pointed out before) resolves the Could NOT find ThriftAlt error.

And also thank you for putting together that conan-center-index PR on my behalf! I guess an approval from some random Conan user isn't helpful but ... ship it!

[danimtb]

Thank you for the feedback @kevin-mcvey, glad it did the trick! 😄