sentencepiece when running cargo install

[jwijffels]

Hello,

I'm having a look if I can use this library in order to later on build an R wrapper around it, as this setup seems to be the only software providing some functionalities similar to UDPipe 2.

I'm new to rust however and although I've built some R wrappers around c++ libraries (namely UDPipe (in casu this on https://github.com/bnosac/udpipe) and sentencepiece (in casu this one: https://github.com/bnosac/sentencepiece), I don't know how to fix this sentencepiece build error.
Could you indicate what goes wrong and how to solve this here?

$ cargo install --no-default-features --path syntaxdot-cli
  Installing syntaxdot-cli v0.4.0 (C:\Users\Jan\Dropbox\Work\RForgeBNOSAC\OpenSource\syntaxdot\syntaxdot-cli)
    Updating crates.io index
   Compiling sentencepiece-sys v0.7.1
   Compiling torch-sys v0.5.0
error: failed to run custom build command for `sentencepiece-sys v0.7.1`

Caused by:
  process didn't exit successfully: `C:\Users\Jan\Dropbox\Work\RForgeBNOSAC\OpenSource\syntaxdot\target\release\build\sentencepiece-sys-0ade1f02ad77b1a5\build-script-build` (exit code: 101)
  --- stdout
  cargo:rerun-if-env-changed=SENTENCEPIECE_NO_PKG_CONFIG
  cargo:rerun-if-env-changed=PKG_CONFIG
  cargo:rerun-if-env-changed=SENTENCEPIECE_STATIC
  cargo:rerun-if-env-changed=SENTENCEPIECE_DYNAMIC
  cargo:rerun-if-env-changed=PKG_CONFIG_ALL_STATIC
  cargo:rerun-if-env-changed=PKG_CONFIG_ALL_DYNAMIC
  cargo:rerun-if-env-changed=PKG_CONFIG_PATH_x86_64-pc-windows-msvc
  cargo:rerun-if-env-changed=PKG_CONFIG_PATH_x86_64_pc_windows_msvc
  cargo:rerun-if-env-changed=HOST_PKG_CONFIG_PATH
  cargo:rerun-if-env-changed=PKG_CONFIG_PATH
  cargo:rerun-if-env-changed=PKG_CONFIG_LIBDIR_x86_64-pc-windows-msvc
  cargo:rerun-if-env-changed=PKG_CONFIG_LIBDIR_x86_64_pc_windows_msvc
  cargo:rerun-if-env-changed=HOST_PKG_CONFIG_LIBDIR
  cargo:rerun-if-env-changed=PKG_CONFIG_LIBDIR
  cargo:rerun-if-env-changed=PKG_CONFIG_SYSROOT_DIR_x86_64-pc-windows-msvc
  cargo:rerun-if-env-changed=PKG_CONFIG_SYSROOT_DIR_x86_64_pc_windows_msvc
  cargo:rerun-if-env-changed=HOST_PKG_CONFIG_SYSROOT_DIR
  cargo:rerun-if-env-changed=PKG_CONFIG_SYSROOT_DIR
  running: "cmake" "C:\\Users\\Jan\\.cargo\\registry\\src\\github.com-1ecc6299db9ec823\\sentencepiece-sys-0.7.1\\source" "-G" "Visual Studio 16 2019" "-Thost=x64" "-Ax64" "-DCMAKE_INSTALL_PREFIX=C:\\Users\\Jan\\Dropbox\\Work\\RForgeBNOSAC\\OpenSource\\syntaxdot\\target\\release\\build\\sentencepiece-sys-2d2a243b21575a42\\out" "-DCMAKE_C_FLAGS= -nologo -MD -Brepro" "-DCMAKE_C_FLAGS_RELEASE= -nologo -MD -Brepro" "-DCMAKE_CXX_FLAGS= -nologo -MD -Brepro" "-DCMAKE_CXX_FLAGS_RELEASE= -nologo -MD -Brepro" "-DCMAKE_ASM_FLAGS= -nologo -MD -Brepro" "-DCMAKE_ASM_FLAGS_RELEASE= -nologo -MD -Brepro" "-DCMAKE_BUILD_TYPE=Release"
  -- VERSION: 0.1.96
  -- Selecting Windows SDK version 10.0.18362.0 to target Windows 6.3.9600.
  -- Not Found TCMalloc: TCMALLOC_LIB-NOTFOUND
  -- Configuring done
  -- Generating done
  -- Build files have been written to: C:/Users/Jan/Dropbox/Work/RForgeBNOSAC/OpenSource/syntaxdot/target/release/build/sentencepiece-sys-2d2a243b21575a42/out/build
  running: "cmake" "--build" "." "--target" "install" "--config" "Release" "--"
  Microsoft (R) Build Engine version 16.6.0+5ff7b0c9e for .NET Framework
  Copyright (C) Microsoft Corporation. All rights reserved.

    Auto build dll exports
    sentencepiece.vcxproj -> C:\Users\Jan\Dropbox\Work\RForgeBNOSAC\OpenSource\syntaxdot\target\release\build\sentencepiece-sys-2d2a243b21575a42\out\build\src\Release\sentencepiece.dll
    sentencepiece-static.vcxproj -> C:\Users\Jan\Dropbox\Work\RForgeBNOSAC\OpenSource\syntaxdot\target\release\build\sentencepiece-sys-2d2a243b21575a42\out\build\src\Release\sentencepiece.lib
    Auto build dll exports
       Creating library C:/Users/Jan/Dropbox/Work/RForgeBNOSAC/OpenSource/syntaxdot/target/release/build/sentencepiece-sys-2d2a243b21575a42/out/build/src/Release/sentencepiece_train_import.lib and object C:/Users/Jan/Dropbox/Work/RForgeBNOSAC/OpenSource/syntaxdot/target/release/build/sentencepiece-sys-2d2a243b21575a42/out/build/src/Release/sentencepiece_train_import.exp
  trainer_interface.obj : error LNK2019: unresolved external symbol "private: static class google::protobuf::internal::LazyString const sentencepiece::TrainerSpec::_i_give_permission_to_break_this_code_default_unk_piece_" (?_i_give_permission_to_break_this_code_default_unk_piece_@TrainerSpec@sentencepiece@@0VLazyString@internal@protobuf@google@@B) referenced in function "public: bool __cdecl <lambda_00046828aa1a5cfb8c470ee6e720106a>::operator()(class std::basic_string<char,struct std::char_traits<char>,class std::allocator<char> > const &,enum sentencepiece::ModelProto_SentencePiece_Type)const " (??R<lambda_00046828aa1a5cfb8c470ee6e720106a>@@QEBA_NAEBV?$basic_string@DU?$char_traits@D@std@@V?$allocator@D@2@@std@@W4ModelProto_SentencePiece_Type@sentencepiece@@@Z) [C:\Users\Jan\Dropbox\Work\RForgeBNOSAC\OpenSource\syntaxdot\target\release\build\sentencepiece-sys-2d2a243b21575a42\out\build\src\sentencepiece_train.vcxproj]
  sentencepiece_trainer.obj : error LNK2001: unresolved external symbol "private: static class google::protobuf::internal::LazyString const sentencepiece::TrainerSpec::_i_give_permission_to_break_this_code_default_unk_piece_" (?_i_give_permission_to_break_this_code_default_unk_piece_@TrainerSpec@sentencepiece@@0VLazyString@internal@protobuf@google@@B) [C:\Users\Jan\Dropbox\Work\RForgeBNOSAC\OpenSource\syntaxdot\target\release\build\sentencepiece-sys-2d2a243b21575a42\out\build\src\sentencepiece_train.vcxproj]
  trainer_interface.obj : error LNK2019: unresolved external symbol "private: static class google::protobuf::internal::LazyString const sentencepiece::TrainerSpec::_i_give_permission_to_break_this_code_default_bos_piece_" (?_i_give_permission_to_break_this_code_default_bos_piece_@TrainerSpec@sentencepiece@@0VLazyString@internal@protobuf@google@@B) referenced in function "public: bool __cdecl <lambda_00046828aa1a5cfb8c470ee6e720106a>::operator()(class std::basic_string<char,struct std::char_traits<char>,class std::allocator<char> > const &,enum sentencepiece::ModelProto_SentencePiece_Type)const " (??R<lambda_00046828aa1a5cfb8c470ee6e720106a>@@QEBA_NAEBV?$basic_string@DU?$char_traits@D@std@@V?$allocator@D@2@@std@@W4ModelProto_SentencePiece_Type@sentencepiece@@@Z) [C:\Users\Jan\Dropbox\Work\RForgeBNOSAC\OpenSource\syntaxdot\target\release\build\sentencepiece-sys-2d2a243b21575a42\out\build\src\sentencepiece_train.vcxproj]
  sentencepiece_trainer.obj : error LNK2001: unresolved external symbol "private: static class google::protobuf::internal::LazyString const sentencepiece::TrainerSpec::_i_give_permission_to_break_this_code_default_bos_piece_" (?_i_give_permission_to_break_this_code_default_bos_piece_@TrainerSpec@sentencepiece@@0VLazyString@internal@protobuf@google@@B) [C:\Users\Jan\Dropbox\Work\RForgeBNOSAC\OpenSource\syntaxdot\target\release\build\sentencepiece-sys-2d2a243b21575a42\out\build\src\sentencepiece_train.vcxproj]
  trainer_interface.obj : error LNK2019: unresolved external symbol "private: static class google::protobuf::internal::LazyString const sentencepiece::TrainerSpec::_i_give_permission_to_break_this_code_default_eos_piece_" (?_i_give_permission_to_break_this_code_default_eos_piece_@TrainerSpec@sentencepiece@@0VLazyString@internal@protobuf@google@@B) referenced in function "public: bool __cdecl <lambda_00046828aa1a5cfb8c470ee6e720106a>::operator()(class std::basic_string<char,struct std::char_traits<char>,class std::allocator<char> > const &,enum sentencepiece::ModelProto_SentencePiece_Type)const " (??R<lambda_00046828aa1a5cfb8c470ee6e720106a>@@QEBA_NAEBV?$basic_string@DU?$char_traits@D@std@@V?$allocator@D@2@@std@@W4ModelProto_SentencePiece_Type@sentencepiece@@@Z) [C:\Users\Jan\Dropbox\Work\RForgeBNOSAC\OpenSource\syntaxdot\target\release\build\sentencepiece-sys-2d2a243b21575a42\out\build\src\sentencepiece_train.vcxproj]
  sentencepiece_trainer.obj : error LNK2001: unresolved external symbol "private: static class google::protobuf::internal::LazyString const sentencepiece::TrainerSpec::_i_give_permission_to_break_this_code_default_eos_piece_" (?_i_give_permission_to_break_this_code_default_eos_piece_@TrainerSpec@sentencepiece@@0VLazyString@internal@protobuf@google@@B) [C:\Users\Jan\Dropbox\Work\RForgeBNOSAC\OpenSource\syntaxdot\target\release\build\sentencepiece-sys-2d2a243b21575a42\out\build\src\sentencepiece_train.vcxproj]
  trainer_interface.obj : error LNK2019: unresolved external symbol "private: static class google::protobuf::internal::LazyString const sentencepiece::TrainerSpec::_i_give_permission_to_break_this_code_default_pad_piece_" (?_i_give_permission_to_break_this_code_default_pad_piece_@TrainerSpec@sentencepiece@@0VLazyString@internal@protobuf@google@@B) referenced in function "public: bool __cdecl <lambda_00046828aa1a5cfb8c470ee6e720106a>::operator()(class std::basic_string<char,struct std::char_traits<char>,class std::allocator<char> > const &,enum sentencepiece::ModelProto_SentencePiece_Type)const " (??R<lambda_00046828aa1a5cfb8c470ee6e720106a>@@QEBA_NAEBV?$basic_string@DU?$char_traits@D@std@@V?$allocator@D@2@@std@@W4ModelProto_SentencePiece_Type@sentencepiece@@@Z) [C:\Users\Jan\Dropbox\Work\RForgeBNOSAC\OpenSource\syntaxdot\target\release\build\sentencepiece-sys-2d2a243b21575a42\out\build\src\sentencepiece_train.vcxproj]
  sentencepiece_trainer.obj : error LNK2001: unresolved external symbol "private: static class google::protobuf::internal::LazyString const sentencepiece::TrainerSpec::_i_give_permission_to_break_this_code_default_pad_piece_" (?_i_give_permission_to_break_this_code_default_pad_piece_@TrainerSpec@sentencepiece@@0VLazyString@internal@protobuf@google@@B) [C:\Users\Jan\Dropbox\Work\RForgeBNOSAC\OpenSource\syntaxdot\target\release\build\sentencepiece-sys-2d2a243b21575a42\out\build\src\sentencepiece_train.vcxproj]
  trainer_interface.obj : error LNK2019: unresolved external symbol "const sentencepiece::ModelProto::`vftable'" (??_7ModelProto@sentencepiece@@6B@) referenced in function "private: class sentencepiece::util::Status __cdecl sentencepiece::TrainerInterface::SaveModel(class absl::string_view)const " (?SaveModel@TrainerInterface@sentencepiece@@AEBA?AVStatus@util@2@Vstring_view@absl@@@Z) [C:\Users\Jan\Dropbox\Work\RForgeBNOSAC\OpenSource\syntaxdot\target\release\build\sentencepiece-sys-2d2a243b21575a42\out\build\src\sentencepiece_train.vcxproj]
  unigram_model_trainer.obj : error LNK2001: unresolved external symbol "const sentencepiece::ModelProto::`vftable'" (??_7ModelProto@sentencepiece@@6B@) [C:\Users\Jan\Dropbox\Work\RForgeBNOSAC\OpenSource\syntaxdot\target\release\build\sentencepiece-sys-2d2a243b21575a42\out\build\src\sentencepiece_train.vcxproj]
  sentencepiece_trainer.obj : error LNK2001: unresolved external symbol "const sentencepiece::ModelProto::`vftable'" (??_7ModelProto@sentencepiece@@6B@) [C:\Users\Jan\Dropbox\Work\RForgeBNOSAC\OpenSource\syntaxdot\target\release\build\sentencepiece-sys-2d2a243b21575a42\out\build\src\sentencepiece_train.vcxproj]
  unigram_model_trainer.obj : error LNK2019: unresolved external symbol "class sentencepiece::TrainerSpecDefaultTypeInternal sentencepiece::_TrainerSpec_default_instance_" (?_TrainerSpec_default_instance_@sentencepiece@@3VTrainerSpecDefaultTypeInternal@1@A) referenced in function "public: virtual bool __cdecl sentencepiece::ModelInterface::ByteFallbackEnabled(void)const " (?ByteFallbackEnabled@ModelInterface@sentencepiece@@UEBA_NXZ) [C:\Users\Jan\Dropbox\Work\RForgeBNOSAC\OpenSource\syntaxdot\target\release\build\sentencepiece-sys-2d2a243b21575a42\out\build\src\sentencepiece_train.vcxproj]
  sentencepiece_trainer.obj : error LNK2019: unresolved external symbol "private: static class google::protobuf::internal::LazyString const sentencepiece::TrainerSpec::_i_give_permission_to_break_this_code_default_unk_surface_" (?_i_give_permission_to_break_this_code_default_unk_surface_@TrainerSpec@sentencepiece@@0VLazyString@internal@protobuf@google@@B) referenced in function "class std::basic_string<char,struct std::char_traits<char>,class std::allocator<char> > __cdecl sentencepiece::PrintProto(class sentencepiece::TrainerSpec const &,class absl::string_view)" (?PrintProto@sentencepiece@@YA?AV?$basic_string@DU?$char_traits@D@std@@V?$allocator@D@2@@std@@AEBVTrainerSpec@1@Vstring_view@absl@@@Z) [C:\Users\Jan\Dropbox\Work\RForgeBNOSAC\OpenSource\syntaxdot\target\release\build\sentencepiece-sys-2d2a243b21575a42\out\build\src\sentencepiece_train.vcxproj]
  sentencepiece_trainer.obj : error LNK2019: unresolved external symbol "const sentencepiece::TrainerSpec::`vftable'" (??_7TrainerSpec@sentencepiece@@6B@) referenced in function "public: static class sentencepiece::util::Status __cdecl sentencepiece::SentencePieceTrainer::Train(class std::unordered_map<class std::basic_string<char,struct std::char_traits<char>,class std::allocator<char> >,class std::basic_string<char,struct std::char_traits<char>,class std::allocator<char> >,struct std::hash<class std::basic_string<char,struct std::char_traits<char>,class std::allocator<char> > >,struct std::equal_to<class std::basic_string<char,struct std::char_traits<char>,class std::allocator<char> > >,class
std::allocator<struct std::pair<class std::basic_string<char,struct std::char_traits<char>,class std::allocator<char> > const ,class std::basic_string<char,struct std::char_traits<char>,class std::allocator<char> > > > > const &,class sentencepiece::SentenceIterator *,class std::basic_string<char,struct std::char_traits<char>,class std::allocator<char> > *)" (?Train@SentencePieceTrainer@sentencepiece@@SA?AVStatus@util@2@AEBV?$unordered_map@V?$basic_string@DU?$char_traits@D@std@@V?$allocator@D@2@@std@@V12@U?$hash@V?$basic_string@DU?$char_traits@D@std@@V?$allocator@D@2@@std@@@2@U?$equal_to@V?$basic_string@DU?$char_traits@D@std@@V?$allocator@D@2@@std@@@2@V?$allocator@U?$pair@$$CBV?$basic_string@DU?$char_traits@D@std@@V?$allocator@D@2@@std@@V12@@std@@@2@@std@@PEAVSentenceIterator@2@PEAV?$basic_string@DU?$char_traits@D@std@@V?$allocator@D@2@@6@@Z) [C:\Users\Jan\Dropbox\Work\RForgeBNOSAC\OpenSource\syntaxdot\target\release\build\sentencepiece-sys-2d2a243b21575a42\out\build\src\sentencepiece_train.vcxproj]
  sentencepiece_trainer.obj : error LNK2019: unresolved external symbol "const sentencepiece::NormalizerSpec::`vftable'" (??_7NormalizerSpec@sentencepiece@@6B@) referenced in function "public: static class sentencepiece::NormalizerSpec __cdecl sentencepiece::SentencePieceTrainer::GetNormalizerSpec(class absl::string_view)" (?GetNormalizerSpec@SentencePieceTrainer@sentencepiece@@SA?AVNormalizerSpec@2@Vstring_view@absl@@@Z) [C:\Users\Jan\Dropbox\Work\RForgeBNOSAC\OpenSource\syntaxdot\target\release\build\sentencepiece-sys-2d2a243b21575a42\out\build\src\sentencepiece_train.vcxproj]
  C:\Users\Jan\Dropbox\Work\RForgeBNOSAC\OpenSource\syntaxdot\target\release\build\sentencepiece-sys-2d2a243b21575a42\out\build\src\Release\sentencepiece_train.dll : fatal error LNK1120: 9 unresolved externals [C:\Users\Jan\Dropbox\Work\RForgeBNOSAC\OpenSource\syntaxdot\target\release\build\sentencepiece-sys-2d2a243b21575a42\out\build\src\sentencepiece_train.vcxproj]
    sentencepiece_train-static.vcxproj -> C:\Users\Jan\Dropbox\Work\RForgeBNOSAC\OpenSource\syntaxdot\target\release\build\sentencepiece-sys-2d2a243b21575a42\out\build\src\Release\sentencepiece_train.lib
  spm_decode_main.obj : error LNK2019: unresolved external symbol "class absl::Flag<int> FLAGS_minloglevel" (?FLAGS_minloglevel@@3V?$Flag@H@absl@@A) referenced in function main [C:\Users\Jan\Dropbox\Work\RForgeBNOSAC\OpenSource\syntaxdot\target\release\build\sentencepiece-sys-2d2a243b21575a42\out\build\src\spm_decode.vcxproj]
  spm_decode_main.obj : error LNK2019: unresolved external symbol "const sentencepiece::SentencePieceText::`vftable'" (??_7SentencePieceText@sentencepiece@@6B@) referenced in function main [C:\Users\Jan\Dropbox\Work\RForgeBNOSAC\OpenSource\syntaxdot\target\release\build\sentencepiece-sys-2d2a243b21575a42\out\build\src\spm_decode.vcxproj]
  C:\Users\Jan\Dropbox\Work\RForgeBNOSAC\OpenSource\syntaxdot\target\release\build\sentencepiece-sys-2d2a243b21575a42\out\build\src\Release\spm_decode.exe : fatal error LNK1120: 2 unresolved externals [C:\Users\Jan\Dropbox\Work\RForgeBNOSAC\OpenSource\syntaxdot\target\release\build\sentencepiece-sys-2d2a243b21575a42\out\build\src\spm_decode.vcxproj]
  spm_encode_main.obj : error LNK2019: unresolved external symbol "class absl::Flag<int> FLAGS_minloglevel" (?FLAGS_minloglevel@@3V?$Flag@H@absl@@A) referenced in function main [C:\Users\Jan\Dropbox\Work\RForgeBNOSAC\OpenSource\syntaxdot\target\release\build\sentencepiece-sys-2d2a243b21575a42\out\build\src\spm_encode.vcxproj]
  spm_encode_main.obj : error LNK2019: unresolved external symbol "const sentencepiece::SentencePieceText::`vftable'" (??_7SentencePieceText@sentencepiece@@6B@) referenced in function main [C:\Users\Jan\Dropbox\Work\RForgeBNOSAC\OpenSource\syntaxdot\target\release\build\sentencepiece-sys-2d2a243b21575a42\out\build\src\spm_encode.vcxproj]
  spm_encode_main.obj : error LNK2019: unresolved external symbol "const sentencepiece::NBestSentencePieceText::`vftable'" (??_7NBestSentencePieceText@sentencepiece@@6B@) referenced in function main [C:\Users\Jan\Dropbox\Work\RForgeBNOSAC\OpenSource\syntaxdot\target\release\build\sentencepiece-sys-2d2a243b21575a42\out\build\src\spm_encode.vcxproj]
  C:\Users\Jan\Dropbox\Work\RForgeBNOSAC\OpenSource\syntaxdot\target\release\build\sentencepiece-sys-2d2a243b21575a42\out\build\src\Release\spm_encode.exe : fatal error LNK1120: 3 unresolved externals [C:\Users\Jan\Dropbox\Work\RForgeBNOSAC\OpenSource\syntaxdot\target\release\build\sentencepiece-sys-2d2a243b21575a42\out\build\src\spm_encode.vcxproj]
  spm_export_vocab_main.obj : error LNK2019: unresolved external symbol "class absl::Flag<int> FLAGS_minloglevel" (?FLAGS_minloglevel@@3V?$Flag@H@absl@@A) referenced in function main [C:\Users\Jan\Dropbox\Work\RForgeBNOSAC\OpenSource\syntaxdot\target\release\build\sentencepiece-sys-2d2a243b21575a42\out\build\src\spm_export_vocab.vcxproj]
  C:\Users\Jan\Dropbox\Work\RForgeBNOSAC\OpenSource\syntaxdot\target\release\build\sentencepiece-sys-2d2a243b21575a42\out\build\src\Release\spm_export_vocab.exe : fatal error LNK1120: 1 unresolved externals [C:\Users\Jan\Dropbox\Work\RForgeBNOSAC\OpenSource\syntaxdot\target\release\build\sentencepiece-sys-2d2a243b21575a42\out\build\src\spm_export_vocab.vcxproj]

  --- stderr
  thread 'main' panicked at '
  command did not execute successfully, got: exit code: 1

  build script failed, must exit now', C:\Users\Jan\.cargo\registry\src\github.com-1ecc6299db9ec823\cmake-0.1.45\src\lib.rs:894:5
  note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
warning: build failed, waiting for other jobs to finish...
error: failed to compile `syntaxdot-cli v0.4.0 (C:\Users\Jan\Dropbox\Work\RForgeBNOSAC\OpenSource\syntaxdot\syntaxdot-cli)`, intermediate artifacts can be found at `C:\Users\Jan\Dropbox\Work\RForgeBNOSAC\OpenSource\syntaxdot\target`

Caused by:
  build failed

In my R wrapper around sentencepiece protobuf is included, I don't know the setup here.

[danieldk]

This seems like a Visual C++-specific issue. However, I am not familiar enough with Visual C++ (or Windows) to have any constructive ideas here.

The only thing that I could find is that upstream sentencepiece seems to build with mingw, not Visual C++?