Error when running the example in Julia from within WSL

[DanielBrosch]

When I run the basic example on Linux (Debian) from Julia running within WSL, I get the following error:

ERROR: LoadError: SystemError: opening file "/tmp/jl_Q4J93Q/output.dat": No such file or directory Stacktrace: [1] systemerror(p::String, errno::Int32; extrainfo::Nothing) @ Base ./error.jl:168 [2] #systemerror#62 @ ./error.jl:167 [inlined] [3] systemerror @ ./error.jl:167 [inlined] [4] open(fname::String; lock::Bool, read::Bool, write::Nothing, create::Nothing, truncate::Nothing, append::Nothing) @ Base ./iostream.jl:293 [5] open(fname::String, mode::String; lock::Bool) @ Base ./iostream.jl:355 [6] open(fname::String, mode::String) @ Base ./iostream.jl:355 [7] open(::SDPAFamily.var"#23#29"{BigFloat, SDPAFamily.Optimizer{BigFloat}, Vector{BigFloat}, Vector{BigFloat}, SDPAFamily.var"#remove_brackets!#28", SDPAFamily.var"#getnextline#26"{String}}, ::String, ::Vararg{String, N} where N; kwargs::Base.Iterators.Pairs{Union{}, Union{}, Tuple{}, NamedTuple{(), Tuple{}}}) @ Base ./io.jl:328 [8] open @ ./io.jl:328 [inlined] [9] read_results!(optimizer::SDPAFamily.Optimizer{BigFloat}, filepath::String, redundant_entries::Vector{Int64}) @ SDPAFamily ~/.julia/packages/SDPAFamily/J3st0/src/file_io.jl:39 [10] sdpa_gmp_binary_solve!(m::SDPAFamily.Optimizer{BigFloat}, full_input_path::String, full_output_path::String; redundant_entries::Vector{Int64}) @ SDPAFamily ~/.julia/packages/SDPAFamily/J3st0/src/binary_call.jl:58 [11] optimize!(m::SDPAFamily.Optimizer{BigFloat}) @ SDPAFamily ~/.julia/packages/SDPAFamily/J3st0/src/MOI_wrapper.jl:345 [12] optimize!(m::MathOptInterface.Utilities.CachingOptimizer{SDPAFamily.Optimizer{BigFloat}, MathOptInterface.Utilities.UniversalFallback{MathOptInterface.Utilities.GenericModel{BigFloat, MathOptInterface.Utilities.ModelFunctionConstraints{BigFloat}}}}) @ MathOptInterface.Utilities ~/.julia/packages/MathOptInterface/1EYfq/src/Utilities/cachingoptimizer.jl:252 [13] optimize! @ ~/.julia/packages/MathOptInterface/1EYfq/src/Bridges/bridge_optimizer.jl:293 [inlined] [14] solve!(problem::Problem{BigFloat}, optimizer::SDPAFamily.Optimizer{BigFloat}; check_vexity::Bool, verbose::Bool, warmstart::Bool, silent_solver::Bool) @ Convex ~/.julia/packages/Convex/SRJjA/src/solution.jl:243 [15] solve!(problem::Problem{BigFloat}, optimizer::SDPAFamily.Optimizer{BigFloat}) @ Convex ~/.julia/packages/Convex/SRJjA/src/solution.jl:221 [16] solve!(problem::Problem{BigFloat}, optimizer_factory::Function; kwargs::Base.Iterators.Pairs{Union{}, Union{}, Tuple{}, NamedTuple{(), Tuple{}}}) @ Convex ~/.julia/packages/Convex/SRJjA/src/solution.jl:192 [17] solve!(problem::Problem{BigFloat}, optimizer_factory::Function) @ Convex ~/.julia/packages/Convex/SRJjA/src/solution.jl:191 [18] top-level scope

Note that I am not running Julia on Windows. I used to do that in the past, and used this library with no issues with SDPA running in WSL.

Now I am running Julia itself on WSL, too, which I expected to work as well, since it shouldn't be much different to working entirely on Linux. (The reason for doing this is some significant performance increases in some other libraries I am using.)

I checked out that temp folder, it contains the files "input.dat-s" and "params.sdpa" which seem correct, as well as a completely empty file "errors.log".

[ericphanson]

Interesting, can you try on master?

] add SDPAFamily#master

I've merged #48 but haven't tagged a release since I wasn't totally confident if it was better or not, and I wonder if it could help here.

[DanielBrosch]

Now the error is

LoadError: SystemError: opening file "/home/dbrosch/.julia/scratchspaces/bfe18334-aefd-11e9-1109-4bf2b15a5b91/solves/jl_l8Khh4/output.dat": No such file or directory

otherwise everything seems to be the same.

[DanielBrosch]

Hm, if I try to run
~/.julia/scratchspaces/bfe18334-aefd-11e9-1109-4bf2b15a5b91/build_julia_1.6_Linux/usr/bin$ ./sdpa_gmp
it returns "Segmentation fault".

[ericphanson]

Huh, that sounds like the binary is broken. Can you run these two lines?

Sys.islinux()
occursin("WSL", read(`cat /proc/version`, String))

We use those to check if we should install different binaries for WSL here:

SDPAFamily.jl/deps/build.jl

Line 66 in fcacf77

if Sys.islinux() && occursin("WSL", read(`cat /proc/version`, String))

I think exactly to avoid the issue that we had where the usual linux binaries would not work under WSL for some reason.

However that was before WSL2 came out, so things might be different now. In particular, maybe you are getting those custom binaries but actually the standard linux ones would work instead now, or perhaps you aren't getting the custom ones but should be...

[DanielBrosch]

Sys.islinux()

returns true,

occursin("WSL", read(`cat /proc/version`, String))

returns false (despite Julia running in WSL2).

The full string is
Linux version 4.19.104-microsoft-standard (oe-user@oe-host) (gcc version 8.2.0 (GCC)) #1 SMP Wed Feb 19 06:37:35 UTC 2020.

[ericphanson]

Ok interesting! Can you try with

] add SDPAFamily#eph/wsl

(restart julia session)

] build SDPAFamily

?

If I haven't messed up that PR, then that should detect WSL and install the WSL binaries and maybe it will work then...

[DanielBrosch]

It returns the same error (No such file or directory...). But it does create a different executable now in a different folder. The executable was missing execute permission, and after adding it I can at least call it directly successfully. But calling the example still results in the same error.

If I check

collect(SDPAFamily.BB_PATHS)

it still links to the wrong non-WSL executables.

[DanielBrosch]

If I run

SDPAFamily.BB_PATHS[:sdpa_gmp] = "/home/dbrosch/.julia/scratchspaces/bfe18334-aefd-11e9-1109-4bf2b15a5b91/build_WSL_true_julia_1.6_Linux/usr/bin/sdpa_gmp"

it works now! So it just needs the correct path (and execute permissions?).

The old path was

SDPAFamily.BB_PATHS[:sdpa_gmp] = "/home/dbrosch/.julia/scratchspaces/bfe18334-aefd-11e9-1109-4bf2b15a5b91/build_julia_1.6_Linux/usr/bin/sdpa_gmp"

[ericphanson]

Ah thanks for trying it, there was a mistake in the PR. I've updated it, would you mind trying again? If you are on the eph/wsl branch, ] update SDPAFamily should pull the latest commit, and then you probably need to rebuild.

[ericphanson]

Ok, that's promising! I think the newest version of the PR should fix it then

edit: except execute permissions, probably need to add that

[DanielBrosch]

Hm, if I build SDPAFamily now I get a hash missmatch.

[ericphanson]

huh, can you copy the full error?

[DanielBrosch]

ERROR: Error building `SDPAFamily`: 
ERROR: LoadError: Hash Mismatch!
  Expected sha256:   f8ed0c3f2aefa1ab5a90f1999c78548625f6122f969972b8a51b54a0017b3a59
  Calculated sha256: f92250b0c7b0dda39d6c9655090be366dea6434960ba2d605ca91a86cdaf3d7e
Stacktrace:
  [1] error(s::String)
    @ Base ./error.jl:33
  [2] (::BinaryProvider.var"#146#148"{Bool, String, String, String})(file::IOStream)
    @ BinaryProvider ~/.julia/packages/BinaryProvider/U2dKK/src/Prefix.jl:595
  [3] open(f::BinaryProvider.var"#146#148"{Bool, String, String, String}, args::String; kwargs::Base.Iterators.Pairs{Union{}, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
    @ Base ./io.jl:330
  [4] open
    @ ./io.jl:328 [inlined]
  [5] verify(path::String, hash::String; verbose::Bool, report_cache_status::Bool, hash_path::String)
    @ BinaryProvider ~/.julia/packages/BinaryProvider/U2dKK/src/Prefix.jl:581
  [6] download_verify(url::String, hash::String, dest::String; verbose::Bool, force::Bool, quiet_download::Bool)
    @ BinaryProvider ~/.julia/packages/BinaryProvider/U2dKK/src/PlatformEngines.jl:664
  [7] install_custom_WSL_binaries(prefix::Prefix, verbose::Bool, variants::Vector{Symbol})
    @ Main ~/.julia/packages/SDPAFamily/OGQJw/deps/install_custom_WSL_binaries.jl:5
  [8] top-level scope
    @ ~/.julia/packages/SDPAFamily/OGQJw/deps/build.jl:90
  [9] include(fname::String)
    @ Base.MainInclude ./client.jl:444
 [10] top-level scope
    @ none:5
in expression starting at /home/dbrosch/.julia/packages/SDPAFamily/OGQJw/deps/build.jl:81

[ericphanson]

Huh, that's very strange, especially since it seems like you were able to download it earlier in #50 (comment)... I'm going to have to come back to this later, sorry

[DanielBrosch]

Indeed, it's pretty wierd...

No worries, I am done for today myself.

Thanks for the help!

[ericphanson]

I thought about a bit and I think the issue might just be that the old binaries are in the folder already and it realizes they are wrong so it throws the error. When you get a chance, can you delete them with

rm("/home/dbrosch/.julia/scratchspaces/bfe18334-aefd-11e9-1109-4bf2b15a5b91/build_julia_1.6_Linux", force=true, recursive=true)

and then try ] build SDPAFamily again?

[DanielBrosch]

That worked! I still had to fix the execute permission, but the example runs now.

[ericphanson]

Ok, if you don't mind, would you try deleting the folder and rebuilding one last time? I pushed up some code to try to set the executable bits manually so it would be great to hear if that worked or not. (Though I think it's an upstream issue, JuliaPackaging/BinaryProvider.jl#203)

[DanielBrosch]

Everything seems to work correctly now from a fresh install.

Thanks for the help! This will save me a lot of time.

[ericphanson]

Awesome, glad to hear it! I will make a release with those fixes then.