LAPACKException on noisy optimization problem

[jondeuce]

Thanks for the great package! I've noticed a small bug when running a toy optimization problem:

for s in [1e-3, 1e-2, 1e-1, 1e0], _ in 1:3
    x0 = 100 .* (10 .+ randn(10))
    s0 = norm(x0)
    f = x -> norm(x) + s*randn()
    res = CMAEvolutionStrategy.minimize(f, x0, s0;
        noisy = true,
        maxfevals = 100_000,
        verbosity = 0,
    )
    rel_rmse = sqrt(mean(abs2, xbest(res))) / s
    @show s, rel_rmse
end

When maxfevals is set sufficiently small, there is no issue, and the returned solution is close to the origin relative to the noise level, which is fabulous:

(s, rel_rmse) = (0.001, 0.8293057373809105)
(s, rel_rmse) = (0.001, 0.5780459278485734)
(s, rel_rmse) = (0.001, 1.1275220456390496)
(s, rel_rmse) = (0.01, 0.9784417504323123)
(s, rel_rmse) = (0.01, 0.9406688487406037)
(s, rel_rmse) = (0.01, 0.8285373646583917)
(s, rel_rmse) = (0.1, 0.6006660993817257)
(s, rel_rmse) = (0.1, 0.6414766623592318)
(s, rel_rmse) = (0.1, 0.7099850771316121)
(s, rel_rmse) = (1.0, 0.8203895037273008)
(s, rel_rmse) = (1.0, 1.32515671822559)
(s, rel_rmse) = (1.0, 0.7649306247022853)

However, without setting maxfevals, I often get the following LAPACKException, which seems to be some sort of failed eigenvalue solve:

ERROR: LAPACKException(1)
Stacktrace:
 [1] chklapackerror at /home/julia-1.4.0-sourcebuild-mkl/usr/share/julia/stdlib/v1.4/LinearAlgebra/src/lapack.jl:38 [inlined]
 [2] syevr!(::Char, ::Char, ::Char, ::Array{Float64,2}, ::Float64, ::Float64, ::Int64, ::Int64, ::Float64) at /home/julia-1.4.0-sourcebuild-mkl/usr/share/julia/stdlib/v1.4/LinearAlgebra/src/lapack.jl:5085
 [3] #eigen!#103 at /home/julia-1.4.0-sourcebuild-mkl/usr/share/julia/stdlib/v1.4/LinearAlgebra/src/symmetric.jl:658 [inlined]
 [4] #eigen#104 at /home/julia-1.4.0-sourcebuild-mkl/usr/share/julia/stdlib/v1.4/LinearAlgebra/src/symmetric.jl:663 [inlined]
 [5] eigen at /home/julia-1.4.0-sourcebuild-mkl/usr/share/julia/stdlib/v1.4/LinearAlgebra/src/symmetric.jl:661 [inlined]
 [6] CMAEvolutionStrategy.MEigen(::Array{Float64,2}) at /home/.julia/packages/CMAEvolutionStrategy/vYGdv/src/optimizer.jl:99
 [7] update!(::CMAEvolutionStrategy.Covariance{CMAEvolutionStrategy.MEigen}, ::Array{Float64,1}, ::Array{Float64,2}, ::Array{Int64,1}, ::CMAEvolutionStrategy.RecombinationWeights, ::Bool) at /home/.julia/packages/CMAEvolutionStrategy/vYGdv/src/optimizer.jl:136
 [8] update!(::CMAEvolutionStrategy.Parameters{Nothing,CMAEvolutionStrategy.NoiseHandling{CMAEvolutionStrategy.var"#31#36"}}, ::Array{Float64,2}, ::Array{Int64,1}) at /home/.julia/packages/CMAEvolutionStrategy/vYGdv/src/optimizer.jl:183
 [9] run!(::CMAEvolutionStrategy.Optimizer{CMAEvolutionStrategy.Parameters{Nothing,CMAEvolutionStrategy.NoiseHandling{CMAEvolutionStrategy.var"#31#36"}},CMAEvolutionStrategy.BasicLogger{CMAEvolutionStrategy.var"#14#16"},CMAEvolutionStrategy.Stop{Nothing,Nothing,Nothing,Int64,Nothing,Nothing,Float64}}, ::Function) at /home/.julia/packages/CMAEvolutionStrategy/vYGdv/src/CMAEvolutionStrategy.jl:39
 [10] minimize(::Function, ::Array{Float64,1}, ::Float64; kwargs::Base.Iterators.Pairs{Symbol,Integer,Tuple{Symbol,Symbol},NamedTuple{(:noisy, :verbosity),Tuple{Bool,Int64}}}) at /home/.julia/packages/CMAEvolutionStrategy/vYGdv/src/CMAEvolutionStrategy.jl:79

Thanks again for the great work! I've been looking for a robust stochastic optimization package, and have so far avoided writing one myself ;)

[jbrea]

Thanks for reporting. Stagnation was not properly detected.

[jbrea]

ps. In your example noisy = false leads actually to better results (also with pycma). I guess the default noise handler is not optimal...

[jondeuce]

Huh, interesting. I was under the impression that noisy = true was required for non-deterministic objective functions (though I am not very familiar with the underlying algorithm/related literature)

[jbrea]

pps. with noise_handling = CMAEvolutionStrategy.NoiseHandling(ασ = 1.01) I get the best results for your toy example; but the improvement is rather small. The default ασ = 1.1 for 10 dimensions is suboptimal for this problem. I'll update the README to warn the user :)