BoundsError

[rafaqz]

I'm getting this error with some series lengths of large arrays, using InspectDR as a Plots.jl backend.
It doesn't occur with other backends, and seems to be occuring in InspectDR internals - my arrays are much larger than 1007 elements.

BoundsError: attempt to access 1007-element Array{InspectDR.Point2D,1} at index [1008]
_reduce(::InspectDR.IDataset{true}, ::InspectDR.PExtents1D, ::Int64) at /home/raf/.julia/v0.6/InspectDR/src/datasets.jl:141
_reduce(::InspectDR.Waveform{InspectDR.IDataset}, ::InspectDR.PExtents1D, ::InspectDR.PointDropMatrix, ::Int64) at /home/raf/.julia/v0.6/InspectDR/src/base.jl:602
collect_to!(::Array{InspectDR.Waveform{Array{InspectDR.Point2D,1}},1}, ::Base.Generator{Array{InspectDR.Waveform{InspectDR.IDataset},1},InspectDR.##59#60{InspectDR.PExtents1D,InspectDR.PointDropMatrix,Int64}}, ::Int64, ::Int64) at ./array.jl:474
_collect(::Array{InspectDR.Waveform{InspectDR.IDataset},1}, ::Base.Generator{Array{InspectDR.Waveform{InspectDR.IDataset},1},InspectDR.##59#60{InspectDR.PExtents1D,InspectDR.PointDropMatrix,Int64}}, ::Base.EltypeUnknown, ::Base.HasShape) at ./array.jl:455
map(::Function, ::Array{InspectDR.Waveform{InspectDR.IDataset},1}) at ./abstractarray.jl:1865
_reduce(::Array{InspectDR.Waveform{InspectDR.IDataset},1}, ::InspectDR.PExtents1D, ::InspectDR.PointDropMatrix, ::Int64) at /home/raf/.julia/v0.6/InspectDR/src/base.jl:608
preprocess_data(::InspectDR.Plot2D) at /home/raf/.julia/v0.6/InspectDR/src/base.jl:674
update_ddata(::InspectDR.Plot2D) at /home/raf/.julia/v0.6/InspectDR/src/base.jl:691
render(::Cairo.CairoContext, ::InspectDR.Plot2D, ::Graphics.BoundingBox) at /home/raf/.julia/v0.6/InspectDR/src/cairo_top.jl:94
(::InspectDR.##79#80{InspectDR.Multiplot,Float64,Float64,Float64,Int64,Float64,Float64})(::Cairo.CairoContext) at /home/raf/.julia/v0.6/InspectDR/src/cairo_io.jl:97
withsurf(::InspectDR.##79#80{InspectDR.Multiplot,Float64,Float64,Float64,Int64,Float64,Float64}, ::Base.AbstractIOBuffer{Array{UInt8,1}}, ::MIME{Symbol("image/svg+xml")}, ::Float64, ::Float64) at /home/raf/.julia/v0.6/InspectDR/src/cairo_io.jl:45
_show(::Base.AbstractIOBuffer{Array{UInt8,1}}, ::MIME{Symbol("image/svg+xml")}, ::InspectDR.Multiplot, ::Float64, ::Float64) at /home/raf/.julia/v0.6/InspectDR/src/cairo_io.jl:75
_inspectdr_show(::Base.AbstractIOBuffer{Array{UInt8,1}}, ::MIME{Symbol("image/svg+xml")}, ::InspectDR.Multiplot, ::Int64, ::Int64) at /home/raf/.julia/v0.6/Plots/src/backends/inspectdr.jl:500
show(::Base.AbstractIOBuffer{Array{UInt8,1}}, ::MIME{Symbol("image/svg+xml")}, ::Plots.Plot{Plots.InspectDRBackend}) at /home/raf/.julia/v0.6/Plots/src/output.jl:208
show(::Base.AbstractIOBuffer{Array{UInt8,1}}, ::MIME{Symbol("text/html")}, ::Plots.Plot{Plots.InspectDRBackend}) at /home/raf/.julia/v0.6/Plots/src/output.jl:188
show(::Base.AbstractIOBuffer{Array{UInt8,1}}, ::String, ::Plots.Plot{Plots.InspectDRBackend}) at ./multimedia.jl:39
#sprint#228(::Void, ::Function, ::Int64, ::Function, ::String, ::Vararg{Any,N} where N) at ./strings/io.jl:66
display_dict(::Plots.Plot{Plots.InspectDRBackend}) at /home/raf/.julia/v0.6/Plots/src/output.jl:273
display(::IJulia.InlineDisplay, ::Plots.Plot{Plots.InspectDRBackend}) at /home/raf/.julia/v0.6/IJulia/src/inline.jl:72
display(::Plots.Plot{Plots.InspectDRBackend}) at ./multimedia.jl:194

[ma-laforge]

That definitely looks like a bug... but hard to pinpoint without sample code.

Can you build a simple testcase that causes this issue?

You could generate sample data using something like:

x=collect(1:2000)
y=rand(length(x))

Are you using multiple y for the same x? Are you using 2-dimensional x vectors? 2-dimensional y vectors?

...?

[rafaqz]

Just a single 1 dimensional vector.

I can't duplicate the issue with random data, but its reliable with my model output - which has a lot of very small numbers around zero in it, but other than that I have no idea whats going on.

[ma-laforge]

Thanks for that tidbit. It might be a bug with the algorithm for "F1 acceleration".

You can disable F1 acceleration by adding the following to your ~/.juliarc file:

DEFAULTS_INSPECTDR = Dict(
    :droppoints => :never,
)

If you no longer get a bounds error, that is likely the cause.

...Sadly, disabling F1 acceleration reduces the plotting speed somewhat - especially with large datasets.

[rafaqz]

Yep that fixed it, and slowed it down a bit...

[ma-laforge]

Great. That's good to know.

The problem is in the _reduce() function in src/datasets.jl.

The F1 acceleration algorithm is known not to be rock solid (see comment):

#FIXME: improve algorithm
#Algorithm succeptible to cumulative errors (esp w large x-values & large xres_max):

I am not exactly certain how to make this more robust yet.

On hack you could try is to increase the size of the padding at the "end" of the undersampled vector:

sz = min(n_ds, xres_max)+1[INCREASE THIS NUMBER]+2*min_lookahead #Allow extra points in case

Try increasing the value of 1 in this line to something larger... but I still don't feel this to be a real solution to the problem - just a band-aid.

[ma-laforge]

@rafaqz: BTW, I know disabling F1 acceleration slows things down significantly for large datasets (easily 2-10x for GBs of data)... but is it still faster than GR/plotly? Just curious.

[ma-laforge]

Update:

Just added a failsafe that will fall back to plotting with naive solution if code for F1 acceleration crashes (no need for premature de-optimization with :droppoints => :never).

• Availability in "official" stream pending merge from METADATA: Tag InspectDR v0.2.4 [https://github.com/ma-laforge/InspectDR.jl] JuliaLang/METADATA.jl#12075
• Will leave issue open - because algorithm still needs to be made more robust.

[ma-laforge]

Go ahead and try a Pkg.update() to see if the failsafe is ok: You should now get a warning message, but still get a (slowly generated) plot when the algorithm fails.