call to colorbar not thread safe

[apatlpo]

The following call in xarray/xarray/plot/plot.py does not seem to be thread safe:

            cbar = plt.colorbar(primitive, **cbar_kwargs)

It leads to systematic crashes when distributed, with a cryptic error message (ValueError: Unknown element o). I have to call colorbars outside the xarray plot call to prevent crashes.

A call of the following type may fix the problem:

            cbar = fig.colorbar(primitive, **cbar_kwargs)

But fig does not seem to be available directly in plot.py. Maybe:

            cbar = ax.get_figure().colorbar(primitive, **cbar_kwargs)

cheers

[fmaussion]

Thanks for the report! Would you like to open a PR? I think you are the best placed to see if you proposed fixes solve the problem for you

[apatlpo]

Hi,

Yes why not even though I am not too familiar with the process.

I am not even able to properly install the library so far ... python setup.py install creates the following library: /home1/datahome/aponte/.miniconda3/envs/pangeo/lib/python3.6/site-packages/xarray-0.10.0rc1_2_gf83361c-py3.6.egg/
I am surely doing something wrong, I'd like to have xarray.egg.

[fmaussion]

You are lucky, we just added a contributing guide to xarray: http://xarray.pydata.org/en/latest/contributing.html ;-)

The recommended way to install xarray (and any other local library btw) is:

pip install -e .

[apatlpo]

thanks, got it.
I tried the suggested fix but it did not work.
Unfortunately I don't have more time on the subject for a couple of days. I'll keep you posted when I can sort things out.

[shoyer]

I agree that it is certainly a best practice to avoid relying on global state from the pyplot module.

That said, my experience has been that even if you avoid using pyplot functions matplotlib is still not thread-safe (StackOverflow backs me up here). So you probably will need a thread-lock around your use of any plotting functions, anyways.

[apatlpo]

Ok, thanks, do you have a piece of code with a thread-lock that I could get inspiration from?

[shoyer]

@apatlpo sure, something like:

import threading
import matplotlib.pyplot

MPL_LOCK = threading.Lock()

def save_imshow(data, path):
    with MPL_LOCK:
        fig, ax = plt.subplot()
        ax.imshow(data)
        fig.savefig(path)

save_imshow() is now something you safely call in parallel (e.g., from dask.array.map_blocks()).

[jhamman]

Closing for now. @apatlpo, feel free to reopen if there's more to discuss here.

[apatlpo]

I reopen this issue as it came across my road again when generating figures on a dask.Distributed LocalCluster.
I just open a PR suggesting a change that solves the issue in my situation.

[apatlpo]

hehe. I cannot reopen the issue apparently

[dcherian]

Another way to do this would be to use da.map_blocks(plotfunc, template=...)
with

def plotfunc(da):
    f = plt.figure()
    ...
    f.close()

    return da.time  # or similar

I have used this successfully with a distributed cluster.

[stale]

In order to maintain a list of currently relevant issues, we mark issues as stale after a period of inactivity

If this issue remains relevant, please comment here or remove the stale label; otherwise it will be marked as closed automatically