changed url for rasterio network test

[scottyhq]

fix failing rasterio network test by simplifying test and using same image url as rasterio library test suite

• [x ] Closes test_rasterio_vrt_network is failing in continuous integration tests #3083

[max-sixty]

Thanks @scottyhq !

Did we disable some tests / builds before? If so, shall we re-enable them as part of this PR? (I remember seeing something but wasn't involved directly)

[fmaussion]

Yes, network test are not run on travis: https://github.com/pydata/xarray/blob/1a5b6305be4261f04b130a84f7780c9b5dafc943/conftest.py

[scottyhq]

but seems like the tests are now run with each build on Azure? https://dev.azure.com/xarray/xarray/_build/results?buildId=389&view=ms.vss-test-web.build-test-results-tab

the new test currently still fails - I think due to rasterio/rasterio#1709

cc @shoyer who created the issue linked in the first post.

[max-sixty]

It looks like the flakey test failed but that Azure counts it as passing: https://dev.azure.com/xarray/xarray/_build/results?buildId=389&view=logs

...similar to an allow_failures

[max-sixty]

@scottyhq would you prefer to merge this as-is as an improvement, or wait until the test is fixed?

[scottyhq]

I'm in no rush to merge this (just wanted to help get rid of the red on Azure :). I think it's important to have network tests since it's increasingly common to be streaming data from s3, gcs, etc rather than reading locally. But I'm not sure of best practices for setting such tests up, so any suggestions for modifications are welcome.

[max-sixty]

OK great. Let's at least try and get this passing and turn it green.

Do you have a better idea of why it's still broken? Is it an issue with rasterio (re your comment above)? Is it fixed on master? If not we should open an issue there. If so we could install a later version on that build

[shoyer]

The build is already green because this is marked as an allowed failure, but it shows up in the list of failing tests anyways. It's a little confusing but that was the best way I found to make the result of these tests noticeable without counting as a build failure.

If you click on "Checks" at the top of this PR and then click on the top level "pydata.xarray" section of Azure, you see the full details on why the test is failing still:

self = <xarray.tests.test_backends.TestRasterio object at 0x7fb4b5cfa358>

    @network
    def test_rasterio_vrt_network(self):
        # Make sure loading w/ rasterio give same results as xarray
        import rasterio
        # use same url that rasterio package uses in tests
        prefix = "https://landsat-pds.s3.amazonaws.com/L8/139/045/"
        image = "LC81390452014295LGN00/LC81390452014295LGN00_B1.TIF"
        httpstif = prefix + image
>       with rasterio.Env(aws_unsigned=True):

xarray/tests/test_backends.py:3797: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
/usr/share/miniconda/envs/xarray-tests/lib/python3.6/site-packages/rasterio/env.py:193: in __init__
    aws_unsigned=aws_unsigned)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <rasterio.session.AWSSession object at 0x7fb4b4b10da0>, session = None
aws_unsigned = True, aws_access_key_id = None, aws_secret_access_key = None
aws_session_token = None, region_name = None, profile_name = None
requester_pays = False

    def __init__(
            self, session=None, aws_unsigned=False, aws_access_key_id=None,
            aws_secret_access_key=None, aws_session_token=None,
            region_name=None, profile_name=None, requester_pays=False):
        """Create a new boto3 session
    
        Parameters
        ----------
        session : optional
            A boto3 session object.
        aws_unsigned : bool, optional (default: False)
            If True, requests will be unsigned.
        aws_access_key_id : str, optional
            An access key id, as per boto3.
        aws_secret_access_key : str, optional
            A secret access key, as per boto3.
        aws_session_token : str, optional
            A session token, as per boto3.
        region_name : str, optional
            A region name, as per boto3.
        profile_name : str, optional
            A shared credentials profile name, as per boto3.
        requester_pays : bool, optional
            True if the requester agrees to pay transfer costs (default:
            False)
        """
>       import boto3
E       ModuleNotFoundError: No module named 'boto3'

It looks like using the aws_unsigned argument means that boto3 is required, but we don't have that installed in our CI currently. It would be straightforward to add it whenever we install rasterio -- put it in the lists in ci/requirements/

[scottyhq]

ok. as noted rasterio/rasterio#1709 the aws_unsigned=True should bypass importing boto3 in upcoming rasterio 1.0.25. But I've gone ahead and added boto3 to ci requirements.yml files in case people want to add other tests reading from s3, gcs, etc. in the future

[shoyer]

Thanks @scottyhq !