use_nullable_dtypes arg removed in dask

[j-bennet]

In dask/dask#10076, use_nullable_dtypes will be removed from _create_dd_meta signature. Making a corresponding change here.

[TomAugspurger]

Thanks for this.

In terms of order for merging these, I think it makes sense to merge #242 first then this.

[TomAugspurger]

Once dask/dask#10076 is merged, I think things will fail in CI until 2023.3.2 is released.

Perhaps we use something like DASK_2023_03_2_DEV which is > Version("2023.3.1") (strictly greater than)?

[j-bennet]

👍

[jorisvandenbossche]

I think it makes sense to merge #242 first then this.

Do you mean first merging it on the dask side? (#242 is this PR)

[TomAugspurger]

Yes, sorry :)

[j-bennet]

It doesn't look like use_nullable_dtypes worked at all with dask-geopandas, even though it's being passed around. I mean, even before the current PR. This is because here:

dask-geopandas/dask_geopandas/io/arrow.py

Line 213 in 474c6c3

return _arrow_to_geopandas(arrow_table)

dask-geopandas calls _arrow_to_geopandas passing the table and nothing else. Then, geopandas converts it without considering any mappers (should be passing mappers as kwargs when calling to_pandas):

https://github.com/geopandas/geopandas/blob/efcb3675d94935ee19b06c75467f9ccc24eb8843/geopandas/io/arrow.py#L364

Here is a small reproducible example. Running it with released dask and geopandas/main, I see that x.dtype=object.

import numpy as np
import geopandas as gp
import pandas as pd

import dask_geopandas as dgp

pdf = gp.GeoDataFrame(
    pd.DataFrame({"x": list("abcdefghij")}),
    geometry=gp.points_from_xy(np.random.randn(10), np.random.randn(10)),
)
outdir = "foo.parquet"
pdf.to_parquet(outdir)

res = dgp.read_parquet(outdir, use_nullable_dtypes=True).compute()
print(f"{res =}")
print(f"{res.dtypes =}")

This PR makes sure that dask-geopandas will not break with dask, but it would not fix the problem.

[jrbourbeau]

Just wanted to check in here, dask/dask#10076 looks good to go. Is it okay to merge that change in on the dask side?

Also, thanks @j-bennet for handling things here too

[jrbourbeau]

Went ahead and merged dask/dask#10076. Let us know if there are still any outstanding issues that need to be resolved here

[TomAugspurger]

Thanks, and sorry for the delay. But thanks to the delay we can use the released version of Dask now.

[martinfleis]

Thanks @j-bennet and @TomAugspurger! I'll merge this now hoping that #251 will get all green afterwards.