TST: Test non-nanosecond datetimes in PyArrow Parquet dataframes

[EduardAkhmetshin]

This PR tests that the current version of pandas supports non-nanosecond PyArrow dataframes without using timestamp_as_object param.

• closes ENH: Expose to_pandas_kwargs in read_parquet for pyarrow engine #49236
• Tests added and passed if fixing a bug or adding a new feature
• All code checks passed.
• Added type annotations to new arguments/methods/functions.
• Added an entry in the latest doc/source/whatsnew/vX.X.X.rst file if fixing a bug or adding a new feature.

[EduardAkhmetshin]

@natmokval @WillAyd Please review when you have time. Thank you!

[EduardAkhmetshin]

All tests pass in my Docker container, but the CI build fails. I'll look into it.
Update: I see other people's PRs fail with similar errors. Is there a problem with CI?

[WillAyd]

Thanks for the PR, though I'm wondering if this is that much different than the test_read_dtype_backend_pyarrow_config that already exists?

@jbrockmendel does the most with datetimes so may want to chime in here as well

[WillAyd]

Also this does potentially solve the example from the OP in #49236 , although we may not want to make this as closing that. Not sure if there is a need for the requested keyword beyond this one example

[EduardAkhmetshin]

@WillAyd Thank you very much for your review!

I'm wondering if this is that much different than the test_read_dtype_backend_pyarrow_config that already exists?

While test_read_dtype_backend_pyarrow_config tests microseconds as well, it uses a datetime that is in the normal nanosecond date range (1678 - 2262 as mentioned in PyArrow docs). My test uses the date 1600-01-01 from the issue, which is outside the normal range.
Let me know if you think it's redundant and test_read_dtype_backend_pyarrow_config covers everything.

[WillAyd]

Sounds good and thanks for explaining. I think it's ok to add - generally more tests are good.

[WillAyd]

Whoops sorry for the bad guidance on ensure_clean()

[EduardAkhmetshin]

@mroeschke thanks for the suggestions!

[EduardAkhmetshin]

In the failed environment, pandas 1.5.3 is used, which fails with the same error from the original issue ticket.
Is using an old version of pandas in that environment intended for testing backwards compatibility? Should I somehow isolate my test to run in pandas 2 environments only? @WillAyd

[EduardAkhmetshin]

@mroeschke could you please help with my question above? Thanks!

[mroeschke]

You'll probably need specify the minversion argument in pytest.importorskip("pyarrow") call. There's probably a version of pyarrow where this behavior was fixed

[mroeschke]

Thanks @EduardAkhmetshin