Support arrow:schema in Parquet writer to faithfully roundtrip duration types with Arrow

[mhaseeb123]

Description

Closes #15847

This PR adds the support to construct and write base64-encoded serialized arrow:schema-type IPC message to parquet file footer to allow faithfully roundtrip with Arrow via Parquet for duration type.

Answered

• Only construct and write arrow:schema if asked by the user via store_schema argument (cudf) or write_arrow_schema (libcudf). i.e. Default these variables to false otherwise.
• The internal/libcudf variable name for store_schema can stay write_arrow_schema and it should be fine. This has been done to disambiguate which schema (arrow or parquet) we are talking about.
• Separate PR: int96_timestamps cannot be deprecated/removed in cuDF as Spark is actively using it. Remove INT96 timestamps in cuDF Parquet writer #15901
• cuDF Parquet writer supports decimal32 and decimal64 fixed types. These are not directly supported by Arrow so we will convert decimal32/decimal64 columns to decimal128.
• is_col_nullable() function moved to writer_impl_helpers.cpp along with some other helper functions.
• A common convert_data_to_decimal128 can be separated out and used in writer_impl.cu and to_arrow.cu. Tracking in a separate issue. [FEA] Deduplicate convert_data_to_decimal128() function #16194

CC @vuule @etseidl @nvdbaranec @GregoryKimball @galipremsagar for vis.

Checklist

• I am familiar with the Contributing Guidelines.
• New or existing tests cover these changes.
• The documentation is up to date with these changes.

[nvdbaranec]

Generally looks great. Just a couple small things so far.

[lithomas1]

Python changes look good to me (just 1 comment).

Might want to have @vyasr or @galipremsagar take another look though.

[vuule]

great stuff!
just a few comments

[vuule]

Merging these into a single kernel (per decimal type) would be great for performance with many decimal columns, but looks like it would not fit well into the recursive implementation.
How about a stream pool? 4-8 streams that we use in round robin order might help when we have to convert may decimal columns.

[mhaseeb123]

Tracking this in a separate issue #16194.

[bdice]

Is there a reason we think this should be False by default? It seems like faithful roundtrips with Arrow would be a benefit by default. However, it seems like enabling this feature will cast / convert some data types (e.g. "days" aren't supported, and only decimal128 is supported -- if I read the rest of this PR correctly). Are those conversions potentially lossy / do they change metadata? If so, are those conversions worth documenting?

[mhaseeb123]

A couple of reasons why we chose to set it False by default. Actually it's great that you asked this so this will be documented here.

One arrow:schema should only be used when we want to round-trip certain col types (primarily durations for now) with arrow only. Otherwise, cudf roundtrips with itself and arrow perfectly fine.

Second, cudf still supports int96 timestamps as Spark has actively been using it. Enabling store_schema breaks Sparks existing and future workflows requiring them to set this to False whenever using int96 timestamps.

Third, like you mentioned, we can't roundtrip decimal32 and decimal64 with cuDF itself with arrow::schema by default without (losslessly) converting them to decimal128.

To summarize, things work perfectly fine without arrow::schema for the most part until we need to faithfully round-trip duration with arrow only in which case it is enabled.

@galipremsagar please feel free to add any reasons that we discussed during the cuIO standup meeting a couple weeks ago.

[bdice]

Thanks! This kind of information would be good to have in the docstrings!

[mhaseeb123]

/merge