You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Writing out a column with pandas type object with no values appears to create a parquet type of INT32 when I would expect it to be BINARY L:STRING or similar. I have a daily process that outputs a set of records to parquet and on days where there are no values in an object column the parquet datatype changes to INT32, thus breaking my process as the schemas has changed relative to previous days.
Expected Output
Output of pd.show_versions()
INSTALLED VERSIONS
commit : db08276
python : 3.7.2.final.0
python-bits : 64
OS : Darwin
OS-release : 19.6.0
Version : Darwin Kernel Version 19.6.0: Thu Jun 18 20:49:00 PDT 2020; root:xnu-6153.141.1~1/RELEASE_X86_64
machine : x86_64
processor : i386
byteorder : little
LC_ALL : None
LANG : None
LOCALE : en_US.UTF-8
For other people running into a similar issue, I was able to work around this by mapping the object type columns in question to StringDtype(). Columns of this type appear to map to BINARY L:STRING in Parquet regardless of their contents.
This still seems like a bug to me though. I can understand there may be a need to default types, but I don't see how INT32 is a reasonable default for the catch-all Pandas type of object
This is happening in our processes as well.
We have some Decimal values that we map to objects but when we try to read those on Spark 3.0 it breaks our pipelines.
Hey @raginjason, I’m facing the same issue. Using StringDtype it writes correctly as BINARY, however my NULLs are being written as ‘None’ (i.e. the string representation), not a real NULL value. Did you face this as well? Wondering how you dealt with that. Thanks a lot!
I have checked that this issue has not already been reported.
I have confirmed this bug exists on the latest version of pandas.
(optional) I have confirmed this bug exists on the master branch of pandas.
Code Sample, a copy-pastable example
Once that runs,
parquet-tools
illustrates the issue. I would expect a datatype such asOPTIONAL BINARY L:STRING
, as in:But instead got a datatype of
OPTIONAL INT32
, as in:Problem description
Writing out a column with pandas type
object
with no values appears to create a parquet type ofINT32
when I would expect it to beBINARY L:STRING
or similar. I have a daily process that outputs a set of records to parquet and on days where there are no values in an object column the parquet datatype changes toINT32
, thus breaking my process as the schemas has changed relative to previous days.Expected Output
Output of
pd.show_versions()
INSTALLED VERSIONS
commit : db08276
python : 3.7.2.final.0
python-bits : 64
OS : Darwin
OS-release : 19.6.0
Version : Darwin Kernel Version 19.6.0: Thu Jun 18 20:49:00 PDT 2020; root:xnu-6153.141.1~1/RELEASE_X86_64
machine : x86_64
processor : i386
byteorder : little
LC_ALL : None
LANG : None
LOCALE : en_US.UTF-8
pandas : 1.1.3
numpy : 1.19.2
pytz : 2020.1
dateutil : 2.8.1
pip : 20.2.3
setuptools : 50.3.0
Cython : None
pytest : None
hypothesis : None
sphinx : None
blosc : None
feather : None
xlsxwriter : None
lxml.etree : None
html5lib : None
pymysql : None
psycopg2 : None
jinja2 : None
IPython : None
pandas_datareader: None
bs4 : None
bottleneck : None
fsspec : None
fastparquet : None
gcsfs : None
matplotlib : None
numexpr : None
odfpy : None
openpyxl : None
pandas_gbq : None
pyarrow : 1.0.1
pytables : None
pyxlsb : None
s3fs : None
scipy : None
sqlalchemy : None
tables : None
tabulate : None
xarray : None
xlrd : None
xlwt : None
numba : None
The text was updated successfully, but these errors were encountered: