csv Saved Data downloads - Updating csv download to support UTF-8 ?

[joshuazong]

Hello,

Wondering how we may set up the .csv file downloads from the Saved Data Adapter to handle "utf-8" encoding? We have various language users where the text submissions include special characters. The submission texts are saved correctly in the saveDataAdapter with the encoding intact, but the downloaded/exported .csv file doesn't handle the encoding. Thus, the .csv files are showing text with character errors.

Thank you for your time! Any guidance is fully appreciated.

Best,
-Joshua Z

[lingxiaoyang]

The encoded characters in simple fields work correctly at my side (encoded utf-8):
View from Plone:

CSV:

However, we have the problem in multiple choice fields. All the encoded characters are displayed as Python repr(), both from Plone and in CSV:

I am not sure if it is because of the version. Our project is using:

    Plone 4.3.10 (4313)
    CMF 2.2.9
    Zope 2.13.24
    Python 2.7.3 (default, Oct 26 2016, 21:01:49) [GCC 4.6.3]
    PIL 3.2.0 (Pillow)

And PloneFormGen 1.7.8

Please let me know if it has been fixed in later versions. I am not quite familiar with Plone ecosystem. Thank you!

[mauritsvanrees]

@joshuazong: I tried with simple accented characters and with some Chinese text copied from WikiPedia: Chinese (汉语/漢語; Hànyǔ or 中文; Zhōngwén).
That showed up fine in OpenOffice and Numbers (Mac).
I think Excel defaults to some Windows encoding, so when opening/importing the csv, you may need to specifically tell Excel to treat it as utf-8, if possible.

The code for creating the csv download is here. It uses self.getCharset(), which actually always returns utf-8, to encode the headers. Then it uses getSavedFormInputForEdit to get the saved data. That last method is also the method that is used for showing the saved data in the browser when you edit the save data adapter. So if it looks good there, the csv should be good too.

@lingxiaoyang: There were a few encoding related changes recently, so it might be that this is solved already. I don't know for sure. The most recent PloneFormGen that works with Plone 4.3 is 1.7.22. You could try that.
Note that in 1.7.9 there were some security fixes which you are missing, so I recommend to update. Do make backups first. :-)

[petschki]

just had a related issue with the CSV export in latest 1.7-maintenance Plone 4.3 setup. Excel still does not convert to utf8 encoding correctly when opening the downloaded CSV directly (though it could be a Excel/Mac problem only). As a workaround you have to choose "Import Data from Textfile" and choose UTF8 encoding manually. As an enhancement I've backported Excel Export functionality from 1.8 to 1.7 because this works like a charm ... see #216. would be nice to make a backport release 1.7.25 ... @mauritsvanrees @smcmahon anyone?

btw. the hidden excel export feature (with installed xlwt package) could be mentioned in the documentation ...