You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
The response returned from Amazon MWS is encoded in ISO-8859-1, according to their documentation. However the gem currently does not take this into account and is preserving the encoding of the responses from excon. This will cause an issue later, either when parsing or when persisting the parsed data.
In one case where the chosen parser is ox and the response body happens to contain characters that are valid in ISO-8859-1 but make the strings in the parsed data invalid UTF-8 strings then this will cause an issue when the parsed data is persisted.
As an example of the case above, if we retrieve order data from Amazon and the customer's name happens to be [0x4C, 0xE2, 0x6D].pack('c*') or Lâm in ISO-8859-1, and if we save the data in a JSON attribute in the database, then an Encoding::UndefinedConversionError would be raised, as the encoder expects two more valid bytes after 0xE2.
In the case where the chosen parser is rexml, a REXML::ParseException would be raised when the parser is initiated.
To resolve the issue I am currently monkey patching your gem so that I can pass a middleware forcing the encoding of responses from excon, but I really hope that you patch this issue in the gem. Specifically, I added an encoding attribute to the prolog of the response xml so that ox can detect the encoding.
The author of excon is aware that the gem does not properly handle the encoding of the responses, but closed the issue where the problem is raised. His justification is that users can resolve the issue on their ends by using middlewares.
The response returned from Amazon MWS is encoded in ISO-8859-1, according to their documentation. However the gem currently does not take this into account and is preserving the encoding of the responses from
excon
. This will cause an issue later, either when parsing or when persisting the parsed data.In one case where the chosen parser is
ox
and the response body happens to contain characters that are valid in ISO-8859-1 but make the strings in the parsed data invalid UTF-8 strings then this will cause an issue when the parsed data is persisted.As an example of the case above, if we retrieve order data from Amazon and the customer's name happens to be
[0x4C, 0xE2, 0x6D].pack('c*')
orLâm
in ISO-8859-1, and if we save the data in a JSON attribute in the database, then anEncoding::UndefinedConversionError
would be raised, as the encoder expects two more valid bytes after0xE2
.In the case where the chosen parser is
rexml
, aREXML::ParseException
would be raised when the parser is initiated.To resolve the issue I am currently monkey patching your gem so that I can pass a middleware forcing the encoding of responses from
excon
, but I really hope that you patch this issue in the gem. Specifically, I added anencoding
attribute to the prolog of the response xml so thatox
can detect the encoding.I also want to note that:
excon
is aware that the gem does not properly handle the encoding of the responses, but closed the issue where the problem is raised. His justification is that users can resolve the issue on their ends by using middlewares.Here is a gist containing an example to illustrate the bug
The text was updated successfully, but these errors were encountered: