jq on Linux On Power Big Endian

[ghost]

Hi,

I'm using a compiled version of jq on Linux on Power (RHEL 6.5)

Something like that is working great

echo '{"foo": "42", "bar": "less interesting data"}' | jq .foo

"42"
If json "leaf" is not behing double quote jq stay stucked

echo '{"foo": 42, "bar": "less interesting data"}' | jq .foo

STUCK HERE

All precompiled versions found on the web have the same issue. Try the same syntax on X86_64 is working like a charm.

Does anybody have an idea ?

Thank you.
B - chmod666.org

[pkoppstein]

@chmod666org - Thanks to @slapresta I'll retract my comments, but if at all possible, I'd suggest you compile from source.

[ghost]

@pkoppstein Note the first one has "42" as a string, but the second one has 42 as an integer. I don't understand much about these low-level things but it sounds like some endianness issue to me.

[nicowilliams]

echo '{"foo": 42, "bar": "less interesting data"}' | jq .foo

STUCK HERE

Without access to the system in question it's hard to say, but what you
could do is get a stack trace with gdb:

$ echo '{"foo": 42, "bar": "less interesting data"}' > f
$ gdb --args jq .foo f
(gdb) run
<wait a bit>
^C
(gdb) where
<save the output and post it here>

Also, what does this do:

$ jq -n 42

?

Hopefully it hangs the same way.

Also, what about this:

$ jq -n '"42"|fromjson'

?

The hang is most likely to be occurring in jvp_strtod() in
jv_dtoa.c.

It might be useful to look at the pre-processed version of jv_dtoa.c:

$ make
$ rm jv_dtoa.*o
$ make V=1
...
$ # Now run the same command, but add the -S option (gcc), drop the
$ # -o ..., and post the output in a gist.

so we have a better idea of what's going on.

[nicowilliams]

Maybe we should have an option to use sscanf() and snprintf() for dealing with double<->string conversion, but the result wouldn't parse all JSON texts.

[ghost]

Hi all,
First of all thank for all your comments, I really appreciate your work on it.

@nicowilliams I've also tried with a version compiled by myself on Linux On Power (BE) and the issue is the same.
I'll try tomorrow with gdb an post the ouput here, and give you all my feedback.
I think the rpm version of jq found on the net in rpm format were not tested at all .... anyway I'll give you a feedback as soon as possible.

Once again thanks.

[ghost]

Hi all, here is the gdb output

gdb --args jq .foo f

GNU gdb (GDB) Red Hat Enterprise Linux (7.2-60.el6_4.1)
Copyright (C) 2010 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later http://gnu.org/licenses/gpl.html
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "ppc64-redhat-linux-gnu".
For bug reporting instructions, please see:
http://www.gnu.org/software/gdb/bugs/...
Reading symbols from /usr/local/bin/jq...done.
(gdb) run
Starting program: /usr/local/bin/jq .foo f
^C
Program received signal SIGINT, Interrupt.
mult (C=, a=0xfffb7ab0010, b=) at jv_dtoa.c:805
805 z = *x++ * (ULLong)y + *xc + carry;
Missing separate debuginfos, use: debuginfo-install glibc-2.12-1.132.el6.ppc64
(gdb) where
#0 mult (C=, a=0xfffb7ab0010, b=) at jv_dtoa.c:805
#1 0x0000000010013b7c in pow5mult (C=0xfffffffc9e0, b=0xfffb7bc0010, k=) at jv_dtoa.c:889
#2 0x0000000010014384 in jvp_dtoa (C=0xfffffffc9e0, dd=, mode=, ndigits=, decpt=0xfffffffc7a0, sign=, rve=0xfffffffc7a8) at jv_dtoa.c:3937
#3 0x0000000010015394 in jvp_dtoa_fmt (C=0xfffffffc9e0, b=0xfffffffc8d8 "", x=) at jv_dtoa.c:4221
#4 0x000000001001241c in jv_dump_term (C=0xfffffffc9e0, x=..., flags=, indent=, F=0xfffb7f606a0, S=0x0) at jv_print.c:149
#5 0x0000000010012c2c in jv_dump (x=..., flags=) at jv_print.c:238
#6 0x0000000010023c24 in process (value=..., flags=) at main.c:87
#7 0x0000000010024500 in main (argc=, argv=) at main.c:274

[nicowilliams]

@chmod666org What does the ABI on your system say for C doubles? Are they 64-bit IEEE754? And if so, are they big- or little-endian? If they aren't 64-bit IEEE754 doubles, is there a compiler option to get 64-bit IEEE754 doubles?

EDIT: 20e6292 isn't wrong, I just read it wrong.

[ghost]

..can you tell me how to check that ? I'm sorry I'm not a developper and don't really know how to check it.
Anyway, thanks a lot for your help

[nicowilliams]

@chmod666org What does uname -a say? I think I can figure it out from that.

[ghost]

@nicowilliams Linux powervchost 2.6.32-431.el6.ppc64 #1 SMP Sun Nov 10 22:17:43 EST 2013 ppc64 ppc64 ppc64 GNU/Linux

[nicowilliams]

Endianness is the same for integers as for floating point types on ppc64.

[nicowilliams]

This is easily reproduced on x86 by adding this in jv_dtoa.c:

#ifdef IEEE_8087
#undef IEEE_8087
#define IEEE_MC68k
#endif

then running ./jq -n 42; jq then loops tightly here:

804                         do {
805                                 z = *x++ * (ULLong)y + *xc + carry;
806                                 carry = z >> 32;
807                                 *xc++ = z & FFFFFFFF;
808                                 }
809                                 while(x < xae);

[khepler]

It's an issue on 32 and 64 bit PowerPC Macs as well, although Macports compiled jq on both systems for ppc7400 (32 bit).

[lukenels]

Did anybody find a solution for this? I'm trying to run jq on AIX and I think it's having the same issue

[nicowilliams]

Basically, I have no time for this at this time. It's very hairy code that I haven't yet understood. Realistically, I won't manage to debug this for a few months given my current schedule. Maybe someone can take a look? @wtlangford ? @stedolan ?

[lukenels]

I fixed it by removing
#define IEEE_8087
and putting
#define IEEE_MC68k
for anyone still curious

[nicowilliams]

On Thu, Jun 18, 2015 at 04:09:33PM -0700, Luke Nelson wrote:

I fixed it by removing
#define IEEE_8087
and putting
#define IEEE_MC68k
for anyone still curious

That does it? Fine, I'll push a fix then.

Thanks!

[nicowilliams]

@lukenels @khepler @chmod666org What is the autoconf $host string on your systems?

[khepler]

Would that be the

host='powerpc-apple-darwin9.8.0'

line from config.log?

[nicowilliams]

@khepler Yes, that. I also need one for AIX.

[nicowilliams]

Though it's probably enough to match powerpc-*.

[khepler]

That would match my powerpc-apple-darwin8.11.0 too. :) And hopefully the linux partitions as well, which I believe is 'powerpc-unknown-linux' (that's from memory and shouldn't be trusted).

[earonesty]

jq -n 42 fails on all big endian systems i've tested with the same error. most likely, it's the big integer library, which makes lots of bitwise operator assumptions. maybe "https://gist.github.com/bloopletech/338338"?

[khepler]

What versions and architectures did you test? jq 1.5 works on the big endian system I just tested.