Assertion and core dump with certain filter and JSON with array

[elutz]

Describe the bug
Using this call to jq leads to an error message (assertion) and a core dump:

jq '.[].propB.results[]' jqtest.json

The error message:

assertion "cb == jq_util_input_next_input_cb" failed: file "/usr/src/ports/jq/jq-1.6-1.x86_64/src/jq-1.6/src/util.c", line 371, function: jq_util_input_get_position
Aborted (core dumped)

To Reproduce
Example JSON data (file jqtest.json, see above):

[
  {
    "propA": {
      "results": [
        {
          "text": "Stadtrat"
        }
      ]
    }
  },
  {
    "propB": {
      "results": [
        {
          "id": 2
        },
        {
          "id": 13
        }
      ]
    }
  }
]

This call works:

jq '.[].propB.results' qtest.json

but this not what I want. I want to go deeper in the JSON, e.g.

jq '.[].propB.results[].id' jqtest.json

Raises the same error.

Expected behavior
An output like this:

  {
    "id": 2
  },
  {
    "id": 13
  }

or even this: 2,13

Environment

• Windows 10 and cygwin

$ uname -srvmpio
CYGWIN_NT-10.0 3.1.4(0.340/5/3) 2020-02-19 08:49 x86_64 unknown unknown Cygwin

• jq version

$ jq --version
jq-1.6

[wtlangford]

That definitely shouldn't segfault, and doesn't do so for me on macOS.

Our build toolchain for windows uses MinGW, not cygwin, so the executables we provide run natively on windows. Relatedly, this path: /usr/src/ports/jq/jq-1.6-1.x86_64/src/jq-1.6/src/util.c from your assertion output doesn't match the path I used when I built the jq-1.6 release binaries, so I'm not super clear on what's going on with your binary. How did you install it?

All that said, the expected output for your program should actually be:

jq: error (at jqtest.json:22): Cannot iterate over null (null)

(This is consistent with the segfault you're seeing- the assertion that failed is in code that tries to print the line number where the error occurred)

The issue is that the first object in your input array doesn't have a propB key, so when you try to do .results[] on it, you end up doing null[], and you can't iterate over null.

To get the behavior you want, you could use this jq program instead:

.[].propB.results[]?

Or, something like:

.[].propB.results | select(values)[]

[elutz]

Thanks for the quick answer. The hint with

.[].propB.results[]?

was the right one. The filter language of jq has its own complexity, I have to learn it yet.

I installed jq using the recommend way at cygwin. There is an installer setup-x86_64.exe and this tool is used to update cygwin and packages and to install new ones too.
You can find it on the cygwin page at https://cygwin.com/ or https://cygwin.com/setup-x86_64.exe
The package search reveals some information at https://cygwin.com/cgi-bin2/package-cat.cgi?file=x86_64%2Fjq%2Fjq-1.6-1&grep=jq
Regarding the source packages look here https://cygwin.com/cgi-bin2/package-cat.cgi?file=x86_64%2Fjq-src%2Fjq-1.6-1-src&grep=jq
and here
https://cygwin.com/packages/summary/jq-src.html

Thanks!

[divinity76]

found another way to trigger this assertion:

echo W3sKICAiY29kZV9ldmVudCI6ICIyMzU4NTYyfDI2NXwiLAogICJjb2RlX2V2ZW50X3N5c3RlbSI6ICJTVEFUSUMiLAogICJjb21wYW55X2lkIjogIjMiLAogICJkYXRlX2V2ZW50IjogIjIwMjAtMDMtMTQgMTc6MjA6MDAuMDAwIiwKICAiZGF0ZV9ldmVudF9yZWFsIjogIjAwMDEtMDEtMDEgMDA6MDA6MDAuMDAwIiwKICAiZWNvZGVfY2xhc3MiOiAiMjY1IiwKICAiZWNvZGVfZXZlbnQiOiAiNDgwNCIsCiAgImVwZXJpb2RfZXZlbnQiOiAiIiwKICAiZXRsX2RhdGUiOiAiMjAyMC0wNS0yNyIKfSx7CiAgImNvZGVfZXZlbnQiOiAiMjM1ODU2MnwyNjV8IiwKICAiY29kZV9ldmVudF9zeXN0ZW0iOiAiU1RBVElDIiwKICAiY29tcGFueV9pZCI6ICIzIiwKICAiZGF0ZV9ldmVudCI6ICIyMDIwLTAzLTE0IDE3OjIwOjAwLjAwMCIsCiAgImRhdGVfZXZlbnRfcmVhbCI6ICIwMDAxLTAxLTAxIDAwOjAwOjAwLjAwMCIsCiAgImVjb2RlX2NsYXNzIjogIjI2NSIsCiAgImVjb2RlX2V2ZW50IjogIjQ4MDQiLAogICJlcGVyaW9kX2V2ZW50IjogIiIsCiAgImV0bF9kYXRlIjogIjIwMjAtMDUtMjciCn0KXQ== | base64 -d | jq ".code_event"

outputs:

assertion "cb == jq_util_input_next_input_cb" failed: file "/usr/src/ports/jq/jq-1.6-1.x86_64/src/jq-1.6/src/util.c", line 371, function: jq_util_input_get_position
Aborted (core dumped)

the json in question is:

[{
  "code_event": "2358562|265|",
  "code_event_system": "STATIC",
  "company_id": "3",
  "date_event": "2020-03-14 17:20:00.000",
  "date_event_real": "0001-01-01 00:00:00.000",
  "ecode_class": "265",
  "ecode_event": "4804",
  "eperiod_event": "",
  "etl_date": "2020-05-27"
},{
  "code_event": "2358562|265|",
  "code_event_system": "STATIC",
  "company_id": "3",
  "date_event": "2020-03-14 17:20:00.000",
  "date_event_real": "0001-01-01 00:00:00.000",
  "ecode_class": "265",
  "ecode_event": "4804",
  "eperiod_event": "",
  "etl_date": "2020-05-27"
}
]

[itchyny]

I cannot reproduce the issue, both on macOS and Ubuntu, both on jq-1.6 and latest jq (jq-1.6-207-g98835ee).

[divinity76]

@itchyny I can still reproduce it on jq 1.6:

$ jq --version
jq-1.6
$ echo W3sKICAiY29kZV9ldmVudCI6ICIyMzU4NTYyfDI2NXwiLAogICJjb2RlX2V2ZW50X3N5c3RlbSI6ICJTVEFUSUMiLAogICJjb21wYW55X2lkIjogIjMiLAogICJkYXRlX2V2ZW50IjogIjIwMjAtMDMtMTQgMTc6MjA6MDAuMDAwIiwKICAiZGF0ZV9ldmVudF9yZWFsIjogIjAwMDEtMDEtMDEgMDA6MDA6MDAuMDAwIiwKICAiZWNvZGVfY2xhc3MiOiAiMjY1IiwKICAiZWNvZGVfZXZlbnQiOiAiNDgwNCIsCiAgImVwZXJpb2RfZXZlbnQiOiAiIiwKICAiZXRsX2RhdGUiOiAiMjAyMC0wNS0yNyIKfSx7CiAgImNvZGVfZXZlbnQiOiAiMjM1ODU2MnwyNjV8IiwKICAiY29kZV9ldmVudF9zeXN0ZW0iOiAiU1RBVElDIiwKICAiY29tcGFueV9pZCI6ICIzIiwKICAiZGF0ZV9ldmVudCI6ICIyMDIwLTAzLTE0IDE3OjIwOjAwLjAwMCIsCiAgImRhdGVfZXZlbnRfcmVhbCI6ICIwMDAxLTAxLTAxIDAwOjAwOjAwLjAwMCIsCiAgImVjb2RlX2NsYXNzIjogIjI2NSIsCiAgImVjb2RlX2V2ZW50IjogIjQ4MDQiLAogICJlcGVyaW9kX2V2ZW50IjogIiIsCiAgImV0bF9kYXRlIjogIjIwMjAtMDUtMjciCn0KXQ== | base64 -d | jq ".code_event"

assertion "cb == jq_util_input_next_input_cb" failed: file "/usr/src/ports/jq/jq-1.6-1.x86_64/src/jq-1.6/src/util.c", line 371, function: jq_util_input_get_position
                                                                                                                                                                    Aborted (core dumped)

and yes there really is a bunch of whitespace THEN the Aborted (core dumped) message.
The exact output of jq with base64 encoding, generated with the command

echo W3sKICAiY29kZV9ldmVudCI6ICIyMzU4NTYyfDI2NXwiLAogICJjb2RlX2V2ZW50X3N5c3RlbSI6ICJTVEFUSUMiLAogICJjb21wYW55X2lkIjogIjMiLAogICJkYXRlX2V2ZW50IjogIjIwMjAtMDMtMTQgMTc6MjA6MDAuMDAwIiwKICAiZGF0ZV9ldmVudF9yZWFsIjogIjAwMDEtMDEtMDEgMDA6MDA6MDAuMDAwIiwKICAiZWNvZGVfY2xhc3MiOiAiMjY1IiwKICAiZWNvZGVfZXZlbnQiOiAiNDgwNCIsCiAgImVwZXJpb2RfZXZlbnQiOiAiIiwKICAiZXRsX2RhdGUiOiAiMjAyMC0wNS0yNyIKfSx7CiAgImNvZGVfZXZlbnQiOiAiMjM1ODU2MnwyNjV8IiwKICAiY29kZV9ldmVudF9zeXN0ZW0iOiAiU1RBVElDIiwKICAiY29tcGFueV9pZCI6ICIzIiwKICAiZGF0ZV9ldmVudCI6ICIyMDIwLTAzLTE0IDE3OjIwOjAwLjAwMCIsCiAgImRhdGVfZXZlbnRfcmVhbCI6ICIwMDAxLTAxLTAxIDAwOjAwOjAwLjAwMCIsCiAgImVjb2RlX2NsYXNzIjogIjI2NSIsCiAgImVjb2RlX2V2ZW50IjogIjQ4MDQiLAogICJlcGVyaW9kX2V2ZW50IjogIiIsCiAgImV0bF9kYXRlIjogIjIwMjAtMDUtMjciCn0KXQ== | base64 -d | jq ".code_event" 2>&1 | base64 -w0

is

YXNzZXJ0aW9uICJjYiA9PSBqcV91dGlsX2lucHV0X25leHRfaW5wdXRfY2IiIGZhaWxlZDogZmlsZSAiL3Vzci9zcmMvcG9ydHMvanEvanEtMS42LTEueDg2XzY0L3NyYy9qcS0xLjYvc3JjL3V0aWwuYyIsIGxpbmUgMzcxLCBmdW5jdGlvbjoganFfdXRpbF9pbnB1dF9nZXRfcG9zaXRpb24K

• I can reproduce it on Cygwin but not on Ubuntu nor Windows (tested both jq-win32 and jq-win64) 🤔 (all on 1.6)

[itchyny]

@divinity76 Can you test if the master version fixes the issue?
The latest executable is available here: https://github.com/jqlang/jq/actions/runs/5398715451.

[divinity76]

The latest executable is available here

yeah that's not quite how Cygwin works, that is not a Cygwin-native build, but regardless

I compiled the current master revision ( 7c1efd6 ) on Cygwin and can not reproduce it on current master 👍

$ ./jq --version
jq-master-7c1efd6-dirty
$ echo W3sKICAiY29kZV9ldmVudCI6ICIyMzU4NTYyfDI2NXwiLAogICJjb2RlX2V2ZW50X3N5c3RlbSI6ICJTVEFUSUMiLAogICJjb21wYW55X2lkIjogIjMiLAogICJkYXRlX2V2ZW50IjogIjIwMjAtMDMtMTQgMTc6MjA6MDAuMDAwIiwKICAiZGF0ZV9ldmVudF9yZWFsIjogIjAwMDEtMDEtMDEgMDA6MDA6MDAuMDAwIiwKICAiZWNvZGVfY2xhc3MiOiAiMjY1IiwKICAiZWNvZGVfZXZlbnQiOiAiNDgwNCIsCiAgImVwZXJpb2RfZXZlbnQiOiAiIiwKICAiZXRsX2RhdGUiOiAiMjAyMC0wNS0yNyIKfSx7CiAgImNvZGVfZXZlbnQiOiAiMjM1ODU2MnwyNjV8IiwKICAiY29kZV9ldmVudF9zeXN0ZW0iOiAiU1RBVElDIiwKICAiY29tcGFueV9pZCI6ICIzIiwKICAiZGF0ZV9ldmVudCI6ICIyMDIwLTAzLTE0IDE3OjIwOjAwLjAwMCIsCiAgImRhdGVfZXZlbnRfcmVhbCI6ICIwMDAxLTAxLTAxIDAwOjAwOjAwLjAwMCIsCiAgImVjb2RlX2NsYXNzIjogIjI2NSIsCiAgImVjb2RlX2V2ZW50IjogIjQ4MDQiLAogICJlcGVyaW9kX2V2ZW50IjogIiIsCiAgImV0bF9kYXRlIjogIjIwMjAtMDUtMjciCn0KXQ== | base64 -d | ./jq ".code_event"
jq: error (at <stdin>:21): Cannot index array with string "code_event"

(not sure why it says dirty, i did 0 code changes)

the Cygwin project itself is probably not going to update their bundled jq until 1.6.1 or something is released though 🤔 but it's nice to see that whatever the issue was, it's seemingly no longer in master.

[itchyny]

Thanks, we're working hard for the next release.