You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
As best as I've been able to tell, the JSON module currently provided
by wren-essentials does not directly support the efficient entity-by-entity
reading of JSON texts in files with streams of JSON entities, unless
each such entity is on a separate line.
Consider for example a file with the four lines:
{
"a:" 1
}
2
This illustrates what is no doubt the main need, but it is worth
keeping in mind that streams of JSON entities can be presented without
any newlines at all, e.g.:
1 2 3
For reference, jq's input built-in meets the requirements I have in mind.
A related but also quite distinct need is to be able to process very large JSON entities
in a "stream-oriesnted manner" (in the sense of streaming XML parsers).
Again, jq's --stream option provides a convenient point of reference, especially
since jq allows this option to be used in conjunction with streams of JSON texts.
The text was updated successfully, but these errors were encountered:
joshgoebel
changed the title
ER: reading a stream of JSON texts
Request: Read additional JSON objects from a Stream
Nov 17, 2021
The underlying parser pdjson (which we wrap) would seem to suggest this is supported:
By default only one value is read from the stream. The parser can be reset to read more objects. The overall line number and position are preserved.
So I'd imagine it's something that could be added if someone wanted to hook things up (and decide on an appropriate API). Perhaps even as simple as exposing access to json_reset from JSONStream... I think we'd be open to a PR (and tests) if it's truly that simple.
Of course at this point you're already working with a string object... doing something like streaming from a file as you're reading it (in the MOST performant way possible) might be a lot harder. I'd suggest that even without any changes that something simple like JSONL could be parsed by what we have now with a little glue on top of the input stream to first split it into lines, etc...
As best as I've been able to tell, the JSON module currently provided
by wren-essentials does not directly support the efficient entity-by-entity
reading of JSON texts in files with streams of JSON entities, unless
each such entity is on a separate line.
Consider for example a file with the four lines:
This illustrates what is no doubt the main need, but it is worth
keeping in mind that streams of JSON entities can be presented without
any newlines at all, e.g.:
For reference, jq's
input
built-in meets the requirements I have in mind.A related but also quite distinct need is to be able to process very large JSON entities
in a "stream-oriesnted manner" (in the sense of streaming XML parsers).
Again, jq's
--stream
option provides a convenient point of reference, especiallysince jq allows this option to be used in conjunction with streams of JSON texts.
The text was updated successfully, but these errors were encountered: