Fixes behaviour for incomplete lines when recover_with_nulls is enabled

[elstehle]

Description

Closes #14227. Adapts the behaviour of the JSON finite-state transducer (FST) when recover_with_nulls is true to be more strict and reject lines that contain incomplete JSON objects (aka records) or JSON arrays (aka lists).

Checklist

• I am familiar with the Contributing Guidelines.
• New or existing tests cover these changes.
• The documentation is up to date with these changes.

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[vuule]

/ok to test

[vuule]

could you leave a comment on the changes in logic in the FST (how does this translate into the intended behavior change)? I can glean some logic from the code, but it's possibly made up :D

[elstehle]

I've added comments to the code (get_translation_table and the nl_tokens lambda).

Essentially, we just changed the behaviour when we're in recover_from_error, such that we fail if the JSON line is an incomplete JSON value. E.g.,

• when we see a newline while we are still parsing a LIST or a STRUCT, we emit an ErrorBegin token that will mark this incomplete JSON line as invalid.
• when we see a newline while we are still parsing a string or field name (e.g., {"a":"123\n) , we emit an ErrorBegin token that will mark this incomplete JSON line as invalid.

[vuule]

Thank you for the comments.
It's still challenging to understand case-to-case why regular_tokens and recovering_tokens might be different, but I think I got it.

[karthikeyann]

/ok to test

[elstehle]

/merge