Improve the coverage of the JSON Schema specification

[rlouf]

In this PR I simplify the codebase by using the referencing library to dereference the specification. We also check the validity of the schema before compiling it. We add coverage for the following:

• Situations where minLength and maxLength are both defined;
• The pattern keyword that allows to constrain the string to follow a regular expression;
• oneOf, allOf, anyOf are now all properly covered;
• type given an array value
• array without a specified type
• enums without a specified type

TODO

• More testing
• Handle optional fields (those not specified in "required")
• minItems for arrays
• maxItems for arrays

Note

We cannot handle minItems and maxItems because of interegular's lack of support for lookaheads:

 interegular.patterns.Unsupported: Group can not have lookbacks/lookaheads that go beyond the group bounds.

Related issues

#330 #215

[janvandervegt-db]

Regarding the minItems and maxItems lookaheads issue, would it be possible if both minItems and maxItems are the same value? As you could generate a regular expression that "hardcodes" the number of items?

[rlouf]

Regarding the minItems and maxItems lookaheads issue, would it be possible if both minItems and maxItems are the same value? As you could generate a regular expression that "hardcodes" the number of items?

We could, would that be useful?

[janvandervegt-db]

For our use case that would be super helpful. We are using LLMs to synthesize a lot of structured data. Frequently a single input maps to a list of different outputs, the exact number is not super important but it needs to be at least a few. Doing them in the same call leads to higher diversity in the outputs and much simpler pipelines.

Two alternatives for us would be:

• Manually provide hardcoded properties for each of the entries (item1, item2, ...)
• Manipulate the JSON schema generated by Pydantic to dynamically do this before passing it to outlines and then include custom parsing logic

Both of these options are quite ugly compared to restricting the guided generation to a fixed number of items in the array.

[rlouf]

Noted, I added the feature. Don't hesitate to open an issue if there is something missing.