-
-
Notifications
You must be signed in to change notification settings - Fork 205
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
A suite of vexatious JSON Schemas #589
Comments
Prior discussion on the general idea testing of undefined behavior:
Prior (and current) discussion about testing particular undefined behaviors: |
(CC @gregsdennis in particular even though you've probably already been pinged, since yeah obviously this is related to the |
I agree with this being incorporated into the test suite, but it'll need to be JSON, not YAML. |
Yeah, same here, especially for it to go in this repo. @notEthan if you're not dying on the hill :) probably that's some bikeshedding we can immediately get out of the way. In terms of "pivoting" so that we don't copy paste tests across drafts as we do for normal tests... though I've thought of doing that before for normal tests I ultimately decided against it since things get quite hairy with all the combinations we have :/ But I don't know that I object for these new ones, if they're anyways different, and in a different directory, I think I'm fine with them having a different structure in that sense. |
I would though, to continue light bikeshedding, prefer to not have spaces in the filenames, and in particular to try to have some sort of similar naming structure where filenames roughly correspond to keywords. |
Another option would be to have a build step (converting YAML to JSON), which might be useful for other reasons as well (such as consistently inserting I don't feel strongly about this, but if we're doing some light bikeshedding, it's worth a thought. |
I have started writing a collection of schemas that violate some part of JSON schema's specifications or are in some way vexatious to an implementation. This is a topic that has come up a number of times over the years, and in the past couple days. My work on this is very initial, starting to integrate it with my own implementation's tests, but fleshed out enough to share for discussion.
I have a readme started here: https://github.com/notEthan/JSON-Schema-Vex-Suite/pull/1/files
The few schemas I've put together so far here: https://github.com/notEthan/JSON-Schema-Vex-Suite/pull/2/files
and a schema describing those files here: https://github.com/notEthan/JSON-Schema-Vex-Suite/pull/3/files
My goal is to have at least one nonconformant schema for every MUST/MUST NOT which apply to schemas of specs my own implementation supports. Other schemas will also fit - where a spec states behavior is undefined or implementation-defined, or where behavior is not forbidden but also not specified (e.g.
pattern: "("
,$ref: "/nowhere"
)This may be something that could be adopted by json-schema-org, or integrated into JSON-Schema-Test-Suite. It generally shares motivation and principles with the test suite, but I made no attempt to follow the format or structure here. Others will have opinions on decisions I made writing this collection; I'll describe decisions that seem relevant.
I wrote it in YAML, not JSON. As a human I find it nicer to read and SO much nicer to write. Most people whose opinions I'm aware of agree, but certainly not universal. I'd prefer to keep it YAML but it's not a hill I'd die on. There will be an automated check that all of the YAML is limited to JSON's data model.
The structure is pretty different from JSON-Schema-Test-Suite. I have, at the moment:
cases
; when the directives vary by specification (e.g. the keyword beingid
or$id
), thecases
array allows multiple combinations of specifications and applicable schemas.schemas
object property name. This is a dot-separated string, with each part indicating some scope, broadest to narrowest. This part feels a little underdeveloped, but the intention is that implementations can filter using these property names to group similar behaviors. e.g.keyword.$schema.type.null
andkeyword.$schema.type.integer
uniquely identify schemas, butkeyword.$schema.type.*
would likely all be handled the same.I am interested in feedback and thoughts on how people see this in relation to the JSON Schema organization.
The text was updated successfully, but these errors were encountered: