Skip to content

Commit

Permalink
Add additional notes about writing good tests.
Browse files Browse the repository at this point in the history
Refs: #439
  • Loading branch information
Julian committed Oct 11, 2022
1 parent a6520db commit e17668e
Showing 1 changed file with 15 additions and 1 deletion.
16 changes: 15 additions & 1 deletion CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -57,7 +57,21 @@ The definition of "trivial" is intentionally slightly ambiguous, and intended to
An example of a trivial change is fixing a typo in the README, or bumping a version of a dependency used by the continuous integration suite.
If another contributor takes issue with a change merged in this fashion, simply commenting politely that they have concerns about the change (either in an issue or directly) is the right remedy.

## Test-Specific Changes
## Writing Good Tests

Be familiar with the test structure and assumptions documented in the [README](README.md).

Test cases should include both valid and invalid instances which exercise the test case schema whenever possible.
Exceptions of course include schemas where only one result is ever possible (such as the `false` schema, or ones using keywords which only produce annotations).

Schemas should be *minimal*, by which we mean that they should contain only those keywords which are being tested by the specific test case, and should not contain complex values when simpler ones would do.
The same applies to instances -- prefer simpler instances to more complex ones, and when testing string instances, consider using ones which are self-descriptive whenever it aids readability.

Comments can and should be used to explain tests which are unclear or complex.
The `comment` field is present both for test cases and individual tests for this purpose.
Links to the relevant specification sections are also encouraged, though they can be tedious to maintain from one version to the next.

When adding test cases, they should be added to all past (and future) versions of the specification which they apply to, potentially with minor modifications (e.g. changing `$id` to `id` or accounting for `$ref` not allowing siblings on older drafts).

Changing the schema used in a particular test case should be done with extra caution, though it is not formally discouraged if the change simplifies the schema.
Contributors should not generally append *additional* behavior to existing test case schemas, unless doing so has specific justification.
Expand Down

0 comments on commit e17668e

Please sign in to comment.