Complete coverage of JSON Schema production unit tests

[wendellpiez]

User Story:

As an OSCAL developer, I can refer to unit tests to validate JSON Schema production from metaschemas for the different OSCAL models. While this currently functions well, it is still incomplete. A complete set of unit tests will exercise as much of the Metaschema semantics (as they are expressed in JSON) as anyone (ever) uses, and certainly any features of Metaschema that are used in OSCAL models.

Goals:

Unit testing can be trusted to provide solid regression testing going forward.

This unit testing needs to support OSCAL schema production, but also be portable into a standalone Metaschema repository cf usnistgov/OSCAL#500.

Dependencies:

We need some way to determine what "complete" means in this context. The Metaschema XSD should probably define the boundaries.

If/as we document Metaschema semantics, we could tie the documentation into (the) unit tests that illustrate the functionality. This suggests we also go the other way; i.e., as we document the Metaschema semantics, we come back to unit tests to ensure coverage.

Acceptance Criteria

TBD

• All OSCAL website and readme documentation affected by the changes in this issue have been updated. Changes to the OSCAL website can be made in the docs/content directory of your branch.
• A Pull Request (PR) is submitted that fully addresses the goals of this User Story. This issue is referenced in the PR.
• The CI-CD build process runs without any reported errors on the PR. This can be confirmed by reviewing that all checks have passed in the PR.

{The items above are general acceptance criteria for all User Stories. Please describe anything else that must be completed for this issue to be considered resolved.}

[david-waltermire]

@wendellpiez will do a gap analysis of what tests are missing here. We will plan what tests to add based on this review.

[wendellpiez]

Notes here including some preliminary gap analysis:

https://github.com/wendellpiez/metaschema/blob/initial-xsd-unit-tests/test-suite/metaschema-test-coverage.md

It's work in progress. I can make a PR.

[david-waltermire]

The link above is to a private repository. Here is a copy of the linked information:

What we are testing explicitly

Flag definitions

Datatypes

Allowed values

Markdown and character representation

A full suite of markdown conversion tests is probably in order.

Grouping

JSON behaviors but not yet much on XML behaviors. Could use more thorough coverage.

ALL FEATURES

Checked if started - may be partial coverage

• METASCHEMA: Root element of an OSCAL Metaschema metaschema. Defines a family of data structures.
• METASCHEMA/@root
• define-assembly: An element with structured element content in XML; in JSON, an object with properties.
• define-assembly/@name
• model
• define-field: In JSON, an object with a nominal string value (potentially with internal inline - not fully structured - markup). In XML, an element with string or mixed content.
• define-field/@as-type
  • define-field/@collapsible
• define-field/@name
• json-key: In the XML, produces an attribute with the given name, whose value is used as a key value (aka object property name) in the JSON, enabling objects to be 'lifted' out of arrays when such values are distinct. Implies that siblings will never share values. Overloading with datatype 'ID' and naming the key 'id' is legitimate and useful. Even without ID validation, uniqueness of these values among siblings is validable.
• json-key/@flag-name
• group-as: When a given referenced field or assembly must be wrapped in an outer grouping, these settings apply, including a name for the group, and how to express the grouping in the respective formats. Not necessary when a field or assembly has max-occurs='1'
• group-as/@name
• group-as/@in-json: How to represent a grouping in JSON
  • group-as/@in-xml: Whether to represent a grouping explicitly in XML
• json-value-key: Used inside a field definition, designates a flag to be used as a label (key) to be used for the field value in the JSON on the field being defined. When a flag-name is provided, indicates that the value of the field is to be labeled in the JSON with the value of the flag.
• json-value-key/@flag-name
• define-flag: A data point to be expressed as an attribute in the XML or a name/value pair in the JSON. A flag may also be defined implicitly with the assembly or field to which it applies.
• define-flag/@name
• define-flag/@as-type
• formal-name: A formal name for the data construct, to be presented in documentation. It is permissible for a formal name to provide nothing but an expanson of what is already given by a tag (for example, this element could have formal name "Formal name") but it should at the very least confirm the intended semantics for the user, not confuse them.
• namespace: The XML namespace governing the names of elements in XML documents, which expect to be conformant to the schemas expressed by this metaschema. By using this namespace, documents and document fragments used in mixed-format environments may be distinguished from neighbor XML formats using other namespaces. NB this is only for the convenience of XML users; this value is not reflected in OSCAL JSON, and OSCAL applications should not rely on namespaces alone to disambiguate or resolve semantics.
• description: A short description of the data construct, to be inserted into documentation. Unlike 'formal-name' this should not simply repeat what is readily discernible from a tag (element name or JSON label), but say a little more about it.
• remarks: Any explanatory or helpful information to be provided in the documentation of an assembly, field or flag.
  • remarks/@class: Mark as 'XML' for XML-only or 'JSON' for JSON-only remarks.
• schema-name: The name of the information model to be represented by derived schemas.
• schema-version: The version of the information model to be represented by derived schemas.
• short-name: A short (code) name to be used for the metaschema, for example as a constituent of names assigned to derived artifacts such as schemas and conversion utilities.
• import: To import a set of declarations from an out-of-line schema, supporting reuse of common information structures.
  • import/@href: A relative or absolute URI for retrieving an out-of-line metaschema module.
• assembly: Referencing an assembly definition to include an assembly or assemblies of a given type in a model.
• assembly/@ref
  • assembly/@min-occurs: Minimum occurrence of assemblies within a valid model. The default value is 0, for an optional occurrence.
  • assembly/@max-occurs: Maximum occurrence of assemblies within a valid model. The default value is 1, for a single occurrence. 'unbounded' permits any number of assemblies of the designated type.
• field: Referencing a field definition to include a field or fields of a given type in a model.
• field/@ref
  • field/@min-occurs: Minimum occurrence of assemblies within a valid model. The default value is 0, for an optional occurrence.
  • field/@max-occurs: Maximum occurrence of assemblies within a valid model. The default value is 1, for a single occurrence. 'unbounded' permits any number of assemblies of the designated type.
  • field/@in-xml: A field with assigned datatype 'markup-multiline' may be designated for representation with or without a containing (wrapper) element in XML.
• allowed-values: With in-xml='UNWRAPPED', a field contents will be represented in the XML with no wrapper. The field itself will be implicit, to be inferred from the presence of the contents. Among sibling fields in a given model, only one of them may be designated as UNWRAPPED.
• enum: An enumerated value for a flag or field. The value is indicated by the 'value' attribute while the element contents describe the intended semantics for documentation.
• enum/@value: A value recognized for a flag or field.
• flag
• flag/@as-type
• flag/@name
• flag/@ref
• flag/@required
• choice: Within a model, indicates that only one of a set of fields or assemblies, referenced in the choice, may occur in valid instances.
• any: Within a model, a foreign element may be permitted here..
• example
  • example/@href
  • example/@path
• augment: To augment a definition with localized documentation, when a definition from a separate metaschema module is used.
  • augment/@name

[david-waltermire]

This issue needs to be split into two.

1. The gap analysis of what testing is missing (this issue).
2. One or more issues to address these gaps.

We need to build the additional issues. We can create issues in "feature clusters". Something like "assembly", "import", and "example" clusters that contain all gaps in each area.

@wendellpiez Can you create these issues?

[david-waltermire]

I added a unit test for collapsible in the refactored Metaschema project. This project needs to be integrated with the current CI/CD workflow. I'll take care of this as part of this issue.