Added artifact usage + corrected dead links (#1865)

README.md

@@ -1,4 +1,4 @@
-[![Gitter](https://img.shields.io/gitter/room/usnistgov-OSCAL/Lobby)](https://gitter.im/usnistgov-OSCAL/Lobby) [![CI/CD](https://github.com/usnistgov/OSCAL/actions/workflows/status-ci-cd.yml/badge.svg)](https://github.com/usnistgov/OSCAL/actions/workflows/status-ci-cd.yml) [![GitHub release (latest by date)](https://img.shields.io/github/v/release/usnistgov/OSCAL?color=green)](https://github.com/usnistgov/OSCAL/releases)
+[![Gitter](https://img.shields.io/gitter/room/usnistgov-OSCAL/Lobby)](https://gitter.im/usnistgov-OSCAL/Lobby) [![CI/CD](https://github.com/usnistgov/OSCAL/actions/workflows/status.yml/badge.svg)](https://github.com/usnistgov/OSCAL/actions/workflows/status.yml) [![GitHub release (latest by date)](https://img.shields.io/github/v/release/usnistgov/OSCAL?color=green)](https://github.com/usnistgov/OSCAL/releases)
 
 # Open Security Controls Assessment Language (OSCAL)
 

README_validations.md

@@ -8,7 +8,7 @@ Well-formedness also applies to JSON and YAML and their respective parsers.
 
 But *validating* goes beyond parsing: it tells you not only that a file can be read successfully, but also that its data structures conform to rules for a particular application of XML, JSON, or YAML. [Well-formed JSON, XML, or YAML is not necessarily valid OSCAL JSON, XML, or YAML](https://pages.nist.gov/OSCAL/concepts/layer/validation/).
 
-If working with XML you will need an XML *schema* (or functional equivalent) to validate your OSCAL. If working with JSON or YAML, you will need a JSON schema. Normative schemas suitable for validating the various OSCAL models are available in the project's source code repository [for XML](https://github.com/usnistgov/OSCAL/tree/main/xml/schema) and [for JSON and YAML](https://github.com/usnistgov/OSCAL/tree/main/json/schema).
+If working with XML you will need an XML *schema* (or functional equivalent) to validate your OSCAL. If working with JSON or YAML, you will need a JSON schema. Normative schemas suitable for validating the various OSCAL models are [made available as part of a release](https://github.com/usnistgov/OSCAL/releases/latest/) and [can also be built locally](./build/README.md).
 
 ## What are the OSCAL models?
 
@@ -83,13 +83,13 @@ In this case the processor reports the syntax issues it detects, to the extent p
 
 A validation error tells you, first and foremost, that your file is incorrect in a basic way. For example, an element (data structure) may be out of place, or missing a required piece. Tools that fail to implement schema validation, misconfigured tools, data quality issues, workflow issues, or many other causes may be the cause. They also vary in how easy they are to correct. The most common problem is that the wrong schema is being used.
 
-The most important consideration in correcting a validation error is selecting the appropriate schema and the appropriate version of that schema. OSCAL maintains backward compatibility across all schemas as expressed through [the use of semantic versioning](https://github.com/usnistgov/OSCAL/blob/main/docs/content/downloads/_index.md#oscal-releases).
+The most important consideration in correcting a validation error is selecting the appropriate schema and the appropriate version of that schema. OSCAL maintains backward compatibility across all schemas as expressed through [the use of semantic versioning](./versioning-and-branching.md).
 
 A *well-formedness error*, however, tells you your file is incorrect in a more fundamental way. It is both more severe (further processing cannot continue) but also easier to address by using tools correctly. In a mature system, well-formedness errors should be rare, and when they do occur it is because preliminary checks for well-formedness were not implemented or ignored.
 
 ## How can I correct my errors?
 
-End users will ordinarily find it easiest and best to edit a file in an OSCAL tool with a graphical user interface or a structured editor. These would be tools that are especially built or configured to support changing the file while respecting the rulesof well-formedness and validating against the rules of an intended schema.
+End users will ordinarily find it easiest and best to edit a file in an OSCAL tool with a graphical user interface or a structured editor. These would be tools that are especially built or configured to support changing the file while respecting the rules of well-formedness and validating against the rules of an intended schema.
 
 In order to build such tools and support other operations, developers may also do this work. Commercial off-the-shelf developer environments may support these basic operations for a range of formats.
 

build/README.md

@@ -13,7 +13,9 @@ The build tools in this directory require a Unix environment with the following
 
 Additionally ensure your environment has standard build tools such as [`git` 2.34.1 or newer](https://git-scm.com/), [`make` 4.3 or newer](https://www.gnu.org/software/make/), [`zip` 3.0 or newer](https://infozip.sourceforge.net/), and [`bash` 3.2 or newer for syntax we require](https://git.savannah.gnu.org/cgit/bash.git/tree/NEWS?h=bash-3.2-beta).
 
-## Overview
+## Building
+
+### Overview
 
 All OSCAL build targets are defined in a [`Makefile`](./Makefile).
 To summarize the build targets, run `make help`.
@@ -22,7 +24,7 @@ To generate all artifacts and run all checks, simply run `make all`.
 To run targets in parallel, pass a `--jobs <n>` flag, where `<n>` is the number of concurrent jobs and should be `<=` your machine's core count.
 Note that running jobs concurrently will make the output harder to follow and debug.
 
-## Artifact Generation
+### Artifact Generation
 
 As of `v1.1.0`, artifacts are no longer tracked in the OSCAL git repository ([discussion link](https://github.com/usnistgov/OSCAL/discussions/1852)).
 Instead artifacts are [generated as part of a release](../.github/workflows/release.yml).
@@ -34,8 +36,7 @@ Developers can also generate individual artifacts using the following commands:
 * `make schemas`: Generates the JSON Schemas and XSDs off of the source Metaschemas;
 * `make converters`: Generates the XSLT stylesheets for JSON<->XML conversion off of the source Metaschemas.
 
-## Checks
-
+### Checks
 
 Developers can run checks using the `make checks` command.
 
@@ -46,7 +47,7 @@ The checks can also be run individually:
 * `make validate-xsds`: Validates that the XSD files generated by `make schemas` are schema valid against the XSD specification;
 * `make test-profile-resolution`: Runs the test suite in [`/src/utils/resolver-pipeline`](../src/utils/resolver-pipeline/).
 
-## Archive Generation
+### Archive Generation
 
 `make archives` generates `.zip` and `.tar.bz2` archives with the [generated artifacts](#archive-generation).
 
@@ -55,3 +56,126 @@ By default, archives will have the suffix `SNAPSHOT`, but this can be overridden
 ```sh
 $ make archive RELEASE=my_release
 ```
+
+## Artifact Usage
+
+### Schemas
+
+The OSCAL project provides JSON Schemas based on the [JSON Schema Draft-07](https://json-schema.org/specification-links.html#draft-7) and XML Schema based on the [XML Schema Definition Language (XSD) 1.1](https://www.w3.org/TR/xmlschema11-1/) for all [OSCAL models](https://pages.nist.gov/OSCAL/documentation/schema/).
+
+After running `make schemas`, JSON Schemas and XSD files will be created in the `generated/` directory, following the general pattern `oscal_[MODEL_NAME]_schema.json` and `oscal_[MODEL_NAME]_schema.xsd`.
+
+Pre-built schemas are also distributed as part of a [release](https://github.com/usnistgov/OSCAL/releases).
+
+#### Validating OSCAL JSON Content
+
+The OSCAL project uses the [Another Json Validator](https://ajv.js.org/) (AJV) [command line interface](https://github.com/jessedc/ajv-cli) (CLI) to perform automated validation of all JSON-based OSCAL [content](https://github.com/usnistgov/oscal-content) provided in this repository. *ajv-cli* is an open source tool that can be [installed](https://github.com/jessedc/ajv-cli#installation), and run on OSX, Linux, and Windows environments, with a [Node.js](https://nodejs.org/en/download/current/) environment installed. Node.js is the JavaScript runtime environment that is required to run AJV.
+
+The following example uses the **ajv-cli** to perform validation of an OSCAL catalog JSON file.
+
+```
+ajv validate -s "oscal_catalog_schema.json" -d "catalog.json" --extend-refs=true --verbose
+```
+
+The [online documentation](https://github.com/jessedc/ajv-cli) for *ajv-cli* provides more information on the command-line arguments.
+
+#### Validating OSCAL XML Content
+
+The OSCAL project uses [xmllint](http://xmlsoft.org/xmllint.html) to perform automated validation of all XML-based OSCAL [content](https://github.com/usnistgov/oscal-content) provided in this repository. *xmllint* is an open source tool that can be [downloaded](http://xmlsoft.org/downloads.html), installed, and run on OSX, Linux, and Windows environments.
+
+The following example uses **xmllint** to perform validation of an OSCAL catalog XML file.
+
+```
+xmllint --noout --schema "oscal_catalog_schema.xsd" "catalog.xml"
+```
+
+The [online documentation](http://xmlsoft.org/xmllint.html) for *xmllint* provides more information on the command-line arguments.
+
+### Converters
+
+The OSCAL project provides Extensible Stylesheet Language Transformation (XSLT) templates based on [XSLT 3.0](https://www.w3.org/TR/xslt-30/) and [XPath 3.1](https://www.w3.org/TR/xpath-31/) for all [OSCAL models](https://pages.nist.gov/OSCAL/documentation/schema/).
+
+**IMPORTANT** : XSLT 3.0 and XPath 3.1 are only required when using the NIST provided tools for converting OSCAL content between JSON and XML. Any version of XSLT or XPath may be used when transforming or querying OSCAL files for other reasons. These newer versions of XSLT and XPath offer more robust capabilities, that are needed to support the OSCAL converters.
+
+After running `make converters`, converter stylesheets will be created in the `generated/` directory, following the general pattern `oscal_[MODEL_NAME]_json-to-xml-converter.xsl` and `oscal_[MODEL_NAME]_xml-to-json-converter.xsl`.
+
+Pre-built converters are also distributed as part of a [release](https://github.com/usnistgov/OSCAL/releases).
+
+#### Converting OSCAL XML Content to JSON
+
+The OSCAL project uses [Saxon-HE (Saxon Home Edition)](http://saxon.sourceforge.net/) to evaluate the XSLT templates supporting conversion of OSCAL XML and JSON [content](https://github.com/usnistgov/oscal-content) provided in this repository. *Saxon-HE* is an open source implementation of XSLT 3.0, XPath 2.0 and 3.1, and XQuery 3.1 supporting Java and .NET programming environments. These versions of *Saxon-HE* can be [downloaded directly](http://saxon.sourceforge.net/#F9.9HE) or the Java version can be [downloaded](https://search.maven.org/artifact/net.sf.saxon/Saxon-HE) using Apache Maven. Saxonica also offers Saxon PE and EE versions, which are commercial products with technical support and redistribution rights.
+
+The OSCAL project uses *Saxon-HE* with Java version 8 or greater.
+
+The following example uses **Saxon HE** to convert an OSCAL catalog XML file to JSON using one of the NIST-provided JSON to XML XSLT converters. This example assumes that Java 8+ has been installed and the Saxon-HE jar files have already unzipped.
+
+```
+java -jar "saxon10he.jar" -xsl:"oscal_catalog_xml-to-json-converter.xsl" -s:"oscal-catalog.xml" -o:"oscal-catalog.json" json-indent=yes
+```
+
+> *Note*: at time of writing, Saxon 9 users are being encouraged to upgrade systems to use Saxon 10, and the stylesheets provided should function equally well or better with the later software. Please feel free to use the latest Saxon or indeed any conformant XSLT 3.0 processor.
+>
+> Operators of XSLT-based platforms should by all means test these processes with any XSLT 3.0 conformant processor, and report problems to us via Github Issues.
+ 
+Paths\names of these files need to be provided based on the location of the files on your computer:
+
+* The Saxon JAR file is named ```saxon10he.jar```.
+* The catalog converter is specified as ```-xsl:"oscal_catalog_xml-to-json-converter.xsl"```
+* The source catalog XML file is specified as ```-s:"oscal-catalog.xml"```
+* The destination catalog JSON file is specified as ```-o:"oscal-catalog.json"```.
+
+The [online documentation](http://www.saxonica.com) for *Saxon* provides more information on the command line arguments.
+
+##### Alternate invocations
+
+The configuration just provided will convert a JSON file given as a file reference, into OSCAL XML. There are also different configurations available for debugging:
+
+* `-it:from-xml` (indicating initial template) - provides the default XSLT entry point explicitly.
+* `-file:mycatalog.xml` used with explicit `-it:from-xml` will look for the XML at the location given by the parameter, instead of on the source port (given by `-s`). This configuration parallels the JSON-to-XML converter.
+* `produce=supermodel` as a runtime parameter will emit not OSCAL XML, but the intermediate format produced by the converter (a so-called 'OSCAL supermodel' derived from its metaschema): useful for debugging or as a pivot to other serializations.
+* `produce=xpath-json` will produce the results in an XML format defined by the XPath function `json-to-xml()`, which when consumed by the complementary function `xml-to-json()` can deterministically provide syntactically correct JSON. This XML format is used internally to provide the JSON data description, to be cast into JSON as a (terminal) serialization step. Expressing the transformation results in this format directly can provide insight for debugging, especially in failure conditions (when the syntax cannot be written). See https://www.w3.org/TR/xpath-functions-31/#func-json-to-xml for more details on this format.
+
+#### Converting OSCAL JSON Content to XML
+
+The OSCAL project uses [Saxon-HE (Saxon Home Edition)](http://saxon.sourceforge.net/) to evaluate the XSLT templates supporting conversion of OSCAL XML and JSON [content](https://github.com/usnistgov/oscal-content) provided in this repository. *Saxon-HE* is an open source implementation of XSLT 3.0, XPath 2.0 and 3.1, and XQuery 3.1 supporting Java and .NET programming environments. These versions of *Saxon-HE* can be [downloaded directly](http://saxon.sourceforge.net/#F9.9HE) or the Java version can be [downloaded](https://search.maven.org/artifact/net.sf.saxon/Saxon-HE) using Apache Maven. Saxonica also offers Saxon PE and EE versions, which are commercial products with technical support and redistribution rights.
+
+The OSCAL project uses *Saxon-HE* with Java version 8 or greater.
+
+The following example uses **Saxon HE** to convert an OSCAL catalog JSON file to XML using one of the NIST-provided JSON to XML XSLT converters. This example assumes that has been installed and the Saxon-HE jar files have already unzipped.
+
+```
+java -jar "saxon10he.jar" -xsl:"oscal_catalog_json-to-xml-converter.xsl" -o:"oscal-catalog.xml" -it:from-json file="oscal-catalog.json"
+```
+
+> *Note*: at time of writing, Saxon 9 users are being encouraged to upgrade systems to use Saxon 10, and the stylesheets provided should function equally well or better with the later software. Please feel free to use the latest Saxon or indeed any conformant XSLT 3.0 processor.
+>
+> Operators of XSLT-based platforms should by all means test these processes with any XSLT 3.0 conformant processor, and report problems to us via Github Issues.
+
+`-it` indicates the initial template (XSLT entry point) should be `from-json`. (As documented in the source code, other entry points are offered for diagnostics or integration.)
+
+Paths/names given to other settings are offered as relative or absolute system paths or URIs:
+
+* The Saxon JAR file is named ```saxon9he.jar``` (system path).
+* The catalog converter is specified as ```-xsl:"oscal_catalog_json-to-xml-converter.xsl"``` (relative or absolute URI)
+* The source catalog JSON file is specified as ```file="oscal-catalog.json"``` (URI)
+* The destination catalog XML file is specified as ```-o:"oscal-catalog.xml"``` (URI)
+
+The [online documentation](http://www.saxonica.com) for *Saxon* provides more information on the command line arguments.
+
+##### Alternate invocations
+
+The configuration just provided will convert a JSON file given as a file reference, into OSCAL XML. However it may sometimes be convenient (for example when invoking programmatically) to pass the JSON into the converter as a literal.
+
+Use the `json` runtime parameter to do this. On the command line this might be:
+
+```
+java -jar "saxon10he.jar" -xsl:"oscal_catalog_json-to-xml-converter.xsl" -o:"oscal-catalog.xml" -it:from-json json="{ "catalog": {} }"
+```
+
+(With allowances made for quote marks etc.)
+
+There are also different configurations available for debugging:
+
+* `-it` (initial template) `from-xdm-json-xml` - assume the source is not given as a URI reference to a file, but as XML conformant to the model returned by the XPath function 'json-to-xml()'. In this case, the `file` parameter must point to this XML file not a JSON file.
+* Alternatively, `-s:file.xml` (with or instead of `-it`) will operate the same way, except finding the XML at `file.xml`.
+* `produce=supermodel` as a runtime parameter will emit not OSCAL XML, but the intermediate format produced by the converter (a so-called 'OSCAL supermodel' derived from its metaschema): useful for debugging or as a pivot to other serializations.