diff --git a/README.md b/README.md index 16d7b59fdc..43297eb42b 100644 --- a/README.md +++ b/README.md @@ -1,4 +1,4 @@ -[![Gitter](https://img.shields.io/gitter/room/usnistgov-OSCAL/Lobby)](https://gitter.im/usnistgov-OSCAL/Lobby) [![CI/CD](https://github.com/usnistgov/OSCAL/actions/workflows/status-ci-cd.yml/badge.svg)](https://github.com/usnistgov/OSCAL/actions/workflows/status-ci-cd.yml) [![GitHub release (latest by date)](https://img.shields.io/github/v/release/usnistgov/OSCAL?color=green)](https://github.com/usnistgov/OSCAL/releases) +[![Gitter](https://img.shields.io/gitter/room/usnistgov-OSCAL/Lobby)](https://gitter.im/usnistgov-OSCAL/Lobby) [![CI/CD](https://github.com/usnistgov/OSCAL/actions/workflows/status.yml/badge.svg)](https://github.com/usnistgov/OSCAL/actions/workflows/status.yml) [![GitHub release (latest by date)](https://img.shields.io/github/v/release/usnistgov/OSCAL?color=green)](https://github.com/usnistgov/OSCAL/releases) # Open Security Controls Assessment Language (OSCAL) diff --git a/README_validations.md b/README_validations.md index 76dbd75c08..6a655cf1ee 100644 --- a/README_validations.md +++ b/README_validations.md @@ -8,7 +8,7 @@ Well-formedness also applies to JSON and YAML and their respective parsers. But *validating* goes beyond parsing: it tells you not only that a file can be read successfully, but also that its data structures conform to rules for a particular application of XML, JSON, or YAML. [Well-formed JSON, XML, or YAML is not necessarily valid OSCAL JSON, XML, or YAML](https://pages.nist.gov/OSCAL/concepts/layer/validation/). -If working with XML you will need an XML *schema* (or functional equivalent) to validate your OSCAL. If working with JSON or YAML, you will need a JSON schema. Normative schemas suitable for validating the various OSCAL models are available in the project's source code repository [for XML](https://github.com/usnistgov/OSCAL/tree/main/xml/schema) and [for JSON and YAML](https://github.com/usnistgov/OSCAL/tree/main/json/schema). +If working with XML you will need an XML *schema* (or functional equivalent) to validate your OSCAL. If working with JSON or YAML, you will need a JSON schema. Normative schemas suitable for validating the various OSCAL models are [made available as part of a release](https://github.com/usnistgov/OSCAL/releases/latest/) and [can also be built locally](./build/README.md). ## What are the OSCAL models? @@ -83,13 +83,13 @@ In this case the processor reports the syntax issues it detects, to the extent p A validation error tells you, first and foremost, that your file is incorrect in a basic way. For example, an element (data structure) may be out of place, or missing a required piece. Tools that fail to implement schema validation, misconfigured tools, data quality issues, workflow issues, or many other causes may be the cause. They also vary in how easy they are to correct. The most common problem is that the wrong schema is being used. -The most important consideration in correcting a validation error is selecting the appropriate schema and the appropriate version of that schema. OSCAL maintains backward compatibility across all schemas as expressed through [the use of semantic versioning](https://github.com/usnistgov/OSCAL/blob/main/docs/content/downloads/_index.md#oscal-releases). +The most important consideration in correcting a validation error is selecting the appropriate schema and the appropriate version of that schema. OSCAL maintains backward compatibility across all schemas as expressed through [the use of semantic versioning](./versioning-and-branching.md). A *well-formedness error*, however, tells you your file is incorrect in a more fundamental way. It is both more severe (further processing cannot continue) but also easier to address by using tools correctly. In a mature system, well-formedness errors should be rare, and when they do occur it is because preliminary checks for well-formedness were not implemented or ignored. ## How can I correct my errors? -End users will ordinarily find it easiest and best to edit a file in an OSCAL tool with a graphical user interface or a structured editor. These would be tools that are especially built or configured to support changing the file while respecting the rulesof well-formedness and validating against the rules of an intended schema. +End users will ordinarily find it easiest and best to edit a file in an OSCAL tool with a graphical user interface or a structured editor. These would be tools that are especially built or configured to support changing the file while respecting the rules of well-formedness and validating against the rules of an intended schema. In order to build such tools and support other operations, developers may also do this work. Commercial off-the-shelf developer environments may support these basic operations for a range of formats. diff --git a/build/README.md b/build/README.md index 18d4e81533..d1606c2694 100644 --- a/build/README.md +++ b/build/README.md @@ -13,7 +13,9 @@ The build tools in this directory require a Unix environment with the following Additionally ensure your environment has standard build tools such as [`git` 2.34.1 or newer](https://git-scm.com/), [`make` 4.3 or newer](https://www.gnu.org/software/make/), [`zip` 3.0 or newer](https://infozip.sourceforge.net/), and [`bash` 3.2 or newer for syntax we require](https://git.savannah.gnu.org/cgit/bash.git/tree/NEWS?h=bash-3.2-beta). -## Overview +## Building + +### Overview All OSCAL build targets are defined in a [`Makefile`](./Makefile). To summarize the build targets, run `make help`. @@ -22,7 +24,7 @@ To generate all artifacts and run all checks, simply run `make all`. To run targets in parallel, pass a `--jobs ` flag, where `` is the number of concurrent jobs and should be `<=` your machine's core count. Note that running jobs concurrently will make the output harder to follow and debug. -## Artifact Generation +### Artifact Generation As of `v1.1.0`, artifacts are no longer tracked in the OSCAL git repository ([discussion link](https://github.com/usnistgov/OSCAL/discussions/1852)). Instead artifacts are [generated as part of a release](../.github/workflows/release.yml). @@ -34,8 +36,7 @@ Developers can also generate individual artifacts using the following commands: * `make schemas`: Generates the JSON Schemas and XSDs off of the source Metaschemas; * `make converters`: Generates the XSLT stylesheets for JSON<->XML conversion off of the source Metaschemas. -## Checks - +### Checks Developers can run checks using the `make checks` command. @@ -46,7 +47,7 @@ The checks can also be run individually: * `make validate-xsds`: Validates that the XSD files generated by `make schemas` are schema valid against the XSD specification; * `make test-profile-resolution`: Runs the test suite in [`/src/utils/resolver-pipeline`](../src/utils/resolver-pipeline/). -## Archive Generation +### Archive Generation `make archives` generates `.zip` and `.tar.bz2` archives with the [generated artifacts](#archive-generation). @@ -55,3 +56,126 @@ By default, archives will have the suffix `SNAPSHOT`, but this can be overridden ```sh $ make archive RELEASE=my_release ``` + +## Artifact Usage + +### Schemas + +The OSCAL project provides JSON Schemas based on the [JSON Schema Draft-07](https://json-schema.org/specification-links.html#draft-7) and XML Schema based on the [XML Schema Definition Language (XSD) 1.1](https://www.w3.org/TR/xmlschema11-1/) for all [OSCAL models](https://pages.nist.gov/OSCAL/documentation/schema/). + +After running `make schemas`, JSON Schemas and XSD files will be created in the `generated/` directory, following the general pattern `oscal_[MODEL_NAME]_schema.json` and `oscal_[MODEL_NAME]_schema.xsd`. + +Pre-built schemas are also distributed as part of a [release](https://github.com/usnistgov/OSCAL/releases). + +#### Validating OSCAL JSON Content + +The OSCAL project uses the [Another Json Validator](https://ajv.js.org/) (AJV) [command line interface](https://github.com/jessedc/ajv-cli) (CLI) to perform automated validation of all JSON-based OSCAL [content](https://github.com/usnistgov/oscal-content) provided in this repository. *ajv-cli* is an open source tool that can be [installed](https://github.com/jessedc/ajv-cli#installation), and run on OSX, Linux, and Windows environments, with a [Node.js](https://nodejs.org/en/download/current/) environment installed. Node.js is the JavaScript runtime environment that is required to run AJV. + +The following example uses the **ajv-cli** to perform validation of an OSCAL catalog JSON file. + +``` +ajv validate -s "oscal_catalog_schema.json" -d "catalog.json" --extend-refs=true --verbose +``` + +The [online documentation](https://github.com/jessedc/ajv-cli) for *ajv-cli* provides more information on the command-line arguments. + +#### Validating OSCAL XML Content + +The OSCAL project uses [xmllint](http://xmlsoft.org/xmllint.html) to perform automated validation of all XML-based OSCAL [content](https://github.com/usnistgov/oscal-content) provided in this repository. *xmllint* is an open source tool that can be [downloaded](http://xmlsoft.org/downloads.html), installed, and run on OSX, Linux, and Windows environments. + +The following example uses **xmllint** to perform validation of an OSCAL catalog XML file. + +``` +xmllint --noout --schema "oscal_catalog_schema.xsd" "catalog.xml" +``` + +The [online documentation](http://xmlsoft.org/xmllint.html) for *xmllint* provides more information on the command-line arguments. + +### Converters + +The OSCAL project provides Extensible Stylesheet Language Transformation (XSLT) templates based on [XSLT 3.0](https://www.w3.org/TR/xslt-30/) and [XPath 3.1](https://www.w3.org/TR/xpath-31/) for all [OSCAL models](https://pages.nist.gov/OSCAL/documentation/schema/). + +**IMPORTANT** : XSLT 3.0 and XPath 3.1 are only required when using the NIST provided tools for converting OSCAL content between JSON and XML. Any version of XSLT or XPath may be used when transforming or querying OSCAL files for other reasons. These newer versions of XSLT and XPath offer more robust capabilities, that are needed to support the OSCAL converters. + +After running `make converters`, converter stylesheets will be created in the `generated/` directory, following the general pattern `oscal_[MODEL_NAME]_json-to-xml-converter.xsl` and `oscal_[MODEL_NAME]_xml-to-json-converter.xsl`. + +Pre-built converters are also distributed as part of a [release](https://github.com/usnistgov/OSCAL/releases). + +#### Converting OSCAL XML Content to JSON + +The OSCAL project uses [Saxon-HE (Saxon Home Edition)](http://saxon.sourceforge.net/) to evaluate the XSLT templates supporting conversion of OSCAL XML and JSON [content](https://github.com/usnistgov/oscal-content) provided in this repository. *Saxon-HE* is an open source implementation of XSLT 3.0, XPath 2.0 and 3.1, and XQuery 3.1 supporting Java and .NET programming environments. These versions of *Saxon-HE* can be [downloaded directly](http://saxon.sourceforge.net/#F9.9HE) or the Java version can be [downloaded](https://search.maven.org/artifact/net.sf.saxon/Saxon-HE) using Apache Maven. Saxonica also offers Saxon PE and EE versions, which are commercial products with technical support and redistribution rights. + +The OSCAL project uses *Saxon-HE* with Java version 8 or greater. + +The following example uses **Saxon HE** to convert an OSCAL catalog XML file to JSON using one of the NIST-provided JSON to XML XSLT converters. This example assumes that Java 8+ has been installed and the Saxon-HE jar files have already unzipped. + +``` +java -jar "saxon10he.jar" -xsl:"oscal_catalog_xml-to-json-converter.xsl" -s:"oscal-catalog.xml" -o:"oscal-catalog.json" json-indent=yes +``` + +> *Note*: at time of writing, Saxon 9 users are being encouraged to upgrade systems to use Saxon 10, and the stylesheets provided should function equally well or better with the later software. Please feel free to use the latest Saxon or indeed any conformant XSLT 3.0 processor. +> +> Operators of XSLT-based platforms should by all means test these processes with any XSLT 3.0 conformant processor, and report problems to us via Github Issues. + +Paths\names of these files need to be provided based on the location of the files on your computer: + +* The Saxon JAR file is named ```saxon10he.jar```. +* The catalog converter is specified as ```-xsl:"oscal_catalog_xml-to-json-converter.xsl"``` +* The source catalog XML file is specified as ```-s:"oscal-catalog.xml"``` +* The destination catalog JSON file is specified as ```-o:"oscal-catalog.json"```. + +The [online documentation](http://www.saxonica.com) for *Saxon* provides more information on the command line arguments. + +##### Alternate invocations + +The configuration just provided will convert a JSON file given as a file reference, into OSCAL XML. There are also different configurations available for debugging: + +* `-it:from-xml` (indicating initial template) - provides the default XSLT entry point explicitly. +* `-file:mycatalog.xml` used with explicit `-it:from-xml` will look for the XML at the location given by the parameter, instead of on the source port (given by `-s`). This configuration parallels the JSON-to-XML converter. +* `produce=supermodel` as a runtime parameter will emit not OSCAL XML, but the intermediate format produced by the converter (a so-called 'OSCAL supermodel' derived from its metaschema): useful for debugging or as a pivot to other serializations. +* `produce=xpath-json` will produce the results in an XML format defined by the XPath function `json-to-xml()`, which when consumed by the complementary function `xml-to-json()` can deterministically provide syntactically correct JSON. This XML format is used internally to provide the JSON data description, to be cast into JSON as a (terminal) serialization step. Expressing the transformation results in this format directly can provide insight for debugging, especially in failure conditions (when the syntax cannot be written). See https://www.w3.org/TR/xpath-functions-31/#func-json-to-xml for more details on this format. + +#### Converting OSCAL JSON Content to XML + +The OSCAL project uses [Saxon-HE (Saxon Home Edition)](http://saxon.sourceforge.net/) to evaluate the XSLT templates supporting conversion of OSCAL XML and JSON [content](https://github.com/usnistgov/oscal-content) provided in this repository. *Saxon-HE* is an open source implementation of XSLT 3.0, XPath 2.0 and 3.1, and XQuery 3.1 supporting Java and .NET programming environments. These versions of *Saxon-HE* can be [downloaded directly](http://saxon.sourceforge.net/#F9.9HE) or the Java version can be [downloaded](https://search.maven.org/artifact/net.sf.saxon/Saxon-HE) using Apache Maven. Saxonica also offers Saxon PE and EE versions, which are commercial products with technical support and redistribution rights. + +The OSCAL project uses *Saxon-HE* with Java version 8 or greater. + +The following example uses **Saxon HE** to convert an OSCAL catalog JSON file to XML using one of the NIST-provided JSON to XML XSLT converters. This example assumes that has been installed and the Saxon-HE jar files have already unzipped. + +``` +java -jar "saxon10he.jar" -xsl:"oscal_catalog_json-to-xml-converter.xsl" -o:"oscal-catalog.xml" -it:from-json file="oscal-catalog.json" +``` + +> *Note*: at time of writing, Saxon 9 users are being encouraged to upgrade systems to use Saxon 10, and the stylesheets provided should function equally well or better with the later software. Please feel free to use the latest Saxon or indeed any conformant XSLT 3.0 processor. +> +> Operators of XSLT-based platforms should by all means test these processes with any XSLT 3.0 conformant processor, and report problems to us via Github Issues. + +`-it` indicates the initial template (XSLT entry point) should be `from-json`. (As documented in the source code, other entry points are offered for diagnostics or integration.) + +Paths/names given to other settings are offered as relative or absolute system paths or URIs: + +* The Saxon JAR file is named ```saxon9he.jar``` (system path). +* The catalog converter is specified as ```-xsl:"oscal_catalog_json-to-xml-converter.xsl"``` (relative or absolute URI) +* The source catalog JSON file is specified as ```file="oscal-catalog.json"``` (URI) +* The destination catalog XML file is specified as ```-o:"oscal-catalog.xml"``` (URI) + +The [online documentation](http://www.saxonica.com) for *Saxon* provides more information on the command line arguments. + +##### Alternate invocations + +The configuration just provided will convert a JSON file given as a file reference, into OSCAL XML. However it may sometimes be convenient (for example when invoking programmatically) to pass the JSON into the converter as a literal. + +Use the `json` runtime parameter to do this. On the command line this might be: + +``` +java -jar "saxon10he.jar" -xsl:"oscal_catalog_json-to-xml-converter.xsl" -o:"oscal-catalog.xml" -it:from-json json="{ "catalog": {} }" +``` + +(With allowances made for quote marks etc.) + +There are also different configurations available for debugging: + +* `-it` (initial template) `from-xdm-json-xml` - assume the source is not given as a URI reference to a file, but as XML conformant to the model returned by the XPath function 'json-to-xml()'. In this case, the `file` parameter must point to this XML file not a JSON file. +* Alternatively, `-s:file.xml` (with or instead of `-it`) will operate the same way, except finding the XML at `file.xml`. +* `produce=supermodel` as a runtime parameter will emit not OSCAL XML, but the intermediate format produced by the converter (a so-called 'OSCAL supermodel' derived from its metaschema): useful for debugging or as a pivot to other serializations. diff --git a/src/release/content-upgrade/README.md b/src/release/content-upgrade/README.md index ad10186bd7..810474fef7 100644 --- a/src/release/content-upgrade/README.md +++ b/src/release/content-upgrade/README.md @@ -22,7 +22,7 @@ java -cp Saxon-HE-10.6.jar net.sf.saxon.Transform -xsl:oscal-rc2-v1-0-0-update.x ### Content Upgrades with Oxygen XML Editor -If you prefer the use of an integrated development environment (IDE), you can use [the Oxygen XML Editor](https://www.oxygenxml.com/) to [configure and execute XSLT transforms](https://www.oxygenxml.com/doc/ug-oxygen/topics/xslt-tab.html). This IDE embeds a Java Runtime Environment and mutliple versions of the Saxon XSLT processor for you convenience. +If you prefer the use of an integrated development environment (IDE), you can use [the Oxygen XML Editor](https://www.oxygenxml.com/) to [configure and execute XSLT transforms](https://www.oxygenxml.com/doc/ug-oxygen/topics/xslt-tab.html). This IDE embeds a Java Runtime Environment and multiple versions of the Saxon XSLT processor for you convenience. After you installed the Oxygen XML Editor, to transform a sample OSCAL document `catalog_v1.0.0-rc2.xml`, upgrade its content from v1.0.0 RC2 to v1.0., and save the result in the file `catalog_v1.0.0.xml`, you can configure the XSLT transform to your preference in the XSLT tab or like so: @@ -45,7 +45,7 @@ Please review each transform (or [contact us for more help](https://pages.nist.g By default, you will upgrade content in a document instance and it does not explicitly add a `xsi:schemaLocation` attribute every time. Often, `xsi:schemaLocation` can be a local path or remote URI that is frequently out of date. The `xsi:schemaLocation` is an empty string by default (`''`), and it will not be included and the transform will emit a warning message. -If you do wish to explicit add `xsi:schemaLocation`, you can apply the transform on the commmand line like so: +If you do wish to explicit add `xsi:schemaLocation`, you can apply the transform on the command line like so: ```sh java -cp Saxon-HE-10.6.jar net.sf.saxon.Transform -xsl:oscal-rc2-v1-0-0-update.xsl -s:catalog_v1.0.0-rc2.xml -o:catalog_v1.0.0.xml schema-location="'http://csrc.nist.gov/ns/oscal/1.0 ../../../../../xml/schema/oscal_catalog_schema.xsd'" @@ -58,11 +58,7 @@ Similarly, you can specify this parameter in [the XSLT transform's' parameter co ### What do you do if you have OSCAL documents in JSON or YAML format? -To upgrade content in JSON or YAML format, you will use the approach outlined below. - -1. Convert the JSON or YAML file into XML with [the appropriate content converter](https://github.com/usnistgov/OSCAL/tree/main/xml). -1. Upgrade the XML content from the current version to target version. -1. Conver the upgraded target content from XML back to JSON or YAML with [the appropriate content converter](https://github.com/usnistgov/OSCAL/tree/main/json). +See the [converter artifact usage section of the build README](../../../build/README.md#converters). ### What do you do if you are several versions behind? diff --git a/versioning-and-branching.md b/versioning-and-branching.md index 7f5b0342c4..b5fbacc1d5 100644 --- a/versioning-and-branching.md +++ b/versioning-and-branching.md @@ -98,7 +98,7 @@ Once the release is ready, the release can be made using the following Git comma ``` git checkout main git merge --no-ff release-1.2 -git tag -a 1.2.0 +git tag -a v1.2.0 git push --follow-tags ```