We are pleased to announce the release of OSCAL 1.0.0 Milestone 1. This release marks an important milestone for the OSCAL project, as this represents the first official release of OSCAL. @@ -36,4 +36,4 @@ The OSCAL team will continue the development of OSCAL focusing our full attentio Our current experimental OSCAL implementation layer is being validated as part of a pilot with GSA/FedRAMP, to ensure the necessary functionality and enough flexibility is provided to support a wide variety of SSPs provided by different organizations. To further validate the implementation layer's functionality and flexibility, NIST is seeking software and service providers that are willing to work with us to represent control implementation information about their products. Please email us at oscal@nist.gov if you are interested. If you have any questions about OSCAL in general or the Milestone 1 release specifically, or if you would like to get involved in the OSCAL project, please contact us at: oscal@nist.gov. -\ No newline at end of file + diff --git a/build/ci-cd/README.md b/build/ci-cd/README.md index 5035a99515..5d0832bef5 100644 --- a/build/ci-cd/README.md +++ b/build/ci-cd/README.md @@ -30,7 +30,7 @@ The bash scripts used in the CI/CD workflow can be run locally in the correct en - [Apache Maven](https://maven.apache.org/) 3.6.1 or greater - [Node.js and Node Package Manager (NPM)](https://nodejs.org/en/) 10.15.3 or greater - 'Another Json Validator' Command Line Interface ([ajv-cli](https://github.com/jessedc/ajv-cli)) v3.0.0 or greater -- [Python](https://www.python.org/) 3.7.3 or greater +- [Python](https://www.python.org/) tested with 3.7.3 Additionally, the following packages need to be installed. @@ -38,9 +38,11 @@ NodeJS Packages: ``` npm install -g ajv-cli +``` Python Packages: +``` pip install lxml ``` @@ -68,3 +70,111 @@ git clone --depth 1 --no-checkout https://github.com/Schematron/schematron.git " cd "$SCHEMATRON_HOME" git checkout master -- trunk/schematron/code ``` + +## Further installation notes + +Setting up to run in WSL (Windows subsystem for Linux) avoids many Windows-related configuration issues, but requires the entire environment be available from within the subsystem. The easiest way to accomplish this is to install the components into the subsystem directly. + +To check whether xmllint is installed: + +``` +xmllint -version +``` + +To install it + +``` +sudo apt install libxml2-utils +``` + +To install Java: + +``` +sudo apt install openjdk-11-jre-headless +``` + +To install NodeJS: + +``` +sudo apt install nodejs +``` + +And npm (Node package manager, separate from NodeJS): + +``` +sudo apt install npm +``` + + +To install AJV (nodeJS JSON validator) + +``` +npm install -g ajv-cli +``` + + +or if you want to install globally (for all users): + +``` +sudo npm install -g ajv-cli +``` + +Ubuntu comes with Python. To check the version: + +``` +python3 -V +``` + +Scripts were tested with version 3.7.3 but 3.6.8 (the default in Ubuntu under WSL) should work. + +Along with Python you also need a library, for which you need pip (a Python installer): + +``` +sudo apt install python-pip +pip install lxml + +``` + + +To install Maven: + +``` +sudo apt install maven +``` + +Then run Maven to acquire Saxon: + + +``` +export SAXON_VERSION=9.9.1-3 +mvn org.apache.maven.plugins:maven-dependency-plugin:2.10:get -DartifactId=Saxon-HE -DgroupId=net.sf.saxon -Dversion=$SAXON_VERSION +``` + +mkdir $HOME/oscal-oss + +export SCHEMATRON_HOME=$HOME/oscal-oss/git-schematron +git clone --depth 1 --no-checkout https://github.com/Schematron/schematron.git "$SCHEMATRON_HOME" +cd "$SCHEMATRON_HOME" +git checkout master -- trunk/schematron/code + + +export OSCAL_TOOLS_DIR=$HOME/oscal-oss/oscal_tools +git clone --depth 1 https://github.com/usnistgov/oscal-tools.git "${OSCAL_TOOLS_DIR}" +cd $OSCAL_TOOLS_DIR/json-cli +mvn clean install + +Finally, export instructions in the preceding must also be copied into .bashrc so they persist in your environment. + +prettyson + +``` +sudo npm install -g prettyjson +``` + + +jq + +``` +sudo apt-get install jq +``` + diff --git a/build/ci-cd/common-environment.sh b/build/ci-cd/common-environment.sh old mode 100644 new mode 100755 diff --git a/build/ci-cd/config/metaschema b/build/ci-cd/config/metaschema index 5cd80002c1..4366041dd6 100644 --- a/build/ci-cd/config/metaschema +++ b/build/ci-cd/config/metaschema @@ -1,4 +1,7 @@ # path to metaschema source|generate schema format(s)|generate converter target format(s)|generate docs format(s) src/metaschema/oscal_catalog_metaschema.xml|xml,json|xml,json|xml,json src/metaschema/oscal_profile_metaschema.xml|xml,json|xml,json|xml,json -src/metaschema/oscal_ssp_metaschema.xml|xml,json|xml,json| +src/metaschema/oscal_component_metaschema.xml|xml,json|xml,json|xml,json +src/metaschema/oscal_ssp_metaschema.xml|xml,json|xml,json|xml,json + + diff --git a/build/ci-cd/copy-and-convert-content.sh b/build/ci-cd/copy-and-convert-content.sh old mode 100644 new mode 100755 index e435dcf460..ae69e97c48 --- a/build/ci-cd/copy-and-convert-content.sh +++ b/build/ci-cd/copy-and-convert-content.sh @@ -69,9 +69,10 @@ while IFS="|" read path format model converttoformats || [[ -n "$path" ]]; do # validate generated file schema="$working_dir/json/schema/oscal_${model}_schema.json" - validate_json "$schema" "$dest" + result=$(validate_json "$schema" "$dest") cmd_exitcode=$? if [ $cmd_exitcode -ne 0 ]; then + echo "${P_ERROR}${result}${P_END}" echo "${P_ERROR}Validation of '${dest}' failed.${P_END}" exitcode=1 fi @@ -79,9 +80,10 @@ while IFS="|" read path format model converttoformats || [[ -n "$path" ]]; do # produce pretty JSON dest_pretty="$working_dir/${newpath}.${altformat}" jq . "$dest" > "$dest_pretty" - validate_json "$schema" "$dest_pretty" + result=$(validate_json "$schema" "$dest_pretty") cmd_exitcode=$? if [ $cmd_exitcode -ne 0 ]; then + echo "${P_ERROR}${result}${P_END}" echo "${P_ERROR}Validation of '${dest_pretty}' failed.${P_END}" exitcode=1 fi diff --git a/build/ci-cd/generate-content-converters.sh b/build/ci-cd/generate-content-converters.sh old mode 100644 new mode 100755 diff --git a/build/ci-cd/generate-model-documentation.sh b/build/ci-cd/generate-model-documentation.sh old mode 100644 new mode 100755 index 56559f18cd..b9b437c1c6 --- a/build/ci-cd/generate-model-documentation.sh +++ b/build/ci-cd/generate-model-documentation.sh @@ -17,6 +17,9 @@ echo "${P_INFO}Working in '${P_END}${working_dir}${P_INFO}'.${P_END}" # the stylesheet used to generate the documentation stylesheet="$OSCALDIR/build/metaschema/xml/produce-and-run-either-documentor.xsl" +# the URL of the github repo where generated artifacts will be accessible +github_url="https://github.com/raw/usnistgov/OSCAL/master" + # the directory to generate the documentation in schema_doc_dir="${working_dir}/docs/content/documentation/schemas" mkdir -p "$schema_doc_dir" # ensure this directory exists @@ -37,7 +40,7 @@ while IFS="|" read path gen_schema gen_converter gen_docs || [[ -n "$path" ]]; d files_to_process="$OSCALDIR"/"$path" - IFS= # disable word splitting + IFS= # disable word splitting for metaschema in $files_to_process do filename=$(basename -- "$metaschema") @@ -61,12 +64,16 @@ while IFS="|" read path gen_schema gen_converter gen_docs || [[ -n "$path" ]]; d # skip blanks continue; fi - + # Run the XSL template for the format case $format in xml) + # determine web location of schema + schema_url="${github_url}/xml/schema/${base}_schema.xsd" ;; json) + # determine web location of schema + schema_url="${github_url}/json/schema/${base}_schema.json" ;; *) echo "${P_WARN}Generating documentation for '${format^^}' is unsupported for '$metaschema'.${P_END}" @@ -78,7 +85,8 @@ while IFS="|" read path gen_schema gen_converter gen_docs || [[ -n "$path" ]]; d xsl_transform "$stylesheet_path" "$metaschema_path" "" \ "target-format=${format}" \ "example-converter-xslt-path=${converter_path}" \ - "output-path=${output_path}" + "output-path=${output_path}" \ + "schema-path=${schema_url}" cmd_exitcode=$? if [ $cmd_exitcode -ne 0 ]; then echo "${P_ERROR}Generating ${format^^} model documentation failed for '${P_END}${metaschema_path}${P_ERROR}'.${P_END}" diff --git a/build/ci-cd/generate-schema.sh b/build/ci-cd/generate-schema.sh old mode 100644 new mode 100755 index b28b6b6dbb..5b94b00ae8 --- a/build/ci-cd/generate-schema.sh +++ b/build/ci-cd/generate-schema.sh @@ -75,7 +75,7 @@ while IFS="|" read path gen_schema gen_converter gen_docs || [[ -n "$path" ]]; d # xml) # TODO: Add support for XML schema validation json) - validate_json "$OSCALDIR/build/ci-cd/json-schema-schema.json" "$schema" + result=$(validate_json "$OSCALDIR/build/ci-cd/json-schema-schema.json" "$schema") cmd_exitcode=$? ;; *) @@ -85,6 +85,7 @@ while IFS="|" read path gen_schema gen_converter gen_docs || [[ -n "$path" ]]; d esac if [ $cmd_exitcode -ne 0 ]; then + echo "${P_ERROR}${result}${P_END}" echo "${P_ERROR}Invalid ${format^^} schema '$schema'.${P_END}" exitcode=1 fi diff --git a/build/ci-cd/init-validate-json.sh b/build/ci-cd/init-validate-json.sh old mode 100644 new mode 100755 index feb917b129..152779b202 --- a/build/ci-cd/init-validate-json.sh +++ b/build/ci-cd/init-validate-json.sh @@ -36,10 +36,14 @@ validate_json() { fi java -cp "$classpath" gov.nist.oscal.json.JsonCLI "$@" "${extra_params[@]}" - - if [ "$?" -ne 0 ]; then - echo "${P_ERROR}Error running JsonCLI.${P_END}" - return 3 + exitcode=$? + if [ "$exitcode" -ne 0 ]; then + if [ "$exitcode" -gt 1 ]; then + echo "${P_ERROR}Error running JsonCLI.${P_END}" + else + echo "${json_file} is invalid." + fi + return $exitcode fi return 0 } diff --git a/build/ci-cd/package-release.sh b/build/ci-cd/package-release.sh old mode 100644 new mode 100755 diff --git a/build/ci-cd/run-all.sh b/build/ci-cd/run-all.sh old mode 100644 new mode 100755 index ed62a400b6..0dc8fe464e --- a/build/ci-cd/run-all.sh +++ b/build/ci-cd/run-all.sh @@ -74,3 +74,4 @@ echo WORKING DIR = "${WORKING_DIR}" [[ "$PERFORM_GENERATION" == "YES" ]] && ("$CIDIR/generate-content-converters.sh" "$WORKING_DIR" "$@" || ("Failed to generate content converters" && exit 4) ) [[ "$PERFORM_CONVERSION" == "YES" ]] && ("$CIDIR/copy-and-convert-content.sh" "$WORKING_DIR" "$@" || ("Failed to convert content" && exit 5) ) [[ "$PERFORM_VALIDATION" == "YES" ]] && ("$CIDIR/validate-round-trips.sh" "$WORKING_DIR" "$@" || ("Failed to validate XML->JSON->XML round-trips" && exit 6) ) +[[ "$PERFORM_GENERATION" == "YES" ]] && ("$CIDIR/generate-model-documentation.sh" "$WORKING_DIR" "$@" || ("Failed to generate website schema model content" && exit 7) ) diff --git a/build/ci-cd/saxon-init.sh b/build/ci-cd/saxon-init.sh old mode 100644 new mode 100755 diff --git a/build/ci-cd/schematron-init.sh b/build/ci-cd/schematron-init.sh old mode 100644 new mode 100755 index 3c4c63a9a4..584084f977 --- a/build/ci-cd/schematron-init.sh +++ b/build/ci-cd/schematron-init.sh @@ -38,18 +38,18 @@ validate_with_schematron() { xsl_transform "$compiled_schematron" "$source_file" "$svrl_result" cmd_exitcode=$? if [ $cmd_exitcode -ne 0 ]; then - echo "${P_ERROR}Processing Schematron '$compiled_schematron' failed for target file '$source_file'${P_END}" + echo "Processing Schematron '$compiled_schematron' failed for target file '$source_file'" return 3 fi # check if the SVRL result contains errors if grep --quiet "failed-assert" "$svrl_result"; then - echo "${P_ERROR}The file '$source_file' has the following Schematron errors:" + echo "The file '$source_file' has the following Schematron errors:" # display the errors xsl_transform "$OSCALDIR/build/ci-cd/svrl-to-plaintext.xsl" "$svrl_result" - echo -n "${P_END}" + echo "" return 1 else - echo "${P_OK}File '$source_file' passed Schematron validation.${P_END}" + echo "File '$source_file' passed Schematron validation." fi } diff --git a/build/ci-cd/setup-ubuntu.sh b/build/ci-cd/setup-ubuntu.sh new file mode 100644 index 0000000000..d1bc00d728 --- /dev/null +++ b/build/ci-cd/setup-ubuntu.sh @@ -0,0 +1,29 @@ +#!/bin/bash +# The following commands will install all build dependencies on Ubuntu. + +sudo apt-get update + +sudo apt-get -y install libxml2-utils nodejs npm python-pip jq + +sudo apt-get -y install openjdk-11-jre-headless maven + +sudo npm install -g ajv-cli prettyjson + +pip install --user lxml + +export SAXON_VERSION=9.9.1-3 +export GITHUB_DIR="$HOME/github" +export SCHEMATRON_HOME="$GITHUB_DIR/Schematron/schematron" +export OSCAL_TOOLS_DIR="$GITHUB_DIR/usnistgov/oscal-tools" + + +mvn org.apache.maven.plugins:maven-dependency-plugin:2.10:get -DartifactId=Saxon-HE -DgroupId=net.sf.saxon -Dversion=$SAXON_VERSION + +git clone --depth 1 --no-checkout https://github.com/Schematron/schematron.git "$SCHEMATRON_HOME" +cd "$SCHEMATRON_HOME" +git checkout master -- trunk/schematron/code + +git clone --depth 1 https://github.com/usnistgov/oscal-tools.git "${OSCAL_TOOLS_DIR}" +cd "$OSCAL_TOOLS_DIR/json-cli" +mvn clean install + diff --git a/build/ci-cd/validate-content.sh b/build/ci-cd/validate-content.sh old mode 100644 new mode 100755 index f801c9e61d..107f683308 --- a/build/ci-cd/validate-content.sh +++ b/build/ci-cd/validate-content.sh @@ -36,18 +36,20 @@ while IFS="|" read path format model converttoformats || [ -n "$path" ]; do case $format in xml) schema="$working_dir/xml/schema/oscal_${model}_schema.xsd" - xmllint --noout --schema "$schema" "$file" + result=$(xmllint --noout --schema "$schema" "$file") cmd_exitcode=$? if [ $cmd_exitcode -ne 0 ]; then + echo "${P_ERROR}${result}${P_END}" echo "${P_ERROR}XML schema validation failed for '$file'.${P_END}" exitcode=1 fi ;; json) schema="$working_dir/json/schema/oscal_${model}_schema.json" - validate_json "$schema" "$file" + result=$(validate_json "$schema" "$file") cmd_exitcode=$? if [ $cmd_exitcode -ne 0 ]; then + echo "${P_ERROR}${result}${P_END}" echo "${P_ERROR}JSON schema validation failed for '$file'.${P_END}" exitcode=1 fi diff --git a/build/ci-cd/validate-metaschema.sh b/build/ci-cd/validate-metaschema.sh old mode 100644 new mode 100755 index d8d3acff6e..4df6a958fa --- a/build/ci-cd/validate-metaschema.sh +++ b/build/ci-cd/validate-metaschema.sh @@ -46,18 +46,20 @@ while IFS="|" read path gen_schema gen_converter gen_docs || [[ -n "$path" ]]; d for metaschema in $files_to_process do echo "${P_INFO}Validating metaschema '$metaschema'${P_END}" - xmllint --noout --schema "$metaschema_lib/metaschema.xsd" "$metaschema" + result=$(xmllint --nowarning --noout --schema "$metaschema_lib/metaschema.xsd" "$metaschema" 2>&1) cmd_exitcode=$? if [ $cmd_exitcode -ne 0 ]; then + echo "${P_ERROR}${result}${P_END}" echo "${P_ERROR}Metaschema '$metaschema' is not valid.${P_END}" exitcode=1 else svrl_result="$working_dir/svrl/${file/$OSCALDIR\/src\//}.svrl" svrl_result_dir=${svrl_result%/*} mkdir -p "$svrl_result_dir" - validate_with_schematron "$compiled_schematron" "$metaschema" "$svrl_result" + result=$(validate_with_schematron "$compiled_schematron" "$metaschema" "$svrl_result") cmd_exitcode=$? if [ $cmd_exitcode -ne 0 ]; then + echo "${P_ERROR}${result}${P_END}" exitcode=1 fi fi diff --git a/build/ci-cd/validate-round-trips.sh b/build/ci-cd/validate-round-trips.sh old mode 100644 new mode 100755 diff --git a/build/metaschema/json/json-schema-metamap.xsl b/build/metaschema/json/json-schema-metamap.xsl index d697a24215..abbfce9a21 100644 --- a/build/metaschema/json/json-schema-metamap.xsl +++ b/build/metaschema/json/json-schema-metamap.xsl @@ -26,6 +26,7 @@