Metaschema / XSLT implementation alignment (usnistgov#197)

* Relocate schema resources (usnistgov#191)

- Moved schema resources out of XSLT implementation
- Relocated XSD and datatype XML schemas.
Removed junk and generated files.
Many datatype adjustments
   - Tweeks to schema data types to address unicode issues. Resolves usnistgov/OSCAL#1127. Resolves usnistgov/OSCAL#956.
   - Adjusted type names of Metaschema types to be purely hyphenated.
   - Removed extra character ranges that are causing C# problems. Resolves usnistgov/OSCAL#1127.
- Adjusted URI data type names to be more consistent between XML and JSON.
Many metaschema.xsd adjustments
- Alignment of data types used with new Metschema datatype names.
- Additional repairs of tests, including naming consistency.
   - adding new unit test for a valid json-value key using a label
   - many fixes to broken tests
   - Removed duplicate uuid test scenario.
   - Many fixes to make unit test Metaschemas valid.
- Added JSON schema schema.
- Some schema generation refactoring to support new data-driven test harness.

* Fixed URI of choice unit test.

* Update metaschema-datatypes.xsd

Adjusting whitespace handling for URI types.

* adding leading and trailing whitespace checks to all XML schema datatype derived types

* Adjustments to debug charstrings.

* Update metaschema-datatypes.json

Minor adjustments to data types for comments on #1260.

* refactored metaschema schema to support external constraint definitions

* Added constraint extensibility configuration.

* added formal-name and description to definition references

* XSLT M4 Metaschema pipeline updates (usnistgov#214)

* Composition unit tests now valid to updated schemas; added (missing) tests.

* Adding XSpec for schema generation; nominal schema target examples

* Adding XSpec tests (testing schema generation as transforms) and initial set of targets for testing

* Major reorganization and archiving (some temporary, prior to defenestration); new tests; readme documents in each folder for schema-generation unit tests to help trace efforts
* More details in json-value-key readme; updated top-level JSON schema generation xspec
* Metaschema Schematron rule intercepting a json-key setting with no BY_KEY in the grouping logic
* Cleaned up extra JSON Schema file
* Updated metaschema model wrt json-key, json-value-key flag-ref (no longer flag-name); other Metaschema touchups
* Rewiring and simplifying XSD production pipeline with datatypes now acquired from the normative metaschema /schema subdirectory - removed the namespace fixup step, no longer needed.
* Updated anthology ('menagerie') metaschema example, with cleanup
* In XSD generation, now emitting datatype definitions only for datatypes actually needed by a schema
* Updating schema TODO notes on synching unit tests; bit of cleanup
* Addressing usnistgov#212 - XML to JSON converter no longer chokes on edge cases (array items of SINGLETON_OR_ARRAY groups)
* Updating top-level composition unit tests
* .gitignore covering HTML reports from test runs
* Removing outdated testing results
* Adding back support for old datatype names as indicated in metaschema.xsd (cf line 1252)
* Restoring deprecated datatypes per usnistgov#195
* Bit of cleanup; utility maintenance
* Removing 'INFO' level comment in XSD
* Tweakage to align XSD out (dropping comment; cleaning up namespaces)
* Patching hole in JSON datatype assignment also
* Emitting cleaner namespaces in XSD; slight refactoring of JSON Schema
* Adjusting JSON Schema type definitions to produce valid constraints over Metaschema atomic types
* Extending atomic data type acquisition to collect one level of indirect references
* Cleaning up obsolete datatype support
* Bit more cleanup; updated readme
* Adjusting handling of warnings and exception messages in pipelines
* Removing outdated artifacts from schema generation unit testing.md

Co-authored-by: Wendell Piez <wendell.piez@nist.gov>

toolchains/xslt-M4/schema-gen/metaschema-v090-datatypes.json → schema/json/metaschema-datatypes.json

@@ -44,8 +44,10 @@
       "pattern": "^.+@.+$"
     },
     "HostnameDatatype": {
-      "$ref": "#/definitions/StringDatatype",
-      "format": "idn-hostname"
+           "allOf": [
+           {"$ref": "#/definitions/StringDatatype"},
+           {"format": "idn-hostname"}
+       ]     
     },
     "IntegerDatatype": {
       "type": "integer"
@@ -68,12 +70,18 @@
       "type": "string"
     },
     "NonNegativeIntegerDatatype": {
-      "$ref": "#/definitions/IntegerDatatype",
-      "minimum": 0
+       "allOf": [
+           {"$ref": "#/definitions/IntegerDatatype"},
+           {"minimum": 0,
+            "type": "number"}
+       ]
     },
     "PositiveIntegerDatatype": {
-      "$ref": "#/definitions/IntegerDatatype",
-      "minimum": 1
+       "allOf": [
+           {"$ref": "#/definitions/IntegerDatatype"},
+           {"minimum": 1,
+            "type": "number"}
+       ]
     },
     "StringDatatype": {
       "type": "string",

schema/xml/metaschema-datatypes.xsd

@@ -0,0 +1,241 @@
+<?xml version="1.0" encoding="utf-8"?>
+<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema"
+	elementFormDefault="qualified">
+
+	<xs:simpleType name="Base64Datatype">
+		<xs:restriction base="xs:base64Binary">
+			<xs:pattern value="\S(.*\S)?">
+				<xs:annotation>
+					<xs:documentation>A trimmed string, at least one character with no
+						leading or trailing whitespace.</xs:documentation>
+				</xs:annotation>
+			</xs:pattern>
+		</xs:restriction>
+	</xs:simpleType>
+
+	<xs:simpleType name="BooleanDatatype">
+		<xs:restriction base="xs:boolean">
+			<xs:pattern value="\S(.*\S)?">
+				<xs:annotation>
+					<xs:documentation>A trimmed string, at least one character with no
+						leading or trailing whitespace.</xs:documentation>
+				</xs:annotation>
+			</xs:pattern>
+		</xs:restriction>
+	</xs:simpleType>
+
+	<xs:simpleType name="DateDatatype">
+		<xs:restriction base="xs:date">
+			<xs:pattern value="(((2000|2400|2800|(19|2[0-9](0[48]|[2468][048]|[13579][26])))-02-29)|(((19|2[0-9])[0-9]{2})-02-(0[1-9]|1[0-9]|2[0-8]))|(((19|2[0-9])[0-9]{2})-(0[13578]|10|12)-(0[1-9]|[12][0-9]|3[01]))|(((19|2[0-9])[0-9]{2})-(0[469]|11)-(0[1-9]|[12][0-9]|30)))(Z|[+-][0-9]{2}:[0-9]{2})?"/>
+		</xs:restriction>
+	</xs:simpleType>
+
+	<xs:simpleType name="DateWithTimezoneDatatype">
+		<xs:annotation>
+			<xs:documentation>The xs:date with a required timezone.</xs:documentation>
+		</xs:annotation>
+		<xs:restriction base="DateDatatype">
+			<xs:pattern value="(((2000|2400|2800|(19|2[0-9](0[48]|[2468][048]|[13579][26])))-02-29)|(((19|2[0-9])[0-9]{2})-02-(0[1-9]|1[0-9]|2[0-8]))|(((19|2[0-9])[0-9]{2})-(0[13578]|10|12)-(0[1-9]|[12][0-9]|3[01]))|(((19|2[0-9])[0-9]{2})-(0[469]|11)-(0[1-9]|[12][0-9]|30)))(Z|[+-][0-9]{2}:[0-9]{2})"/>
+		</xs:restriction>
+	</xs:simpleType>
+
+	<xs:simpleType name="DateTimeDatatype">
+		<xs:restriction base="xs:dateTime">
+			<xs:pattern value="(((2000|2400|2800|(19|2[0-9](0[48]|[2468][048]|[13579][26])))-02-29)|(((19|2[0-9])[0-9]{2})-02-(0[1-9]|1[0-9]|2[0-8]))|(((19|2[0-9])[0-9]{2})-(0[13578]|10|12)-(0[1-9]|[12][0-9]|3[01]))|(((19|2[0-9])[0-9]{2})-(0[469]|11)-(0[1-9]|[12][0-9]|30)))T((2[0-3]|[01][0-9]):([0-5][0-9]):([0-5][0-9])(\.[0-9]+)?)(Z|[+-][0-9]{2}:[0-9]{2})?"/>
+		</xs:restriction>
+	</xs:simpleType>
+
+	<xs:simpleType name="DateTimeWithTimezoneDatatype">
+		<xs:annotation>
+			<xs:documentation>The xs:dateTime with a required timezone.</xs:documentation>
+		</xs:annotation>
+		<xs:restriction base="DateTimeDatatype">
+			<xs:pattern value="(((2000|2400|2800|(19|2[0-9](0[48]|[2468][048]|[13579][26])))-02-29)|(((19|2[0-9])[0-9]{2})-02-(0[1-9]|1[0-9]|2[0-8]))|(((19|2[0-9])[0-9]{2})-(0[13578]|10|12)-(0[1-9]|[12][0-9]|3[01]))|(((19|2[0-9])[0-9]{2})-(0[469]|11)-(0[1-9]|[12][0-9]|30)))T((2[0-3]|[01][0-9]):([0-5][0-9]):([0-5][0-9])(\.[0-9]+)?)(Z|[+-][0-9]{2}:[0-9]{2})"/>
+		</xs:restriction>
+	</xs:simpleType>
+
+	<xs:simpleType name="DayTimeDurationDatatype">
+		<xs:restriction base="xs:duration">
+			<xs:pattern value="[-+]?P([-+]?[0-9]+D)?(T([-+]?[0-9]+H)?([-+]?[0-9]+M)?([-+]?[0-9]+([.,][0-9]{0,9})?S)?)?"/>
+		</xs:restriction>
+	</xs:simpleType>
+
+	<xs:simpleType name="DecimalDatatype">
+		<xs:restriction base="xs:decimal">
+			<xs:pattern value="\S(.*\S)?">
+				<xs:annotation>
+					<xs:documentation>A trimmed string, at least one character with no
+						leading or trailing whitespace.</xs:documentation>
+				</xs:annotation>
+			</xs:pattern>
+		</xs:restriction>
+	</xs:simpleType>
+
+	<xs:simpleType name="EmailAddressDatatype">
+		<xs:annotation>
+			<xs:documentation>An email address</xs:documentation>
+		</xs:annotation>
+		<xs:restriction base="StringDatatype">
+			<xs:pattern value="\S.*@.*\S">
+				<xs:annotation>
+					<xs:documentation>Need a better pattern.</xs:documentation>
+				</xs:annotation>
+			</xs:pattern>
+		</xs:restriction>
+	</xs:simpleType>
+
+	<xs:simpleType name="HostnameDatatype">
+		<xs:annotation>
+			<xs:documentation>A host name</xs:documentation>
+		</xs:annotation>
+		<xs:restriction base="StringDatatype">
+			<!-- TODO: Need a good hostname pattern -->
+		</xs:restriction>
+	</xs:simpleType>
+
+	<xs:simpleType name="IntegerDatatype">
+		<xs:restriction base="xs:integer">
+			<xs:pattern value="\S(.*\S)?">
+				<xs:annotation>
+					<xs:documentation>A trimmed string, at least one character with no
+						leading or trailing whitespace.</xs:documentation>
+				</xs:annotation>
+			</xs:pattern>
+		</xs:restriction>
+	</xs:simpleType>
+
+	<xs:simpleType name="IPV4AddressDatatype">
+		<xs:annotation>
+			<xs:documentation>The ip-v4-address type specifies an IPv4 address in
+				dot decimal notation.</xs:documentation>
+		</xs:annotation>
+		<xs:restriction base="StringDatatype">
+			<xs:pattern value="((25[0-5]|2[0-4][0-9]|1[0-9][0-9]|[1-9][0-9]|[0-9]).){3}(25[0-5]|2[0-4][0-9]|1[0-9][0-9]|[1-9][0-9]|[0-9])" />
+		</xs:restriction>
+	</xs:simpleType>
+
+	<xs:simpleType name="IPV6AddressDatatype">
+		<xs:annotation>
+			<xs:documentation>The ip-v6-address type specifies an IPv6 address
+				represented in 8 hextets separated by colons.</xs:documentation>
+			<xs:documentation>This is based on the pattern provided here:
+				https://stackoverflow.com/questions/53497/regular-expression-that-matches-valid-ipv6-addresses
+				with some customizations.</xs:documentation>
+		</xs:annotation>
+		<xs:restriction base="xs:string">
+			<xs:whiteSpace value="collapse" />
+			<xs:pattern
+				value="(([0-9a-fA-F]{1,4}:){7,7}[0-9a-fA-F]{1,4}|([0-9a-fA-F]{1,4}:){1,7}:|([0-9a-fA-F]{1,4}:){1,6}:[0-9a-fA-F]{1,4}|([0-9a-fA-F]{1,4}:){1,5}(:[0-9a-fA-F]{1,4}){1,2}|([0-9a-fA-F]{1,4}:){1,4}(:[0-9a-fA-F]{1,4}){1,3}|([0-9a-fA-F]{1,4}:){1,3}(:[0-9a-fA-F]{1,4}){1,4}|([0-9a-fA-F]{1,4}:){1,2}(:[0-9a-fA-F]{1,4}){1,5}|[0-9a-fA-F]{1,4}:((:[0-9a-fA-F]{1,4}){1,6})|:((:[0-9a-fA-F]{1,4}){1,7}|:)|[fF][eE]80:(:[0-9a-fA-F]{0,4}){0,4}%[0-9a-zA-Z]{1,}|::([fF]{4}(:0{1,4}){0,1}:){0,1}((25[0-5]|2[0-4][0-9]|1[0-9][0-9]|[1-9][0-9]|[0-9]).){3,3}(25[0-5]|2[0-4][0-9]|1[0-9][0-9]|[1-9][0-9]|[0-9])|([0-9a-fA-F]{1,4}:){1,4}:((25[0-5]|2[0-4][0-9]|1[0-9][0-9]|[1-9][0-9]|[0-9]).){3,3}(25[0-5]|2[0-4][0-9]|1[0-9][0-9]|[1-9][0-9]|[0-9]))" />
+		</xs:restriction>
+	</xs:simpleType>
+
+	<xs:simpleType name="NonNegativeIntegerDatatype">
+		<xs:restriction base="xs:nonNegativeInteger">
+			<xs:pattern value="\S(.*\S)?">
+				<xs:annotation>
+					<xs:documentation>A trimmed string, at least one character with no
+						leading or trailing whitespace.</xs:documentation>
+				</xs:annotation>
+			</xs:pattern>
+		</xs:restriction>
+	</xs:simpleType>
+
+	<xs:simpleType name="PositiveIntegerDatatype">
+		<xs:restriction base="xs:positiveInteger">
+			<xs:pattern value="\S(.*\S)?">
+				<xs:annotation>
+					<xs:documentation>A trimmed string, at least one character with no
+						leading or trailing whitespace.</xs:documentation>
+				</xs:annotation>
+			</xs:pattern>
+		</xs:restriction>
+	</xs:simpleType>
+
+	<xs:simpleType name="StringDatatype">
+		<xs:annotation>
+			<xs:documentation>A string, but not empty and not whitespace-only
+				(whitespace is U+9, U+10, U+32 or [ \n\t]+ )</xs:documentation>
+		</xs:annotation>
+		<xs:restriction base="xs:string">
+			<xs:annotation>
+				<xs:documentation>The OSCAL 'string' datatype restricts the XSD type by prohibiting leading 
+					and trailing whitespace, and something (not only whitespace) is required.</xs:documentation>
+			</xs:annotation>
+			<xs:whiteSpace value="preserve" />
+			<xs:pattern value="\S(.*\S)?">
+				<xs:annotation>
+					<xs:documentation>A trimmed string, at least one character with no
+						leading or trailing whitespace.</xs:documentation>
+				</xs:annotation>
+			</xs:pattern>
+		</xs:restriction>
+	</xs:simpleType>
+
+	<xs:simpleType name="TokenDatatype">
+		<xs:annotation>
+			<!--<xs:documentation>Matching XSD NCName, except whitespace is not collapsed.</xs:documentation> -->
+			<xs:documentation>A string token following the rules of XML "no
+				colon" names, with no whitespace. (XML names are single alphabetic
+				characters followed by alphanumeric characters, periods, underscores or dashes.)
+			</xs:documentation>
+		</xs:annotation>
+		<xs:restriction base="StringDatatype">
+<!-- 			value="(\p{L}|_)(\p{L}|\p{N}|[.\-_]|\xb7|[#x0300-#x036F#x203F-#x2040])*">-->
+			<xs:pattern
+				value="(\p{L}|_)(\p{L}|\p{N}|[.\-_])*">
+				<xs:annotation>
+					<!--<xs:documentation>An XML initial character (but not colon), followed 
+						by any XML name character (but not colon).</xs:documentation> -->
+					<xs:documentation>A single token may not contain whitespace.
+					</xs:documentation>
+				</xs:annotation>
+			</xs:pattern>
+		</xs:restriction>
+	</xs:simpleType>
+
+	<xs:simpleType name="URIDatatype">
+		<xs:annotation>
+			<xs:documentation>A URI</xs:documentation>
+		</xs:annotation>
+		<xs:restriction base="xs:anyURI">
+			<xs:pattern value="[a-zA-Z][a-zA-Z0-9+\-.]+:.*\S">
+				<xs:annotation>
+					<xs:documentation>Requires a scheme with colon per RFC 3986.</xs:documentation>
+				</xs:annotation>
+			</xs:pattern>
+		</xs:restriction>
+	</xs:simpleType>
+
+	<xs:simpleType name="URIReferenceDatatype">
+		<xs:annotation>
+			<xs:documentation>A URI reference, such as a relative URL
+			</xs:documentation>
+		</xs:annotation>
+		<xs:restriction base="xs:anyURI">
+			<xs:pattern value="\S(.*\S)?">
+				<xs:annotation>
+					<xs:documentation>A trimmed URI, at least one character with no
+						leading or trailing whitespace.</xs:documentation>
+				</xs:annotation>
+			</xs:pattern>
+		</xs:restriction>
+	</xs:simpleType>
+
+	<xs:simpleType name="UUIDDatatype">
+		<xs:annotation>
+			<xs:documentation>A type 4 ('random' or 'pseudorandom') or type 5 UUID per RFC
+				4122.</xs:documentation>
+		</xs:annotation>
+		<xs:restriction base="StringDatatype">
+			<xs:pattern
+				value="[0-9A-Fa-f]{8}-[0-9A-Fa-f]{4}-[45][0-9A-Fa-f]{3}-[89ABab][0-9A-Fa-f]{3}-[0-9A-Fa-f]{12}">
+				<xs:annotation>
+					<xs:documentation>A sequence of 8-4-4-4-12 hex digits, with extra
+						constraints in the 13th and 17-18th places for version 4 and 5
+					</xs:documentation>
+				</xs:annotation>
+			</xs:pattern>
+		</xs:restriction>
+	</xs:simpleType>
+</xs:schema>
+

schema/xml/metaschema-markup-line.xsd

@@ -0,0 +1,11 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema" elementFormDefault="qualified">
+
+  <xs:include schemaLocation="metaschema-prose-base.xsd"/>
+
+  <xs:complexType name="MarkupLineDatatype" mixed="true">
+    <xs:choice minOccurs="0" maxOccurs="unbounded">
+      <xs:group ref="inlineMarkupGroup"/>
+    </xs:choice>
+  </xs:complexType>
+</xs:schema>

toolchains/xslt-M4/schema-gen/oscal-prose-module.xsd → schema/xml/metaschema-markup-multiline.xsd

@@ -1,14 +1,10 @@
 <?xml version="1.0" encoding="UTF-8"?>
 <xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema" elementFormDefault="qualified">
+
+  <xs:include schemaLocation="metaschema-prose-base.xsd"/>
 
-  <xs:complexType name="markupLineType" mixed="true">
-    <xs:choice minOccurs="0" maxOccurs="unbounded">
-      <xs:group ref="inlineMarkupGroup"/>
-    </xs:choice>
-  </xs:complexType>
-
-  <xs:complexType name="markupMultilineType">
-    <xs:choice minOccurs="0" maxOccurs="unbounded">
+  <xs:complexType name="MarkupMultilineDatatype">
+    <xs:choice minOccurs="1" maxOccurs="unbounded">
       <xs:group ref="blockElementGroup"/>
     </xs:choice>
   </xs:complexType>
@@ -103,77 +99,4 @@
       <xs:enumeration value="right"/>
     </xs:restriction>
   </xs:simpleType>
-
-  <xs:group name="inlineMarkupGroup">
-    <xs:choice>
-      <xs:element name="a" type="anchorType"/>
-      <xs:element name="insert" type="insertType"/>
-      <xs:group ref="phraseMarkupGroup"/>
-      <xs:group ref="imageMarkupGroup"/>
-    </xs:choice>
-  </xs:group>
-  <xs:group name="imageMarkupGroup">
-    <xs:choice>
-      <xs:element name="img" type="imageType"/>
-    </xs:choice>
-  </xs:group>
-  <xs:group name="phraseMarkupGroup">
-    <xs:choice>
-      <xs:element name="code" type="inlineMarkupType" minOccurs="1"/>
-      <xs:element name="em" type="inlineMarkupType" minOccurs="1"/>
-      <xs:element name="i" type="inlineMarkupType" minOccurs="1"/>
-      <xs:element name="b" type="inlineMarkupType" minOccurs="1"/>
-      <xs:element name="strong" type="inlineMarkupType" minOccurs="1"/>
-      <xs:element name="sub" type="inlineMarkupType" minOccurs="1"/>
-      <xs:element name="sup" type="inlineMarkupType" minOccurs="1"/>
-      <xs:element name="q" type="inlineMarkupType" minOccurs="1"/>
-    </xs:choice>
-  </xs:group>
-  <xs:complexType name="inlineMarkupType" mixed="true">
-    <xs:choice minOccurs="0" maxOccurs="unbounded">
-      <xs:group ref="inlineMarkupGroup"/>
-    </xs:choice>
-  </xs:complexType>
-  <xs:complexType name="imageType">
-    <xs:attribute name="alt" type="xs:string"/>
-    <xs:attribute name="src" use="required" type="xs:anyURI"/>
-  </xs:complexType>
-
-  <xs:complexType name="anchorType" mixed="true">
-    <xs:choice minOccurs="0" maxOccurs="unbounded">
-      <xs:group ref="phraseMarkupGroup"/>
-      <xs:group ref="imageMarkupGroup"/>
-    </xs:choice>
-    <xs:attribute name="href" type="xs:anyURI"/>
-  </xs:complexType>
-
-  <!--
-    A value to be assigned by responder via parameter
-    (a parameter must be available)
-  -->
-  <xs:complexType name="insertType">
-    <xs:annotation>
-      <xs:documentation>An insert can be used to identify a placeholder for dynamically inserting text related to a specific object, which is referenced by the object's identifier using an <code>id-ref</code>. This insert mechanism allows the selection of which text value from the object to dynamically include based on the application's display requirements.</xs:documentation>
-    </xs:annotation>
-    <xs:attribute name="type" use="required" type="xs:NCName">
-      <xs:annotation>
-        <xs:documentation>The type of object to include from (e.g., parameter, control, component, role, etc.)</xs:documentation>
-      </xs:annotation>
-    </xs:attribute>
-    <xs:attribute name="id-ref" use="required" type="xs:NCName">
-      <xs:annotation>
-        <xs:documentation>The identity of the object to insert a value for. The identity will be selected from the index of objects of the specified <code>type</code>. The specific value to include is based on the application's display requirements, which will likely use a specific data element associated with the <code>type</code> (e.g., title, identifier, value, etc.) that is appropriate for the application.</xs:documentation>
-      </xs:annotation>
-    </xs:attribute>
-  </xs:complexType>
-
-  <!--
-    Within a selection, a choice - tbd
-    unlike 'value', choice may have inline stuff as well as 'assign' elements
-    choice = element choice { (assign | mix)* }
-  -->
-  <!--
-    IDs are required everywhere for now
-    idAttr      = attribute id { xsd:ID }
-  -->
 </xs:schema>