Example FedRAMP SSP Statement IDs for Implemented Requirements of Control Implementations Incorrect #69
Labels
bug
Something isn't working
Comments
ohsh6o
referenced
this issue
in 18F/fedramp-automation
Dec 14, 2020
ohsh6o
referenced
this issue
in 18F/fedramp-automation
Dec 15, 2020
ohsh6o
added a commit
to 18F/fedramp-automation
that referenced
this issue
Jan 1, 2021
* Update statement infix from _stmt. to _smt. again. Forgot about GSA#69 and GSA#70 so we need to take care of this again since this updated example SSP was provided out of band. * Do not require FedRAMP namespace for responses. What were previously know as response-points that FedRAMP provided as an extension are now part of upstream OSCAL, thereby breaking some of tests that were filtering on a `o:prop` with a `ns` attribute it did not have.
ohsh6o
added a commit
to 18F/fedramp-automation
that referenced
this issue
Jan 14, 2021
* Update statement infix from _stmt. to _smt. again. Forgot about GSA#69 and GSA#70 so we need to take care of this again since this updated example SSP was provided out of band. * Do not require FedRAMP namespace for responses. What were previously know as response-points that FedRAMP provided as an extension are now part of upstream OSCAL, thereby breaking some of tests that were filtering on a `o:prop` with a `ns` attribute it did not have.
ohsh6o
added a commit
to 18F/fedramp-automation
that referenced
this issue
Jan 15, 2021
* Added statement response-points. Changed objective conformity to response-points * cleanup response-point issues * Publishing auto-converted artifacts * Add defined identifiers values for only FedRAMP role types. Per discussion over pending work in #32, we want to further enhance the XML content defined in the FedRAMP Registry as saved in the Excel XSLX file to include defined identifiers for roles particular to FedRAMP not the current milestone release of upstream OSCAL. * Fix statement IDs for example SSP. Fixes #69. * Revert "Fix statement IDs for example SSP." * Publishing auto-converted artifacts * Revert "Revert "Fix statement IDs for example SSP."" * Fixed ajv-cli version to address build error caused by ajv-cli@4.0.1 * changing OSCAL repo commit * restoring original OSCAL commit hash * Publishing auto-converted artifacts * Add initial schematron. This is from experimentations and our pairing sessions. * Actually add Schematron file, not just placeholder README. * initial structure and instructions from https://www.balisage.net/Proceedings/vol19/html/Lizzi01/BalisageVol19-Lizzi01.html additional specs required * add id's an add missing $ to variable * add xpec and additional sample file * Publishing auto-converted artifacts * Report on impact levels and assert if not valid. * Minor fix in security-sensitivity-level versus impact-level. FedRAMP example SSPs define the former over the latter. The impact- level attribute appears to correlate with a risk, not the overall system itself. Further investigation may be needed. From OSCAL documentation: > Often, organizations require the security sensitivity level to correspond with the highest confidentiality, integrity, or availability level identified by security-impact-level. https://pages.nist.gov/OSCAL/documentation/schema/implementation-layer/ssp/xml-schema/#oscal-ssp-xml_security-sensitivity-level * Reorganize let bindings. * Conditional pluralization. * Remove debug stmt for security-sensitivity-level values. * Clean up spacing a little. * Debug: get required controls for low. * Clearer debugging. * Preliminary basic checking. This looks for any control-id that is required that is not within the list of implemented controls. For now, hard-coded to SSPs with a Low Impact categorizaiton for FedRAMP. * Move Schematron to correct ./src folder. * Remove old prototype fedramp_ssp_statistics.sch file. * update to renamed schematron file * add test all harness as example * add instructions for new test_all.xspec * Add back schema. This edit was causing the reports to come back empty. Adding it back resolved the issue. * Update relative paths for relocation. * Add schematron as a submodule. * Update .gitignore to ignore development files. * Add placeholder .gitempty files for reports folders. There will be a subdirectory for XSpec outputs and * Relocate Schematron and XSpec to lib sub-dir. * Relocate reports->report. * Update documentation. * Update .gitignore * add shell for compiling and running validation you must pass in a file name you want validated. by default it will compile and validate the input with all src/*.sch files * add SAXON override and instructions * Fix spacing. * PR review tweaks * add ability to generate report in HTML along with SVRL default output * Spacing and env vars for Saxon option. * Command spacing made multi-line for readability. * Inject SAXON_OPTS into commands. * One more minor space. * Fail fast on any errors from piped commands. * Delete previous reports. Piggy-backing off of #7, and #8. We are cleaning previous XSL * first attempt at xpec test runner * switch order of SAXON_CP export * add debugging * fix order * add additional export scope doesn't cross command runners? * add Test Report * accept AJ suggestion for branch wildcarding Co-authored-by: Alexander Stein <61464190+ohsh6o@users.noreply.github.com> * update label and comments for correctness * remove quotes * make test report path relative * be explicit...just to verify * use junit action step instead of surefire * remove surefire report section * rename file and remove extra reporting parameters * make action comments more correct & change `on` lifecycle * closes #3 add publish test result summary (#15) * add publish test results * better configure plugin for test statistics * run publish stats always * remove extra configs * add badge for schematron unit tests * initiate a model for on demand running validation (#27) * add github.event.inputs. prefix for input values * define inputs in the correct place * workout saxon jar invocation * add better qualified reference to project * incorrect variable reference * fix typo * restructure saxon call * explicitly mkdir /tmp/target * make java action work * Update xspecRunner.yml * update shell and readme based on feedback (#29) * flail * try figuring out class invocation * specify xsl as input * reexport saxon in following job step * contextualize file reference * further qualified * missing slash * add debugging * missed a subdir * remove * define report name * superfolus quote * superfulous monkey bun * double slash issue * proper qualify target * barf * less incorrect parts * avoid DS_Store * fix pathing for html report and java command * fix path in readme * add cache * add key with hash * debug * missing multi line continuance * ignore jars in the lib directory * be more defensive of how to download saxon jar * make instructions more clear with dependencies stated * stylistic consistency changes, hard fail if no saxon specified or ability to download, clean up some echos * remove testing names * add command line argument processing rather than positional parameters and update readme * clarify options * -h for usage instructions * intentionally allow globbing * add safety checks that saxon jar is a file and it has the Trasform we need * New XSpec test format. * Update implemented requirement wording. * Add IDs to assert and report tests. * Add more tests around counts. * Test organization improvements. * Add local name space for XSL lookup tables. * WIP work on relative path offset for FedRAMP data values. * Saxon `allow-foreign` arg needed in XSpec. Our harness makes use of it, but natively XSpec does not. * Update to dynamic load of profiles. Moving forward, we will not hard-code the profile to load based on the categorization. For now, we: - Build a map (`profile-map`) in XSL of sensitivity levels to UUID and file paths. - Take the sensitivity-level computed from the example SSP we are using by way of XSL `key` on the map with an intermediate lookup step. - Refer to the profile in use as `$selected-profile` so we do not have to hard-code it moving forward. * Get implementation statuses from FedRAMP values. * Parameterize context path of global vars & XSpec. As it stands, XSpec will throw errors for global `xsl:variable`, `sch:let` and other variable bindings that make use of XPath and a path without a context. So inside of Schematron patterns, a pattern is set, and when running pure Schematron, it is also set for global variables, but XSpec intentionally does not honor that assumption and will exhibit these errors. * Set '/' for context correctly. This had been discussed on Friday's dev consultation with Wendell, but as of yet not fixed. * Woops, missed the other '/' root context. * Re-order XSL params. * Modularize FedRAMP Registry values and href path. * Remove old commented sensitivity level. * Make SSP unit test file name more specific. * Make naming more consistent (path->href). * Add assertion for FedRAMP values. * Remove UUIDs in profile-map for now. * Switch to full resolved profiles. * Slim down and inline global-context-item. * Add no-security-sensitivity-level test. * Clarify rule label. * Add more clarifying tests on implementation statuses. * Remove unneeded bracket. * Collect test outputs. * Push up test that doesn't work for advice. * Fix spacing. * Fix up global-contex-item per doc review. * Fix misguided test output from debugging. * Update control XPath select per @brianrufgsa's feedback. * Surpress problematic test for now for final PR review. * Simplify GH action to use shell script directly. also, output downloadable validation reports (#30) * flail * try figuring out class invocation * specify xsl as input * reexport saxon in following job step * contextualize file reference * further qualified * missing slash * add debugging * missed a subdir * remove * define report name * superfolus quote * superfulous monkey bun * double slash issue * proper qualify target * barf * less incorrect parts * avoid DS_Store * fix pathing for html report and java command * fix path in readme * add cache * add key with hash * debug * missing multi line continuance * ignore jars in the lib directory * be more defensive of how to download saxon jar * make instructions more clear with dependencies stated * stylistic consistency changes, hard fail if no saxon specified or ability to download, clean up some echos * remove testing names * add command line argument processing rather than positional parameters and update readme * clarify options * -h for usage instructions * intentionally allow globbing * add safety checks that saxon jar is a file and it has the Trasform we need * add a -l command argument for loction of library files downloaded. quiet the output of the script a bit * if lib is specified, but compiled xsl's to that directory * convert to using parameterized shell to avoid duplication * better qualify path to sh * remove . qualifier * add workspace predicate to schematron argument (multi-line formatting of args) * line continuance is important * better handling of target (us lib dir) * extra continuance * add target directory option and associated May still have issue with relative path lib for - html xsl L #176 - submodule relative path L #151 * replace library and target over rides with base option due to intertwined nature of output and upstream relative dependencies * further qualified path * update docs * turn off debugging * add upload action * better qualify report dir * get rid of cache step, tackle later if saxon or submodue downloads cause significant slowness * Bring back Schematron preprocessing. For #26, pulling into master branch changes after a PR review was complete clobbered these changes when my git client rebased. My intermediary errors like this, while debugging further, made me take notice that I had not stashed or properly saved these changes in my branch. using saxon version 10.2 SAXON_CP env variable used is /Users/astein/.m2/repository/net/sf/saxon/Saxon-HE/10.2/Saxon-HE-10.2.jar Saxon JAR at classpath /Users/astein/.m2/repository/net/sf/saxon/Saxon-HE/10.2/Saxon-HE-10.2.jar is valid Schema implementation error: This schema has abstract patterns, yet they are supposed to be preprocessed out already Error: unrecognized element in ISO Schematron namespace: check spelling and capitalizationsch:param Error: unrecognized element in ISO Schematron namespace: check spelling and capitalizationsch:param These changes were tested on the tasks/issue-2-dynamic-requirement-counts branch. * Convert Implemented Requirements Counts to Dynamic Value Sets (#26) * Test out XSL functions for value-set checks. * Remove obsolete comment. * Use fatal error role, no just error. Missing values is serious enough processing out to stop. Per review of this article. http://schematron.com/2018/12/standard-severity-levels-with-schematron-role/ * Return nothing if valid, enum for recommendations if invalid. * Clean up validation function some more. * Put back variable and not test condition. * Clean up validation function. It can be simplified to one xsl:choose, xsl:when closure. * Update parameters to match metaschema pipeline. * Clear intermediate Schematron files too, still no dice. * Make the script actually match docs, thanks @mike-stern! * Revert change to ignore namespacing. * Add back important script variable bindings after rebase. * Add global context fn for sensitivity-level. * Clean up function spacing. * Start new approach with global context from XSL functions. * More function cleanup. * Add empty check util function. * Add formatter functions (even if I cannot get them working). * Update function-based approach for rules. * Test out supressed pending test. * Testing reordering of rules. * Remove relocated rules. * Time for some more cleanup. - Remove some global declarations. - Use function for implemented-requirement statuses. - Format validation corrections inline, chaining formatter fns not working. * Smaller function name that is more coherent. Since we still the results as corrections, might as well name the function correct(). * Update to dynamic grouping of items for analysis. As discussed with @wendellpiez, aggregating analysis, primarily counts and simply statistics over more than the context of a given set of the elements you traditionally use for an assert or report. So how do we do statistics over the collections at the XML document level, without the required context as Schematron operates on? In the interim we will make use of XSL functions to structure XML elements with an `analysis/reports` path, and soon a function that will structure that data in a report message (`lv:collect` is the former, the latter TBD). * Minor type fixes. * Rename count var more appropriately. * Reorganize rules a little. * No, actually do it this time. And some more reporting info. * Include in the reports node the overall count. * Better name XSL functions. * More refactoring of lv:analyze function. * More lv:analyze fn and template touch-ups. * Add lv:report function to call template. The template is still not working, but one step at a time. * Finally, a valid report format response, on to tests. * More touch-up, less whitespace escapes for now. * Remove all-requirements-report, no longer needed. * Hoist up required controls declaration. * Push up updated changes for feedback. We have removed the global items for now and introduced some duplicative function calls. * Clean up XSpec context and see if we can push forward. * Remove global context comments. * Move debugging elements. * Fix bad assert logic. * Clean up XSpec spacing. * Add return type for if-empty-default. * Improve null checks for empty value-sets. If we do not do this, a lot of tests will break it seems. * Rearrange and cleanup tests some more. * Some more function cleanups. * Sigh, more touchups. * More silly formatting. * Fix function for proper empty testing. * Remove tests with params inside x:context. I keep forgetting that will not work. https://github.com/xspec/xspec/wiki/Writing-Scenarios#global-parameters Will have to test the registry XSL and other functions separately without these parameters. * More touch-ups. * Fix up the tests. * Test fix-ups. * Clean up tests and start passing again. * Add back important test. * Remove unused var for now. * Finally figured out how to test registry context function. * Improve default-if-empty to support many types. * And update supporting tests. * Fix up tests. * Fix up if-empty-default tests and other cleanup. * Add empty sequence test. * Add tests for lv:correct. * add reports section to the html conversion * remove cross check Co-authored-by: Mike Stern <michael.stern@gsa.gov> * Updates from GSA Upstream (#33) * Added statement response-points. Changed objective conformity to response-points * cleanup response-point issues * Publishing auto-converted artifacts Co-authored-by: Brian Ruf <brian.ruf@gsa.gov> Co-authored-by: OSCAL GitHub Actions Bot <oscal@nist.gov> * make prerequisite instructions clearer * refine formatting * Check for Additional Extraneous Implemented Requirements for Controls (#34) * Fix analysis report looping bug. Since we were looping through items that are > 1, we were starting to loop through the 2nd item of a list of reports (//reports/report) in a analysis template in a variable. Noticed this when looking at result of a lv:report() on implemented-requirements. * Woops, misread docs syntax. * Fix-up actual assertion. * Add tests. * Fix spacing issue. * And fix misleading comment for negative test. * Add Validations Specifically for Core Controls (#35) * Specify variable for any missing. We will be adding an assert for core controls, so make existing variable more explicit. * Add registry namespace check. Following feedback from @brianrufgsa, make sure to test for namespace of FedRAMP extensions. Here, we do it by introspecting the FedRAMP values registry for defined namespace. * First crack at core control check. * Get rid of unnecessary variable. Instead, just further filter on required controls. * This seems to do the trick. * Clarify assert for all missing requirements and tweak test. * Implementation->implemented for misnamed assert IDs. * Add role levels to control errors. For now, all required controls will fail with a warning. Incomplete core controls will fail with error. * Add missing error role for invalid impl status. * Add test to catch unimplemented core rule assertion. * Add negative test. * Checks for Response Statements for Implemented Requires for Controls (#36) * [WIP] First draft of response checks. * Description, remarks, and UUID checks. The latter is checking in system-implementation, per Guide to OSCAL * Per conversation with @brianrufgsa and @wendellpiez on UUID matching. Declare it a warning not an error. https://gitter.im/usnistgov-OSCAL/FedRAMP-10x-Schematron?at=5fd238391f36292125e22c30 * Remarks are always optional in statements. This is different from annotations, re discussion with @brianrufgsa and @wendellpiez. https://gitter.im/usnistgov-OSCAL/FedRAMP-10x-Schematron?at=5fd237cd0697c1210db283d7 * Only check component remarks if defined. We are using Schematron, so we might as well use the contexts to our advantage. * Better structure for descriptions and remarks. * Fix spacing, silly tabs! * Remove dup test. * Formatting touch-ups. * Have to fix for right XPath. * Remove debug checks. * Delete temp file from OSCAL registry when loaded in Word. * Tweak assertion id for statement description. * Anothher assertoin id tweak. * Add tests for statement-level descr and remarks. * More tests. * Woops, relocate test. * Add another test. * And remove role for not-assert test. * Add more component description tests. * Clarify scenario label for descriptions. * Singular. * Test re-org. * Woops, assertion id is @id, not @name. * Add missing tests. * Missing assert and final wrap-up tests. * Implement response-point assertions, tests, and example SSP. (#38) * fix format of the repo structure * Update SSP to Sample Using OSCAL 1.0.0-rc1 Syntax (#41) * Remove partial file, we are not even using it anymore. * Add updated RC1 SSP from @brianrufgsa. * Update to remote resolved profile catalog. (#42) Per conversation with @brianrufgsa, the current profiles and resolved catalog profiles from github.com/usnistgov/oscal-content that matches with the push from the OSCAL CI/CD system for 1.0.0-rc1. Those in the fedramp-automation repo will be updated at a later time. This is to further testing of the updated SSP pulled in with #41. * Complete Improvements Needed for Changes in OSCAL 1.0.0-rc1 (#43) * Update statement infix from _stmt. to _smt. again. Forgot about GSA#69 and GSA#70 so we need to take care of this again since this updated example SSP was provided out of band. * Do not require FedRAMP namespace for responses. What were previously know as response-points that FedRAMP provided as an extension are now part of upstream OSCAL, thereby breaking some of tests that were filtering on a `o:prop` with a `ns` attribute it did not have. * Make README more high-level. For #40, we will move a high-level mapping of rules to the README and technical information will go to the CONTRIBUTING.md file. * Try checklist format. * Woops no space in checkbox brackets. * Need a list hyphen for tasklists. * Add some rule mappings. * Add 'positive' role to categorize informational checks. Per conversation with @brianrufgsa earlier this week, as we continue to follow the dichotomy of informational validation data in sch:report and encoding warnings and errors in sch:assert checks, it is not completely transparent that the reports are information that is not an error, but a useful reporting item. Brian suggested he would like this to be marked with something like positive, so we shall start doing that and document it. * Add Section 1.a check too for now. * Update README with better info for non-technical users. * Tasks/issue 32b (#49) * adding a spec that is broken in the spirit of: - make it break - make it work - make it pretty Formatting Report... passed: 21 / pending: 0 / failed: 4 / total: 25 * surpress some of the xpspec output * fix the test * make scenario for party be invalid party-uuid instead of role-id * fix id's to be more specific * make scenario for rol be invalid role-id instead of party-uuid * add rule for parties, roles and responsible-party association * restructure test location under master SSP scenario per feedback * Update validation coverage docs after #49. * Fix link for rule mapping. * Update org-id mapping to docs. * Revert OSCAL submodule to 5581a8e to fix build errors. Match GSA/fedramp-automation where they are at per [the Gitter convo](https://gitter.im/usnistgov-OSCAL/FedRAMP-10x-Schematron?at=6000db373d722a42e782a5ff). Co-authored-by: Brian Ruf <brian.ruf@gsa.gov> Co-authored-by: OSCAL GitHub Actions Bot <oscal@nist.gov> Co-authored-by: David Waltermire <david.waltermire@nist.gov> Co-authored-by: Mike Stern <michael.stern@gsa.gov> Co-authored-by: Mike Stern <mike-stern@users.noreply.github.com>
ohsh6o
referenced
this issue
in 18F/fedramp-automation
May 4, 2021
* Added statement response-points. Changed objective conformity to response-points * cleanup response-point issues * Publishing auto-converted artifacts * Add defined identifiers values for only FedRAMP role types. Per discussion over pending work in #32, we want to further enhance the XML content defined in the FedRAMP Registry as saved in the Excel XSLX file to include defined identifiers for roles particular to FedRAMP not the current milestone release of upstream OSCAL. * Fix statement IDs for example SSP. Fixes #69. * Revert "Fix statement IDs for example SSP." * Publishing auto-converted artifacts * Revert "Revert "Fix statement IDs for example SSP."" * Fixed ajv-cli version to address build error caused by ajv-cli@4.0.1 * changing OSCAL repo commit * restoring original OSCAL commit hash * Publishing auto-converted artifacts * Updated Rev 4 baslines to RC1 syntax * Updated titles to further clarify these are FedRAMP Rev 4 baselines * Adjusted paths for profiles and resolved catalogs. * updated catalog reference in profiles to latest OSCAL rc1 800-53 rev4 catalog * Publishing auto-converted artifacts * resource tweaks * Publishing auto-converted artifacts * SSP to RC-1 and Content Updates * updated email address examples * small fixes to SSP template * Cleaned up erroneous duplicate UUIDs) * converted subnets from inventory-items to components * converted subnets from inventory-items to components * Publishing auto-converted artifacts * WIP - updated FedRAMP Extensions * updated Tailored LI-SaaS baseline to be complete * updated UUID and last-modified * Publishing auto-converted artifacts * updating OSCAL build to incorporate latest JSON Schema and YAML content fixes. * Publishing auto-converted artifacts * SAP and POAM templates updated. SAR template and all three guides still WIP * updated SAP, SAR, and POAM Templates to RC2 * SSP, SAP, SAR, and POAM Templates- Updated metadata * SSP, SAP, SAR, and POAM Templates- Updated metadata * added impacted control references * Updated guides and templates to align with OSCAL RC-2 * updated ci/cd config to process all templates * Publishing RC2 XML templates and baselines. Partial posting of JSON and YAML. * updating to latest OSCAL toolchain * Publishing auto-converted artifacts * updating content to OSCAL 1.0.0 RC2 models; updated base catalog for profiles * Publishing auto-converted artifacts * Fix XSpec test run and xspec version in code. * Cherry pick and add back fedramp_values.xml Co-authored-by: Brian Ruf <brian.ruf@gsa.gov> Co-authored-by: OSCAL GitHub Actions Bot <oscal@nist.gov> Co-authored-by: David Waltermire <david.waltermire@nist.gov>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Describe the bug
There is a slight mismatch between the example SSP's statement IDs (
/system-security-plan/control-implementation/implemented-requirement/statement/@idby XPath) and how they ought to be assigned and match the corresponding IDs in the resolved catalog profiles. Per conversation here, this appears to be a minor issue in the example SSP, and the upstream OSCAL Metaschema models & schema, either M3 or M4, and example Rev4 and Rev5 content in oscal-content suggest this is the case.This is a minor cosmetic fix but is useful for testing SSP validations, so I will surface this hear upstream as well.
Who is the bug affecting?
10x team that is writing validations against the examples SSPs as canonical test data.
What is affected by this bug?
See 18F#22 for more details.
This was detected analyzing updates with/for #44.
When does this occur?
Validations attempting to match lettered response points do not match because of the minor disparity in
@id.How do we replicate the issue?
Compare a resolved catalog profile and
grepor search for differences between profilestmt.infix and appropriatesmt.infix.Expected behavior (i.e. solution)
That
/system-security-plan/control-implementation/implemented-requirement/statement/@idmatch profile lettered response point FedRAMP extension declarations (/catalog/group/control/part/part[prop[@name='response-point']]/@id) for proper validation.Other Comments
{Add any other context about the problem here.}
The text was updated successfully, but these errors were encountered: