Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Feature internal 56 me tdataio validate payload #345

Merged
merged 58 commits into from
Oct 23, 2024
Merged

Feature internal 56 me tdataio validate payload #345

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
58 commits
Select commit Hold shift + click to select a range
6aa7fed
Internal issue #56 XML schema for validating to prevent DoS via large…
bikegeek Sep 22, 2024
4364b43
schema for validating the payload (recursive, excessively large)
bikegeek Sep 22, 2024
c620c17
Merge branch 'develop' of https://github.com/dtcenter/METdataio into …
bikegeek Sep 22, 2024
1acd496
Delete METdbLoad/ush/specification_schema.xsd
bikegeek Sep 22, 2024
63cf0aa
Merge branch 'feature_internal_56_METdataio_validate_payload' of http…
bikegeek Sep 23, 2024
a892fc4
internal issue #56 validating payload using XML schema
bikegeek Sep 23, 2024
eb41af1
internal issue #56 invalid XML spec files used to test XML validation
bikegeek Sep 23, 2024
5838ba9
internal issue #56 add the location of the XML schema file to be used…
bikegeek Sep 23, 2024
40e53b4
internal issue #56 tests added to verify validation code is providing…
bikegeek Sep 23, 2024
5771ded
internal issue#56 added new fixture used in testing XML validation code
bikegeek Sep 23, 2024
88b9367
fix import for read_load_xml module
bikegeek Sep 23, 2024
41de8b9
removed extraneous ',' in import
bikegeek Sep 23, 2024
addb8f3
Working version but still needs to check for recursive payloads for s…
bikegeek Sep 25, 2024
23986aa
Valid XML that is used for real-world data and is valid XML
bikegeek Sep 25, 2024
e419dc7
Change the name of the XML schema file
bikegeek Sep 25, 2024
2dbb111
Use the full_example.xml file instead of the test_load_specification.…
bikegeek Sep 25, 2024
422daae
Delete METdbLoad/ush/load_specification.xsd
bikegeek Sep 25, 2024
625b5ce
Test for recursive payload in load_val fields
bikegeek Sep 26, 2024
171d1c0
Added test for recursion under the load_val complex type
bikegeek Sep 26, 2024
5a85a9a
Change values to prevent recursive payloads and remove defunct regex
bikegeek Sep 26, 2024
d97f1a1
Merge branch 'feature_internal_56_METdataio_validate_payload' of http…
bikegeek Sep 26, 2024
1687ed8
Remove unused imports, add test for recursion under the load_val fields
bikegeek Sep 26, 2024
e0ab401
Add some extra elements
bikegeek Sep 26, 2024
b1ad924
Add more recursive elements to trigger ValueError
bikegeek Sep 26, 2024
665b37d
skip testing the recursion in load_val
bikegeek Sep 26, 2024
c0c2851
Remove limit to number of load_val elements
bikegeek Sep 26, 2024
d85c128
Fixed incorrect skip syntax
bikegeek Sep 26, 2024
9fec829
Reinstate the maxOccurs and minOccurs for the field
bikegeek Sep 26, 2024
71c559c
Update temporary XML spec file to match load_specification_schema.xsd
bikegeek Sep 28, 2024
bfa1d77
Config file for testing recursive payload in the fields element
bikegeek Sep 28, 2024
1186e5b
Work-in-progress. Recursive payloads checked for some elements but n…
bikegeek Sep 28, 2024
7368a9d
Added test for recursive payload for fields (in addition to test for …
bikegeek Sep 28, 2024
3b0366a
Update tests and test config files
bikegeek Oct 1, 2024
2a329b0
Additional test configuration files
bikegeek Oct 1, 2024
a289b6e
updated schema, now working
bikegeek Oct 1, 2024
a77daa1
modified test configuration file
bikegeek Oct 1, 2024
d4e7f53
Allow '-' in regex for limited string type
bikegeek Oct 2, 2024
48a31f7
Updated file so it is valid with respect to the schema
bikegeek Oct 2, 2024
2a8ffed
Include testing one of the xml specification files used in testing tw…
bikegeek Oct 2, 2024
cf56335
Reformat code for easier reading, update the load_met_gha_new.xml fil…
bikegeek Oct 2, 2024
94b6c50
include testing the load_met_gha_new.xml file
bikegeek Oct 2, 2024
d3ee5b1
Updated: reformatted and updated to conform to schema
bikegeek Oct 2, 2024
b052217
Added an extra date_list element
bikegeek Oct 3, 2024
0194da7
added testing xml specification file with more than one date_list
bikegeek Oct 3, 2024
0d19e5f
Clean up unnecessary comments
bikegeek Oct 3, 2024
14f5ce4
Update number of date_list items
bikegeek Oct 3, 2024
21b4af7
Explicitly set minLength and maxLength for hostname, db name, passwor…
bikegeek Oct 4, 2024
d3d604c
allow password to be string type and limit length of password
bikegeek Oct 6, 2024
eb95a15
comment out mysql commands. ci-run-all-cases
bikegeek Oct 6, 2024
ac1dde8
Fix comment
bikegeek Oct 6, 2024
085a429
Merge branch 'develop' into feature_internal_56_METdataio_validate_pa…
bikegeek Oct 11, 2024
8ffcc36
Merge branch 'develop' of https://github.com/dtcenter/METdataio into …
bikegeek Oct 11, 2024
e779512
Merge branch 'feature_internal_56_METdataio_validate_payload' of http…
bikegeek Oct 11, 2024
d78459b
Remove main function with hard-coded paths. Only useful during develo…
bikegeek Oct 11, 2024
4ae0299
Merge branch 'develop' of https://github.com/dtcenter/METdataio into …
bikegeek Oct 11, 2024
14f6023
Add support for line_type element
bikegeek Oct 23, 2024
d0dea80
Merge branch 'develop' of https://github.com/dtcenter/METdataio into …
bikegeek Oct 23, 2024
0d868e0
Added line_type element and some values for example xml specification
bikegeek Oct 23, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions METdbLoad/test/full_example.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -59,6 +59,12 @@
<val>grid_stat_cmn</val>
<val>point_stat_cmn</val>
</field>
<line_type>
<val>CNT</val>
<val>ORANK</val>
<val>SEEPS_MPR</val>
<val>ME</val>
</line_type>
</load_val>

<load_xml>true</load_xml>
Expand Down
17 changes: 15 additions & 2 deletions METdbLoad/ush/load_specification_schema.xsd
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -81,6 +81,14 @@
</xs:restriction>
</xs:simpleType>

<xs:simpleType name="linetype">
<xs:restriction base="xs:string">
<xs:minLength value="2"/>
<xs:maxLength value="25"/>
<xs:pattern value="[_a-zA-Z0-9]+"/>
</xs:restriction>
</xs:simpleType>


<!-- COMPLEX TYPES -->
<xs:element name="load_spec">
Expand Down Expand Up @@ -136,7 +144,6 @@
<xs:element name="field" minOccurs="0" maxOccurs="20">
<xs:complexType>
<xs:sequence>

<xs:element name="date_list" minOccurs="0" maxOccurs="5">
<xs:complexType>
<xs:simpleContent>
Expand All @@ -147,11 +154,17 @@
</xs:complexType>
</xs:element>
<xs:element type="xs:string" name="val" minOccurs="0" maxOccurs="100"/>

</xs:sequence>
<xs:attribute type="xs:string" name="name"/>
</xs:complexType>
</xs:element>
<xs:element name="line_type" minOccurs="0" maxOccurs="1">
<xs:complexType>
<xs:sequence>
<xs:element type="xs:string" name="val" maxOccurs="800" minOccurs="0"/>
</xs:sequence>
</xs:complexType>
</xs:element>
</xs:sequence>
</xs:complexType>
</xs:element>
Expand Down
Loading