IS-11: Test suites for Stream Compatibility Management #728

garethsb · 2022-11-04T09:22:16Z

... and BCP-005-01 EDID to Receiver Capabilities Mapping

Replaces #716 with an is-11 activity branch in the AMWA-TV/nmos-testing for simpler collaboration. The group can make PRs to extend the test suites onto this branch, which the activity group can review before merging. Others can use this 'stable' activity branch for testing.

…Interaction with IS-04", and "BCP-005-01 EDID to Receiver Capabilities Mapping", test suites

N-Nagorny

I tried it out and found minor problems. One of them is caused by https://github.com/AMWA-TV/nmos-testing/blob/master/nmostesting/GenericTest.py#L405: nmos-testing expects 200 from /inputs/{inputId}/edid/base but the default state is 204.

nmostesting/suites/IS1101Test.py

nmostesting/suites/IS1102Test.py

Merge lastest code from master on is-11

…eivers (4) (#761) Based on IS-11 Test Plan v0.6 Senders contains tests 2.0, 2.1 and 2.2 Receivers contains tests 4.1 and 4.2 Signed-off-by: ggeorgea <ggeorgea@matrox.com> Co-authored-by: Gareth Sylvester-Bradley <garethsb@nvidia.com>

# Conflicts: # nmostesting/Config.py

Adding IS-11 in README.md

…return either the pertinent IDs or make the message clearer. All messages were kind of normalized. #793

nmostesting/suites/IS1101Test.py

garethsb · 2023-05-01T04:11:16Z

nmostesting/suites/IS1101Test.py

+                            )
+                        state = response.json()["state"]
+                        if state in ["awating_essence", "no_essence"]:
+                            time.sleep(3)


Is maximum 5 x 3 seconds sleep part of the spec? If not, should probably be configurable?

This not not part of the IS-11 specification ... just an expectation that the device somehow is responsive within a few seconds.

…sement

Check IS-11 control advertisement like other test suites do

Merge pull request #802 from gwgeorgea/is-11-Issues-793

Add usage of STABLE_STATE_DELAY from Config.py A few failure comment clarifications

# Conflicts: # nmostesting/suites/IS1101Test.py

[IS-11] - Fix Test 4.4 to support devices without output

- Fix wrong list of receivers with outputs. Needed to test both audio and video properly. - New 256 byte EDID with CEA extension, prevents audio stream from going down during IS-11 test.

# Conflicts: # nmostesting/IS11Utils.py

[IS-11] Make test cases independent

…S and AUTH support

lo-simon · 2024-03-18T23:56:45Z

The following tests,
test_02_02_03_01, test_02_02_03_02, test_02_02_04_01, test_02_02_04_02, test_02_02_05_01, test_02_02_05_02, test_02_02_06_01, test_02_02_06_02, test_02_02_07_01, test_02_02_07_02, test_02_03_05_01, test_02_03_05_02
cannot be tested alone, data are required to be setup beforehand via the test_02_02_00
and
test_02_03_01, test_02_03_02, test_02_03_03, test_02_03_04
cannot be tested alone, data are required to be setup beforehand via the test_02_03_00
test_02_03_05_01 cannot test the video sender, which has only had one grain_rate constraint in its capability set
test_02_03_05_02 cannot test the audio sender, which has only had one sample_rate constraint in its capability set

jonathan-r-thorpe · 2024-03-19T09:26:12Z

nmostesting/suites/IS1101Test.py

+                if response.status_code == 422:
+                    print("Device does not accept grain_rate constraint")


This code will never be executed

jonathan-r-thorpe · 2024-03-19T09:31:27Z

nmostesting/suites/IS1101Test.py

+        """
+        Verify for inputs supporting EDID that the version and the effective EDID change when applying constraints
+        """


This has the same test description as test_02_03_05_01. What's different?

Maybe name one with video and the other one with audio

2.3.5.1 is video, 2.3.5.2 is audio

Suggested change

"""

Verify for inputs supporting EDID that the version and the effective EDID change when applying constraints

"""

"""

Verify for audio inputs supporting EDID that the version and the effective EDID change when applying constraints

"""

Bear in mind that these test descriptions are presented in the NMOS Testing user interface, so by having identical descriptions it makes it difficult for a user to disambiguate one test from the other.

jonathan-r-thorpe · 2024-03-19T09:35:11Z

nmostesting/suites/IS1101Test.py

+            return test.PASS()
+        return test.UNCLEAR("No resources found to perform this test.")
+
+    def test_02_03_05_02(self, test):


The test numbering seems very convoluted. AFAIK test names should be of the form test_XX. Where they are named test_XX_XX this is where additional tests have been added at a later date...

A compromise that takes into account the existing sectioning: test_XX: Senders ... naming (turning subsection numbers into word prefixes).

jonathan-r-thorpe · 2024-03-19T09:37:36Z

nmostesting/suites/IS1101Test.py

+                valid, response = TestHelper.do_request(
+                    "GET", self.compat_url + "inputs/" + input_id + "/properties/"
+                )
+                if not valid:
+                    return test.FAIL(
+                        "Unexpected response from the streamcompatibility API: {}".format(
+                            response
+                        )
+                    )
+                if response.status_code != 200:
+                    return test.FAIL(
+                        "The input {} properties streamcompatibility request has failed: {}".format(
+                            input_id, response
+                        )
+                    )
+                try:
+                    version = response.json()["version"]
+                except json.JSONDecodeError:
+                    return test.FAIL("Non-JSON response returned from Node API")
+                except KeyError as e:
+                    return test.FAIL("Unable to find expected key: {}".format(e))
+                self.version[input_id] = version


This seems to be a repeated pattern through this test. Perhaps factoring out a helper function would allow this test to be more compact?

jonathan-r-thorpe · 2024-03-19T09:42:21Z

nmostesting/suites/IS1101Test.py

+
+                default_edid = self.get_effective_edid(test, input_id)
+
+                self.another_grain_rate_constraints[sender_id] = {


another_grain_rate_constraints only appears to be used locally within this test. Why is it a class variable?

jonathan-r-thorpe · 2024-03-19T09:48:09Z

nmostesting/suites/IS1101Test.py

+    def get_another_grain_rate(self, grain_rate):
+        numerator = grain_rate["numerator"]
+        denominator = grain_rate["denominator"]
+        if (numerator == 30 or numerator == 25) and denominator == 1:
+            return {"numerator": numerator * 2, "denominator": 1}
+        if (numerator == 60 or numerator == 50) and denominator == 1:
+            return {"numerator": numerator / 2, "denominator": 1}
+        if numerator == 24 and denominator == 1:
+            return {"numerator": 30, "denominator": 1}
+        if (numerator == 30000 or numerator == 25000) and denominator == 1001:
+            return {"numerator": numerator * 2, "denominator": 1001}
+        if (numerator == 60000 or numerator == 50000) and denominator == 1001:
+            return {"numerator": numerator / 2, "denominator": 1001}
+        return "grain_rate not valid"


get_another_grain_rate assumes that the Node under test supports all these grain rates.
Similarly get_another_sample_rate assumes the Node supports all the defined sample rates.

If the node does not support the grain rate is will refuse the constraint and the test should be skipped (once the implementer fix the previously observed bug when testing the return status :). Note that there is yet no Sender Capabilities in NMOS.

Our experience is that the test aren't being skipped ('Could Not Test') but being failed when the Node doesn't support a particular grain rate...

nmostesting/suites/IS1101Test.py

lo-simon · 2024-03-19T12:22:19Z

nmostesting/suites/IS1101Test.py

+                if response.status_code == 422:
+                    print("Device does not accept grain_rate constraint")


Fix IS-11 test_02_03_04

…ns-fix [IS-11] Fix occasional exceptions caused by undefined variables.

Placeholders for "IS-11 Stream Compatibility Management API", "IS-11 …

4b52d49

…Interaction with IS-04", and "BCP-005-01 EDID to Receiver Capabilities Mapping", test suites

N-Nagorny reviewed Dec 6, 2022

View reviewed changes

nmostesting/suites/IS1101Test.py Outdated Show resolved Hide resolved

nmostesting/suites/IS1102Test.py Outdated Show resolved Hide resolved

garethsb and others added 4 commits January 27, 2023 16:24

Merge pull request #757 from AMWA-TV/master

08b7444

Merge lastest code from master on is-11

Merge remote-tracking branch 'origin/master' into is-11

2a2e7ce

Merge remote-tracking branch 'origin/master' into is-11

1bb8da3

# Conflicts: # nmostesting/Config.py

garethsb marked this pull request as ready for review April 27, 2023 15:43

gwgeorgea and others added 6 commits April 27, 2023 15:44

Adding IS-11 in README.md

90c9b24

Merge pull request #801 from gwgeorgea/is-11-Readme

1a56c8f

Adding IS-11 in README.md

Fix Issue 793 and in general review all FAIL and UNCLEAR messages to …

b745197

…return either the pertinent IDs or make the message clearer. All messages were kind of normalized. #793

Change wording on a few FAIL message items

3515a39

Check IS-11 control advertisement like other test suites do

c2e4e7e

Remove a blank line

6da067a

garethsb commented May 1, 2023

View reviewed changes

nmostesting/suites/IS1101Test.py Outdated Show resolved Hide resolved

garethsb commented May 1, 2023

View reviewed changes

gwgeorgea and others added 11 commits May 2, 2023 10:19

Fix gareth's mentions

f2f4850

Removing, accidentally commited file

91fc894

Pull up Device control advertisement test to NMOSUtils

82113dd

Reorder args

ffb1650

Merge remote-tracking branch 'origin/master' into is-11-is-04-adverti…

b9e00ca

…sement

Merge pull request #806 from N-Nagorny/is-11-is-04-advertisement

b5ac172

Check IS-11 control advertisement like other test suites do

Merge branch 'is-11' into is-11-Issues-793

38cacef

Fixing comments from gareth in PR-802

c0f7812

Remove accidentally change text regarding "The receivers"

e0852ef

Improve messages

55e9007

Merge pull request #802 from gwgeorgea/is-11-Issues-793

Merge remote-tracking branch 'origin/master' into is-11

6f36b1b

garethsb force-pushed the is-11 branch from 02259a3 to 6f36b1b Compare June 13, 2023 13:06

gwgeorgea and others added 3 commits June 14, 2023 09:10

Adding 'valid' verification to Fix issue #790

ba3a49d

Add usage of STABLE_STATE_DELAY from Config.py A few failure comment clarifications

Removing urlparse unused at this time for this PR

14bccb2

Extract fetching IS-11 resources to set_up_tests

c6c0c91

# Conflicts: # nmostesting/suites/IS1101Test.py

N-Nagorny and others added 12 commits November 1, 2023 14:47

Merge pull request #834 from gwgeorgea/is-11-Add-Support-without-output

4360422

[IS-11] - Fix Test 4.4 to support devices without output

Merge branch 'master' into is-11

404c863

Merge branch 'master' into is-11-independent-test-cases

d52c5f3

- Fix occasional exceptions caused by undefined variables.

d0bbe2b

- Fix wrong list of receivers with outputs. Needed to test both audio and video properly. - New 256 byte EDID with CEA extension, prevents audio stream from going down during IS-11 test.

Add tear_down_tests

d783ba2

Merge remote-tracking branch 'upstream/is-11' into is-11

bf3999c

Merge branch 'is-11' into is-11-independent-test-cases

11dba23

# Conflicts: # nmostesting/IS11Utils.py

Merge pull request #836 from N-Nagorny/is-11-independent-test-cases

57a4d79

[IS-11] Make test cases independent

Use the GenericTest::do_request instead of raw request to enable HTTP…

c6e0864

…S and AUTH support

Fix indent and remove unused request import

e92a93e

Fix indent

0201393

Replace TestHelper with self

4768170

jonathan-r-thorpe reviewed Mar 19, 2024

View reviewed changes

lo-simon reviewed Mar 19, 2024

View reviewed changes

nmostesting/suites/IS1101Test.py Outdated Show resolved Hide resolved

lo-simon reviewed Mar 19, 2024

View reviewed changes

nmostesting/suites/IS1101Test.py Outdated Show resolved Hide resolved

lo-simon reviewed Mar 19, 2024

View reviewed changes

lo-simon and others added 7 commits April 18, 2024 08:52

do_request doesn't take 4 parameters

31aba4a

Update tests description

5e015e2

Correct test_02_03_05_02 warning message

09fc19d

Shorten the lines to prevent flake8 line too lomg error

3882388

Merge pull request #849 from lo-simon/fix-test_02_03_04

2f2a5b1

Fix IS-11 test_02_03_04

Merge pull request #838 from gwgeorgea/is-11-Senders-outputs-exceptio…

adae812

…ns-fix [IS-11] Fix occasional exceptions caused by undefined variables.

Merge branch 'master' into is-11

0dafb47

N-Nagorny mentioned this pull request May 15, 2024

[IS-11] Independent tests for Senders #852

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

IS-11: Test suites for Stream Compatibility Management #728

IS-11: Test suites for Stream Compatibility Management #728

garethsb commented Nov 4, 2022

N-Nagorny left a comment

garethsb May 1, 2023

alabou May 1, 2023

lo-simon commented Mar 18, 2024 •

edited

Loading

jonathan-r-thorpe Mar 19, 2024

jonathan-r-thorpe Mar 19, 2024

lo-simon Mar 19, 2024

alabou Mar 19, 2024

jonathan-r-thorpe Mar 20, 2024 •

edited

Loading

jonathan-r-thorpe Mar 19, 2024

N-Nagorny Apr 27, 2024

jonathan-r-thorpe Mar 19, 2024 •

edited

Loading

jonathan-r-thorpe Mar 19, 2024

jonathan-r-thorpe Mar 19, 2024

alabou Mar 19, 2024

jonathan-r-thorpe Mar 20, 2024

lo-simon Mar 19, 2024

		if response.status_code == 422:
		print("Device does not accept grain_rate constraint")


		default_edid = self.get_effective_edid(test, input_id)

		self.another_grain_rate_constraints[sender_id] = {

IS-11: Test suites for Stream Compatibility Management #728

Are you sure you want to change the base?

IS-11: Test suites for Stream Compatibility Management #728

Conversation

garethsb commented Nov 4, 2022

N-Nagorny left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lo-simon commented Mar 18, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jonathan-r-thorpe Mar 20, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jonathan-r-thorpe Mar 19, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lo-simon commented Mar 18, 2024 •

edited

Loading

jonathan-r-thorpe Mar 20, 2024 •

edited

Loading

jonathan-r-thorpe Mar 19, 2024 •

edited

Loading