-
Notifications
You must be signed in to change notification settings - Fork 30
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Failures in Vulnerability Detection E2E Tests RC 2 #5397
Comments
Testing branch and package creation
OverviewIn order to determine the root of the failures in E2E tests, we have created the temporal branch Packages CreationFor this testing, we have created new packages only for Deb manager and Indexer. No changes were performed in the agents, so no new packages were needed. Note The creation of Indexer packages was necessitated by constraints within the deployment tool utilized for our End-to-End (E2E) tests (https://ci.wazuh.info/job/Wazuh_QA_environment/1166/) Environment deploymentThe testing environment was deployed using the Wazuh_QA_Environment pipeline. |
E2E Vulnerability Detection TestsImportant These tests were conducted with the outdated vulnerability index name (#5401). For results after resolving this issue, please check this comment: #5397 (comment) Report: R1.zip Reported issues:
Initial scan tests 🔴Initial scan tests fail due to the presence of errors in the managers and the absence of any vulnerabilities in the index:
Reviewing the evidence collected we can see the following errors in the report:
The Vulnerability Index is empty
In addition, we can see that the index isn't even created:
This was not a configuration error. We can see that initially, indexer-connected was correctly initialized
However, we can see that vulnerabilities were processed:
After this failure, the rest of the tests were aborted to troubleshoot the environment. The vulnerability state was never recovered even after restarting both managers and waiting more than half an hour. Important It seems that tests are failing due to the index has been renamed wazuh/wazuh#23274 After changing index name initial scans seems to detect the vulnerabilities. However, previously mentioned error was present. Reported in wazuh/wazuh#23512 |
Due to the launch tests running in debug mode, obtaining full results takes a significant amount of time. These tests have been automatically triggered using custom packages through the Test_e2e_system pipeline. We are using a custom branch, |
E2E Vulnerability Detection Tests (Vulnerability Index rename fix) 🔴This test iteration includes the fix for renaming the vulnerability tests. For more details, see: #5401. TestInitialScans 🔴SummaryVulnerability Detection Module E2E Tests
Details
|
Currently trying to replicate |
Research wazuh/wazuh#23530. This seems to be produced by a test bug. Reported in #5410 Initial scan discrepancies seem to be related to changes in VD content. We should consider avoiding increasing the timeout in #5404. Currently researching the issue |
Some final unexpected failures detected in the last iteration of the tests: Build: https://ci.wazuh.info/job/Test_e2e_system/289/ Test consistency initial scansRegarding the analysis performed during wazuh/wazuh#23523, alerts triggered during initial scans were expected due to content updates. This behavior should be taken into account by E2E test (created an issue to perform these changes #5412) vd_disabled_when_agents_registration Manager1
Manager2
However, the test checks for vulnerabilities much later:
The agents completed multiple syscollector scans since the feed update finished, so the final vulnerabilities should match the latest feed changes:
vd_enabled_when_agents_registration
However, the vulnerabilities have changed between scans. For example, the following vulnerability appeared in the second scan:
After discussing with @Dwordcito, it appears this is related to wazuh/wazuh#23482. I'll include this issue in the conclusion and ensure the report is added to the issue thread. Test change agent manager
After some research, it seems that vulnerabilities are correctly generated for this test, although the specified time is not enough. Reported in #5413
|
Some manual testing led to the detection of some issues in the case of cluster rename. Reported in wazuh/wazuh#23540 |
LGTM |
LGTM |
Description
The end-to-end (E2E) VD tests are failing. The tests were executed to validate the issue described in wazuh-qa issue #5368, but it seems the tests are not passing.
Initial thoughts attributed the failures to a database error. However, similar issues appeared in wazuh-qa issue #5319, where vulnerabilities were detected. Therefore, there might be an additional underlying problem.
Report details
Observations
macOS agent triggers an OS Vulnerability alert:
This alert suggests OS updates, which should not occur during the tests.
Windows alerts detected vulnerability mitigations before package installation. This might indicate an unexpected mid-test upgrade or a bad handling of the environment inventory/vulnerabilities.
Agent Reporting: Several agents did not report any alerts.
Analysis
Potential causes for these errors:
Test Errors
Known Product Errors
Unknown Production Errors
Action Plan
Since it is unclear if the database error is solely responsible, and there might be other underlying issues, we propose the following:
Validated by
Conclusion 🔴
E2E Tests
Vulnerability Detection
The text was updated successfully, but these errors were encountered: