Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add test to check if files can be accessed while FIM has them opened #705

Merged
merged 35 commits into from
Jan 27, 2022
Merged

Add test to check if files can be accessed while FIM has them opened #705

merged 35 commits into from
Jan 27, 2022

Conversation

jesusjimsa
Copy link
Contributor

@jesusjimsa jesusjimsa commented May 11, 2020 •
edited by Deblintrake09
Loading

Description

This pull request adds a test that checks if a file can be accessed while FIM has it opened. This test creates a big file (~960MB), waits for the initial scan to start, and modifies or deletes the file. If Wazuh doesn't allow for it to be opened by any other process, a PermissionError exception is raised and the test fails. To pass, the test does the following assertions:

  • Rename: The file with the changed name exists.
  • Delete: The file no longer exists.
  • Both: The time where the modification was done, is before the end of the initial scan. Since there was only one file being monitored, it means the modification was done while the file was being scanned.

Note: The test only works for scheduled scans, since there is no current way of knowing when a scan will start for a file monitored with realtime or whodata modes.

This pull request closes #697.

Packages

Box Specs System Version Revision Link
centos/8 2gb ram / 1 core Linux Agent 4.3.0 0.40301.20211123 https://packages-dev.wazuh.com/staging/yum/wazuh-agent-4.3.0-0.40301.20211123.x86_64.rpm
centos/8 2gb ram / 1 core Windows 4.3.0 0.40301.20211123 https://packages-dev.wazuh.com/staging/windows/wazuh-agent-4.3.0-0.40302.20211123.msi
gusztavvargadr/windows-10 2gb ram / 1 core Manager 4.3.0 0.40301.20211123 https://packages-dev.wazuh.com/staging/yum/wazuh-manager-4.3.0-0.40302.20211123.x86_64.rpm

Testing

test_basic_usage_access_opened_files.py

Type Instance Results Date
Manager R1 🟢 2021/11/29
Manager R2 🟢 2021/11/29
Manager R3 🟢 2021/11/29
Linux Agent R1 🟢 2021/11/29
Linux Agent R2 🟢 2021/11/29
LinuxAgent R3 🟢 2021/11/29
Windows R1 🟢 2021/11/29
Windows R2 🟢 2021/11/29
Windows R3 🟢 2021/11/29
Solaris R1 🟢 - Jenkins 2021/11/29
Solaris R2 🟢 - Jenkins 2021/11/29
Solaris R3 🟢 - Jenkins 2021/11/29
Macos R1 🟢 - Jenkins 2021/11/29
Macos R2 🟢 - Jenkins 2021/11/29
Macos R3 🟢 - Jenkins 2021/11/29

Update 2021/12/03

Test Path OS Type Version Date Status Exectued by
test_fim/test_files/test_basic_usage/test_basic_usage_access_while_open.py Linux Manager 4.3.0 2021/12/03 🟢 @Deblintrake09
test_fim/test_files/test_basic_usage/test_basic_usage_access_while_open.py Linux Manager 4.3.0 2021/12/03 🟢 @Deblintrake09
test_fim/test_files/test_basic_usage/test_basic_usage_access_while_open.py Linux Manager 4.2.5 2021/12/03 🟢 @Deblintrake09
test_fim/test_files/test_basic_usage/test_basic_usage_access_while_open.py Linux Local 4.2.5 2021/12/03 🟢 @Deblintrake09
test_fim/test_files/test_basic_usage/test_basic_usage_access_while_open.py Linux Local 4.3.0 2021/12/03 🟢 @Deblintrake09
test_fim/test_files/test_basic_usage/test_basic_usage_access_while_open.py Windows Local 4.3.0 2021/12/03 🟢 @Deblintrake09
test_fim/test_files/test_basic_usage/test_basic_usage_access_while_open.py Windows Local 4.2.5 2021/12/03 🟢 @Deblintrake09
test_fim/test_files/test_basic_usage/test_basic_usage_access_while_open.py Windows Local 4.3.0 2021/12/03 🟢 @Deblintrake09
test_fim/test_files/test_basic_usage/test_basic_usage_access_while_open.py Macos Jenkins 4.3.0 2021/12/03 🟢 @Deblintrake09
test_fim/test_files/test_basic_usage/test_basic_usage_access_while_open.py Solaris Jenkins 4.3.0 2021/12/03 🟢 @Deblintrake09

@jesusjimsa jesusjimsa requested a review from bah07 May 11, 2020 14:04
@jesusjimsa jesusjimsa self-assigned this May 11, 2020
@jesusjimsa
Copy link
Contributor Author

Windows results

================================================= test session starts =================================================
platform win32 -- Python 3.7.3, pytest-5.1.2, py-1.8.0, pluggy-0.13.0
rootdir: C:\Users\jmv74211\Desktop\wazuh-qa\tests\integration, inifile: pytest.ini
plugins: html-2.0.1, metadata-1.9.0, testinfra-5.0.0
collected 6 items

test_basic_usage_access_opened_files.py ......                                                                   [100%]

============================================ 6 passed in 66.12s (0:01:06) =============================================

@jesusjimsa
Copy link
Contributor Author

Linux results

=============================== test session starts ================================
platform linux -- Python 3.6.8, pytest-5.4.2, py-1.8.1, pluggy-0.13.1
rootdir: /home/vagrant/qa/tests/integration, inifile: pytest.ini
plugins: metadata-1.9.0, html-2.0.1, testinfra-5.0.0
collected 6 items

test_fim/test_basic_usage/test_basic_usage_access_opened_files.py ......     [100%]

========================== 6 passed in 237.32s (0:03:57) ===========================

@jesusjimsa
Copy link
Contributor Author

jesusjimsa commented May 11, 2020 •
edited
Loading

Results above are performed in the 5017-non-blocking-hash-windows wazuh/wazuh branch.

@vikman90 vikman90 added this to the Sprint 111 - Core milestone May 13, 2020
@bah07 bah07 removed this from the Sprint 111 - Core milestone May 25, 2020
@jesusjimsa jesusjimsa changed the base branch from master to develop July 20, 2020 08:43
@jesusjimsa jesusjimsa changed the base branch from develop to master September 2, 2020 12:41
@jesusjimsa jesusjimsa removed their assignment Jul 13, 2021
juliamagan
juliamagan requested changes Sep 7, 2021
View reviewed changes
Copy link
Member

@juliamagan juliamagan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

GJ, but some changes are required.

tests/integration/test_fim/test_basic_usage/test_basic_usage_access_opened_files.py Outdated
@@ -0,0 +1,105 @@
# Copyright (C) 2015-2020, Wazuh Inc.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Update Copyright

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in commit d19040b

tests/integration/test_fim/test_basic_usage/test_basic_usage_access_opened_files.py Outdated
Comment on lines 37 to 39
p, m = generate_params(extra_params=conf_params)

configurations = load_wazuh_configurations(configurations_path, __name__, params=p, metadata=m)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
p, m = generate_params(extra_params=conf_params)
configurations = load_wazuh_configurations(configurations_path, __name__, params=p, metadata=m)
parameters, metadata = generate_params(extra_params=conf_params)
configurations = load_wazuh_configurations(configurations_path, __name__, params=parameters, metadata=metadata)

Variables with just a single letter are not allowed, they should have descriptive names

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in commit c678fdf

tests/integration/test_fim/test_basic_usage/test_basic_usage_access_opened_files.py Outdated
Comment on lines 50 to 65
@pytest.fixture(scope='function')
def create_and_restore_large_file(request):
if not os.path.exists(testdir1):
os.mkdir(testdir1)

file_size = 1024 * 1024 * 768 # 805 MB
chunksize = 1024 * 768
file_path = os.path.join(testdir1, 'large_file')
changed_path = os.path.join(testdir1, 'changed_name')

if os.path.exists(changed_path):
os.rename(changed_path, file_path)
elif not os.path.exists(file_path):
with open(file_path, "a") as f:
while os.stat(file_path).st_size < file_size:
f.write(random.choice(string.printable) * chunksize)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This fixture should create and delete the file at the end of the test, so when we are finished, the environment is clean. This will create a new file for each test case, instead of renaming the file for the second test case. You can take a look at this example:

wazuh-qa/tests/integration/test_fim/test_files/test_report_changes/test_disk_quota_values.py

Lines 53 to 64 in fe154ff

@pytest.fixture(scope='function')
def create_specific_size_file(get_configuration, request):
"""Create a file with a specific size requested from test configuration"""
test_file = os.path.join(temp_dir, 'test')
# Translate given size from string to number in bytes
translated_size = translate_size(configured_size=get_configuration['metadata']['disk_quota_limit'])
write_file(test_file, random_string(translated_size*compression_ratio))
yield
remove_file(test_file)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in commit d9cdee7

tests/integration/test_fim/test_basic_usage/test_basic_usage_access_opened_files.py Outdated
try:
os.rename(file_path, changed_path)
except (OSError, IOError, PermissionError):
pytest.fail("Could not rename file")
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
pytest.fail("Could not rename file")
pytest.fail('Could not rename file')

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in commit c806811

tests/integration/test_fim/test_basic_usage/test_basic_usage_access_opened_files.py Outdated
try:
os.remove(file_path)
except (OSError, IOError, PermissionError):
pytest.fail("Could not delete file")
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
pytest.fail("Could not delete file")
pytest.fail('Could not delete file')

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in commit c806811

@CamiRomero
Copy link
Contributor

CamiRomero commented Oct 20, 2021 •
edited
Loading

Package

Version Revision Link
4.3.0 https://packages-dev.wazuh.com/staging/yum/wazuh-agent-4.3.0-0.40301.20211007.x86_64.rpm

Testing

the test failed, I still work on this

Centos Local Jenkins Notes
R1 🔴
R2
R3

  • 🟢 : All pass

  • 🟡 : Some warnings

  • 🔴 : Some errors/fails

  • 🔵 : In progress

@CamiRomero
Copy link
Contributor

After rebase the test continued with failures I working on this in a temporal branch https://github.com/wazuh/wazuh-qa/tree/tmp_705

Jenkins report

@CamiRomero CamiRomero self-assigned this Oct 20, 2021
@CamiRomero
Copy link
Contributor

Package

Version Revision Link
4.3.0 https://packages-dev.wazuh.com/staging/yum/wazuh-agent-4.3.0-0.40301.20211007.x86_64.rpm

Testing

/test_fim/test_files/test_basic_usage

Centos Local Jenkins Notes
R1 🟢 🟢
R2 🟢 🟢
R3 🟢 🟢

  • 🟢 : All pass

  • 🟡 : Some warnings

  • 🔴 : Some errors/fails

  • 🔵 : In progress

@CamiRomero CamiRomero requested review from juliamagan and removed request for bah07 October 21, 2021 02:43
@Deblintrake09 Deblintrake09 self-assigned this Nov 29, 2021
@Deblintrake09 Deblintrake09 changed the base branch from master to 2040-4.3-full-green November 29, 2021 14:15
@Deblintrake09 Deblintrake09 changed the base branch from 2040-4.3-full-green to master November 29, 2021 14:15
damarisg
damarisg requested changes Nov 30, 2021
View reviewed changes
deps/wazuh_testing/wazuh_testing/fim.py Show resolved Hide resolved
deps/wazuh_testing/wazuh_testing/fim.py Show resolved Hide resolved
deps/wazuh_testing/wazuh_testing/fim.py Show resolved Hide resolved
tests/integration/test_fim/test_files/test_basic_usage/test_basic_usage_access_opened_files.py Outdated Show resolved Hide resolved
tests/integration/test_fim/test_files/test_basic_usage/test_basic_usage_access_opened_files.py Outdated Show resolved Hide resolved
tests/integration/test_fim/test_files/test_basic_usage/test_basic_usage_access_opened_files.py Outdated Show resolved Hide resolved
tests/integration/test_fim/test_files/test_basic_usage/test_basic_usage_access_opened_files.py Outdated
can modified (renamed/deleted), and that wazuh is not
blocking the files.

wazuh_min_version: 4.3
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you check if the min version that covers it is 4.3, please? I think that it also works to 4.2

tests/integration/test_fim/test_files/test_basic_usage/test_basic_usage_access_opened_files.py Outdated Show resolved Hide resolved
tests/integration/test_fim/test_files/test_basic_usage/test_basic_usage_access_opened_files.py Outdated Show resolved Hide resolved
tests/integration/test_fim/test_files/test_basic_usage/test_basic_usage_access_opened_files.py Show resolved Hide resolved
CamiRomero
CamiRomero suggested changes Dec 3, 2021
View reviewed changes
Copy link
Contributor

@CamiRomero CamiRomero left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

GJ, but some changes are required

tests/integration/test_fim/test_files/test_basic_usage/test_basic_usage_access_opened_files.py Outdated Show resolved Hide resolved
tests/integration/test_fim/test_files/test_basic_usage/test_basic_usage_access_opened_files.py Outdated Show resolved Hide resolved
tests/integration/test_fim/test_files/test_basic_usage/test_basic_usage_access_opened_files.py Outdated Show resolved Hide resolved
tests/integration/test_fim/test_files/test_basic_usage/test_basic_usage_access_opened_files.py Show resolved Hide resolved
damarisg
damarisg previously approved these changes Dec 6, 2021
View reviewed changes
Copy link
Member

@damarisg damarisg left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Test Results

Test Path OS Version Date Status
test_fim/test_files/test_basic_usage/test_basic_usage_access_while_open.py Windows - Agent 4.3.0 2021/12/06 🟢
test_fim/test_files/test_basic_usage/test_basic_usage_access_while_open.py Windows - Agent 4.3.0 2021/12/06 🟢
test_fim/test_files/test_basic_usage/test_basic_usage_access_while_open.py Windows - Agent 4.3.0 2021/12/06 🟢
test_fim/ Windows - Agent 4.3.0 2021/12/06 🟢
test_fim/ Windows - Agent 4.3.0 2021/12/06 🟢
test_fim/ Windows - Agent 4.3.0 2021/12/06 🟢
test_fim/ Linux - Agent 4.3.0 2021/12/06 🟢
test_fim/ Macos- Agent 4.3.0 2021/12/06 🟢
test_fim/ Solaris - Agent 4.3.0 2021/12/06 🟢
test_fim/ Linux - Manager 4.3.0 2021/12/06 🟢

@snaow snaow merged commit 6767e7a into master Jan 27, 2022
@snaow snaow deleted the 697-access-while-open branch January 27, 2022 18:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@Deblintrake09 Deblintrake09 Deblintrake09 left review comments

@Rebits Rebits Rebits requested changes

@juliamagan juliamagan juliamagan requested changes

@damarisg damarisg damarisg left review comments

@CamiRomero CamiRomero Awaiting requested review from CamiRomero

Assignees

@Deblintrake09 Deblintrake09

@CamiRomero CamiRomero

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Add test to check access to files monitored by FIM while Wazuh has them open
9 participants
@jesusjimsa @CamiRomero @Rebits @Deblintrake09 @damarisg @juliamagan @snaow @bah07 @vikman90