CRAYSAT-1861: Unmount Ceph and s3fs filesystems on ncn-m001 during shutdown #227
Conversation
|
@shivaprasad-metimath, @annapoorna-s-alt , there is still further cleanup and testing I would like to do on this, but since it is a large PR, I wanted to give you both plenty of time to review it. Here is the testing performed so far on fanta: https://gist.github.com/haasken-hpe/bfc0d8aed96c19cf48cba680e17485c0 |
haasken-hpe
force-pushed
the
CRAYSAT-1861-umount-ceph-s3fs-ncn-m001
branch
from
2fa9867
to
117893f
Compare
haasken-hpe
force-pushed
the
CRAYSAT-1861-umount-ceph-s3fs-ncn-m001
branch
from
f7abb55
to
9d7c69b
Compare
|
Marking this ready for review. I have not yet tested it by actually performing the power off/on of a real Shasta system, but I have thoroughly unit tested it. Please review, @shivaprasad-metimath and @annapoorna-s-alt. I will test this tomorrow, or during our meeting on Monday. |
|
Testing was completed on rocket. I ran the full system shutdown procedure with the build of the cray-sat image from this branch. It is far too much testing output to attach to this PR or a gist, so I'll attach the typescripts to the Jira. There were no issues identified with these code changes. |
Modify the `sat bootsys shutdown --stage ncn-power` command to do the following on ncn-m001: * Find all mounts of filesystems of type `ceph` or `fuse.s3fs` * Find all mounts of filesystems on RBD devices * For each mount point identified, check whether there is any activity on that mount point that would prevent it being unmounted. Prompt the admin when they have stopped the processes before continuing. * Disable systemd cron job that ensures Ceph and s3fs mounts remain mounted. * Unmount each mount point * Unmap all RBD devices Modify the `sat bootsys boot --stage ncn-power` command to re-enable the systemd cron job that ensures Ceph/s3fs mounts are mounted on ncn-m001. Prior to this change, only the unmapping of RBD devices was performed. We want to ensure that all usages of Ceph on ncn-m001 are gracefully stopped before shutting down the storage nodes so that ncn-m001 will shut down gracefully. Test Description: Added many unit tests. Testing is needed on a real system as part of a full-system shutdown.
The `FilteredHostKeys` class was created to work around an issue with Paramiko's loading of SSH host keys that was causing a serious performance problem when the known_hosts file was large. However, this class was not being used to filter host keys in all locations where a Paramiko `SSHClient` was being created. Update the locations which were not filtering host keys to do so. Test Description: Needs to be tested as part of a full-system power off/on on a real system.
haasken-hpe
force-pushed
the
CRAYSAT-1861-umount-ceph-s3fs-ncn-m001
branch
from
5f06612
to
c740815
Compare
Summary and Scope
Unmount filesystems of type
cephandfuse.s3fson ncn-m001 when shutting down. Also do the same for any filesystems mounted from RBD devices.Add an activity check that uses
lsofto check if filesystems can be unmounted or if there are active processes using them, and report these to the admin, so they can stop the offending processes as appropriate.Disable the cron job that automatically mounts Ceph and s3fs filesystems on ncn-m001, so that they won't get remounted before the node is shut down. Re-enable it at the appropriate time during the power on.
Issues and Related PRs
Testing
Tested on:
Test description:
Minimal testing so far. Just tested the functions on their own to limit system disruption.
Still need to test full-system power down.
Risks and Mitigations
Adds a significant amount of code and output to
sat bootsys, so we will want to update the docs, but overall this should be a really positive improvement to the robustness and completeness of the system shutdown automation.Pull Request Checklist