Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for HPSS #22

Merged
merged 33 commits into from
Apr 9, 2024
Merged
Show file tree
Hide file tree
Changes from 32 commits
Commits
Show all changes
33 commits
Select commit Hold shift + click to select a range
bf0e78e
Added initial hsi and htar features.
DavidHuber-NOAA Mar 21, 2024
17b2ff2
Clean up hsi class.
DavidHuber-NOAA Mar 21, 2024
33098af
Add new hsi features.
DavidHuber-NOAA Mar 21, 2024
ad45216
Add htar capabilities.
DavidHuber-NOAA Mar 22, 2024
d54ea50
Refactor hsi and htar utils. (untested)
DavidHuber-NOAA Apr 1, 2024
2888aa8
Merge remote-tracking branch 'emc/develop' into feature/hsi
DavidHuber-NOAA Apr 1, 2024
d43b911
Fix some hsi bugs.
DavidHuber-NOAA Apr 1, 2024
453f6ac
Add mkdir function to hsi.
DavidHuber-NOAA Apr 2, 2024
3f9a529
Rename chgrp function
DavidHuber-NOAA Apr 2, 2024
9997b85
Remove unused executable module from fsutils
DavidHuber-NOAA Apr 2, 2024
ac337ac
Fix pynorms issues.
DavidHuber-NOAA Apr 2, 2024
af941c8
Sort fsutils functions.
DavidHuber-NOAA Apr 2, 2024
c47a003
add documentation for HPSS functions
DavidHuber-NOAA Apr 2, 2024
6609f7c
Convert hsi, htar functions to class methods.
DavidHuber-NOAA Apr 4, 2024
c0021f7
Created hsi and htar tests.
DavidHuber-NOAA Apr 4, 2024
958bd67
Sorted hsi and htar tests.
DavidHuber-NOAA Apr 4, 2024
838d156
Update documentation.
DavidHuber-NOAA Apr 4, 2024
faa4c02
Ignore missing files when removing them.
DavidHuber-NOAA Apr 4, 2024
d5f2cb6
Create parent directories by default when creating hpss tarballs.
DavidHuber-NOAA Apr 4, 2024
34043c2
Hide the command builder, provide default args to __init__.
DavidHuber-NOAA Apr 4, 2024
759e2b3
Hide the htar command builder.
DavidHuber-NOAA Apr 4, 2024
ac81104
Rename args to arg.
DavidHuber-NOAA Apr 4, 2024
3045a8e
Cleanup hsi and htar tests.
DavidHuber-NOAA Apr 4, 2024
470eb4a
Fix spacing for pynorms
DavidHuber-NOAA Apr 4, 2024
cc92cd1
Rename args to arg_list.
DavidHuber-NOAA Apr 4, 2024
0798c10
Parse file lists to htar as individual arguments.
DavidHuber-NOAA Apr 5, 2024
1652b54
hsi: better documentation; add helper function; catch empty args.
DavidHuber-NOAA Apr 8, 2024
ef745ac
htar: better documentation; add helper function; catch empty args
DavidHuber-NOAA Apr 8, 2024
0c9049f
Actually return group ID.
DavidHuber-NOAA Apr 8, 2024
1a661b1
Update hsi tests for renamed keywords.
DavidHuber-NOAA Apr 8, 2024
ccfcf92
Sort hsi and htar imports.
DavidHuber-NOAA Apr 8, 2024
2961846
Fix pycodestyle in hsi.py.
DavidHuber-NOAA Apr 8, 2024
7d7c83d
Add recursive flag for rm.
DavidHuber-NOAA Apr 8, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions docs/api.rst
Original file line number Diff line number Diff line change
Expand Up @@ -8,5 +8,7 @@ API
timetools
jinja
logger
hsi
htar


4 changes: 4 additions & 0 deletions docs/hsi.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
.. currentmodule:: wxflow

.. autoclass:: Hsi
:members:
4 changes: 4 additions & 0 deletions docs/htar.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
.. currentmodule:: wxflow

.. autoclass:: Htar
:members:
4 changes: 3 additions & 1 deletion src/wxflow/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,9 @@
from .executable import CommandNotFoundError, Executable, ProcessError, which
from .factory import Factory
from .file_utils import FileHandler
from .fsutils import chdir, cp, mkdir, mkdir_p, rm_p, rmdir
from .fsutils import chdir, chgrp, cp, get_gid, mkdir, mkdir_p, rm_p, rmdir
from .hsi import Hsi
from .htar import Htar
from .jinja import Jinja
from .logger import Logger, logit
from .sqlitedb import SQLiteDB, SQLiteDBError
Expand Down
22 changes: 21 additions & 1 deletion src/wxflow/fsutils.py
Original file line number Diff line number Diff line change
@@ -1,9 +1,11 @@
import contextlib
import errno
import grp
import os
import shutil

__all__ = ['mkdir', 'mkdir_p', 'rmdir', 'chdir', 'rm_p', 'cp']
__all__ = ['mkdir', 'mkdir_p', 'rmdir', 'chdir', 'rm_p', 'cp',
'get_gid', 'chgrp']


def mkdir_p(path):
Expand Down Expand Up @@ -85,3 +87,21 @@
raise OSError(f"unable to copy {source} to {target}")
except Exception as exc:
raise Exception(exc)


# Group ID number for a given group name
def get_gid(group_name: str):
try:
group_id = grp.getgrnam(group_name).gr_gid
except KeyError:
raise KeyError(f"{group_name} is not a valid group name.")

Check warning on line 97 in src/wxflow/fsutils.py

View check run for this annotation

Codecov / codecov/patch

src/wxflow/fsutils.py#L94-L97

Added lines #L94 - L97 were not covered by tests

return group_id

Check warning on line 99 in src/wxflow/fsutils.py

View check run for this annotation

Codecov / codecov/patch

src/wxflow/fsutils.py#L99

Added line #L99 was not covered by tests


# Change the group of a target file or directory
def chgrp(group_name, target, recursive=False):
# TODO add recursive option
gid = get_gid(group_name)
uid = os.stat(target).st_uid
os.chown(target, uid, gid)

Check warning on line 107 in src/wxflow/fsutils.py

View check run for this annotation

Codecov / codecov/patch

src/wxflow/fsutils.py#L105-L107

Added lines #L105 - L107 were not covered by tests
Loading