-
Notifications
You must be signed in to change notification settings - Fork 0
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
AssertionError: feed_data after feed_eof #293
Comments
FTR here is the status among recently modified zarrs
didn't spot any hint on possible cause yet in the logs, will proceed with reset etc on those not clean |
grr -- it did crash again even with
the log is /mnt/backup/dandi/dandisets/.git/dandi/backups2datalad/2022.11.02.16.24.18Z.log . @jwodder -- please have another fresh look and try to figure out what could be the problem here, either we are not swallowing some exception and either we are logging all stdout/stderr which might be relevant. |
Add lower-level logging of unexpected exceptions
process crashed,
@jwodder does it mean that agronholm/anyio#490 is not our underlying issue?
full listing(venv) (base) dandi@drogon:/mnt/backup/dandi/dandisets/.git/dandi/backups2datalad$ ps auxw -H | grep -A3 git-anne[x]
dandi 3086865 0.0 0.0 1074053024 2072 pts/8 Sl Oct26 0:00 git-annex find --include=* --json
dandi 3086872 0.0 0.0 21876 1656 pts/8 S Oct26 0:00 git --git-dir=.git --work-tree=. --literal-pathspecs ls-files --stage -z --error-unmatch --
dandi 3086873 0.0 0.0 50496 516 pts/8 S Oct26 0:00 git --git-dir=.git --work-tree=. --literal-pathspecs cat-file --batch-check=%(objectname) %(objecttype) %(objectsize) --buffer
dandi 3086874 0.0 0.0 50496 544 pts/8 S Oct26 0:00 git --git-dir=.git --work-tree=. --literal-pathspecs cat-file --batch=%(objectname) %(objecttype) %(objectsize) --buffer
dandi 3086887 0.0 0.0 1074053024 2072 pts/8 Sl Oct26 0:00 git-annex find --include=* --json
dandi 3086894 0.0 0.0 21876 1780 pts/8 S Oct26 0:00 git --git-dir=.git --work-tree=. --literal-pathspecs ls-files --stage -z --error-unmatch --
dandi 3086895 0.0 0.0 50512 1096 pts/8 S Oct26 0:00 git --git-dir=.git --work-tree=. --literal-pathspecs cat-file --batch-check=%(objectname) %(objecttype) %(objectsize) --buffer
dandi 3086896 0.0 0.0 50512 1144 pts/8 S Oct26 0:00 git --git-dir=.git --work-tree=. --literal-pathspecs cat-file --batch=%(objectname) %(objecttype) %(objectsize) --buffer
dandi 2338941 0.0 0.0 1074053024 2408 pts/8 Sl Oct31 0:05 git-annex find --include=* --json
dandi 2338948 0.0 0.0 21052 1840 pts/8 S Oct31 0:00 git --git-dir=.git --work-tree=. --literal-pathspecs ls-files --stage -z --error-unmatch --
dandi 2338949 0.0 0.0 45388 1624 pts/8 S Oct31 0:00 git --git-dir=.git --work-tree=. --literal-pathspecs cat-file --batch-check=%(objectname) %(objecttype) %(objectsize) --buffer
dandi 2338950 0.0 0.0 45388 1700 pts/8 S Oct31 0:00 git --git-dir=.git --work-tree=. --literal-pathspecs cat-file --batch=%(objectname) %(objecttype) %(objectsize) --buffer
dandi 2339079 0.0 0.0 1074053024 2408 pts/8 Sl Oct31 0:05 git-annex find --include=* --json
dandi 2339086 0.0 0.0 20368 1900 pts/8 S Oct31 0:00 git --git-dir=.git --work-tree=. --literal-pathspecs ls-files --stage -z --error-unmatch --
dandi 2339087 0.0 0.0 44612 1392 pts/8 S Oct31 0:00 git --git-dir=.git --work-tree=. --literal-pathspecs cat-file --batch-check=%(objectname) %(objecttype) %(objectsize) --buffer
dandi 2339088 0.0 0.0 44612 1308 pts/8 S Oct31 0:00 git --git-dir=.git --work-tree=. --literal-pathspecs cat-file --batch=%(objectname) %(objecttype) %(objectsize) --buffer
dandi 2339578 0.0 0.0 1074053024 2360 pts/8 Sl Oct31 0:05 git-annex find --include=* --json
dandi 2339585 0.0 0.0 20368 1892 pts/8 S Oct31 0:00 git --git-dir=.git --work-tree=. --literal-pathspecs ls-files --stage -z --error-unmatch --
dandi 2339586 0.0 0.0 44612 1376 pts/8 S Oct31 0:00 git --git-dir=.git --work-tree=. --literal-pathspecs cat-file --batch-check=%(objectname) %(objecttype) %(objectsize) --buffer
dandi 2339587 0.0 0.0 44612 1456 pts/8 S Oct31 0:00 git --git-dir=.git --work-tree=. --literal-pathspecs cat-file --batch=%(objectname) %(objecttype) %(objectsize) --buffer
dandi 2344705 0.0 0.0 1074053024 2348 pts/8 Sl Oct31 0:00 git-annex find --include=* --json
dandi 2344712 0.0 0.0 19164 1776 pts/8 S Oct31 0:00 git --git-dir=.git --work-tree=. --literal-pathspecs ls-files --stage -z --error-unmatch --
dandi 2344713 0.0 0.0 39896 8 pts/8 S Oct31 0:00 git --git-dir=.git --work-tree=. --literal-pathspecs cat-file --batch-check=%(objectname) %(objecttype) %(objectsize) --buffer
dandi 2344714 0.0 0.0 39896 8 pts/8 S Oct31 0:00 git --git-dir=.git --work-tree=. --literal-pathspecs cat-file --batch=%(objectname) %(objecttype) %(objectsize) --buffer
dandi 1003159 0.0 0.0 1074053024 2984 pts/8 Sl Nov02 0:11 git-annex find --include=* --json
dandi 1003166 0.0 0.0 24976 11192 pts/8 S Nov02 0:00 git --git-dir=.git --work-tree=. --literal-pathspecs ls-files --stage -z --error-unmatch --
dandi 1003167 0.0 0.0 43496 1060 pts/8 S Nov02 0:00 git --git-dir=.git --work-tree=. --literal-pathspecs cat-file --batch-check=%(objectname) %(objecttype) %(objectsize) --buffer
dandi 1003168 0.0 0.0 43496 1060 pts/8 S Nov02 0:00 git --git-dir=.git --work-tree=. --literal-pathspecs cat-file --batch=%(objectname) %(objecttype) %(objectsize) --buffer
dandi 1004853 0.0 0.0 1074053024 5424 pts/8 Sl Nov02 0:09 git-annex find --include=* --json
dandi 1004860 0.0 0.0 24976 4616 pts/8 S Nov02 0:00 git --git-dir=.git --work-tree=. --literal-pathspecs ls-files --stage -z --error-unmatch --
dandi 1004861 0.0 0.0 43508 8 pts/8 S Nov02 0:00 git --git-dir=.git --work-tree=. --literal-pathspecs cat-file --batch-check=%(objectname) %(objecttype) %(objectsize) --buffer
dandi 1004862 0.0 0.0 43508 8 pts/8 S Nov02 0:00 git --git-dir=.git --work-tree=. --literal-pathspecs cat-file --batch=%(objectname) %(objecttype) %(objectsize) --buffer
dandi 1005122 0.0 0.0 1074053024 4488 pts/8 Sl Nov02 0:08 git-annex find --include=* --json
dandi 1005129 0.0 0.0 24976 7040 pts/8 S Nov02 0:00 git --git-dir=.git --work-tree=. --literal-pathspecs ls-files --stage -z --error-unmatch --
dandi 1005130 0.0 0.0 52232 12 pts/8 S Nov02 0:00 git --git-dir=.git --work-tree=. --literal-pathspecs cat-file --batch-check=%(objectname) %(objecttype) %(objectsize) --buffer
dandi 1005131 0.0 0.0 52232 8 pts/8 S Nov02 0:00 git --git-dir=.git --work-tree=. --literal-pathspecs cat-file --batch=%(objectname) %(objecttype) %(objectsize) --buffer
dandi 1005596 0.0 0.0 1074053024 6224 pts/8 Sl Nov02 0:08 git-annex find --include=* --json
dandi 1005603 0.0 0.0 24976 8576 pts/8 S Nov02 0:00 git --git-dir=.git --work-tree=. --literal-pathspecs ls-files --stage -z --error-unmatch --
dandi 1005604 0.0 0.0 56708 1724 pts/8 S Nov02 0:00 git --git-dir=.git --work-tree=. --literal-pathspecs cat-file --batch-check=%(objectname) %(objecttype) %(objectsize) --buffer
dandi 1005605 0.0 0.0 56708 1824 pts/8 S Nov02 0:00 git --git-dir=.git --work-tree=. --literal-pathspecs cat-file --batch=%(objectname) %(objecttype) %(objectsize) --buffer
dandi 1012017 0.0 0.0 1074053024 2332 pts/8 Sl Nov02 0:07 git-annex find --include=* --json
dandi 1012044 0.0 0.0 24976 1812 pts/8 S Nov02 0:00 git --git-dir=.git --work-tree=. --literal-pathspecs ls-files --stage -z --error-unmatch --
dandi 1012046 0.0 0.0 58948 1872 pts/8 S Nov02 0:00 git --git-dir=.git --work-tree=. --literal-pathspecs cat-file --batch-check=%(objectname) %(objecttype) %(objectsize) --buffer
dandi 1012048 0.0 0.0 58948 8 pts/8 S Nov02 0:00 git --git-dir=.git --work-tree=. --literal-pathspecs cat-file --batch=%(objectname) %(objecttype) %(objectsize) --buffer
dandi 1023752 0.0 0.0 1074053024 2460 pts/8 Sl Nov02 0:05 git-annex find --include=* --json
dandi 1023759 0.0 0.0 24976 1872 pts/8 S Nov02 0:00 git --git-dir=.git --work-tree=. --literal-pathspecs ls-files --stage -z --error-unmatch --
dandi 1023760 0.0 0.0 60564 8 pts/8 S Nov02 0:00 git --git-dir=.git --work-tree=. --literal-pathspecs cat-file --batch-check=%(objectname) %(objecttype) %(objectsize) --buffer
dandi 1023761 0.0 0.0 60564 8 pts/8 S Nov02 0:00 git --git-dir=.git --work-tree=. --literal-pathspecs cat-file --batch=%(objectname) %(objecttype) %(objectsize) --buffer
dandi 1024182 0.0 0.0 1074053024 2136 pts/8 Sl Nov02 0:05 git-annex find --include=* --json
dandi 1024189 0.0 0.0 24976 1928 pts/8 S Nov02 0:00 git --git-dir=.git --work-tree=. --literal-pathspecs ls-files --stage -z --error-unmatch --
dandi 1024190 0.0 0.0 59532 32 pts/8 S Nov02 0:00 git --git-dir=.git --work-tree=. --literal-pathspecs cat-file --batch-check=%(objectname) %(objecttype) %(objectsize) --buffer
dandi 1024191 0.0 0.0 59532 228 pts/8 S Nov02 0:00 git --git-dir=.git --work-tree=. --literal-pathspecs cat-file --batch=%(objectname) %(objecttype) %(objectsize) --buffer
dandi 1025269 0.0 0.0 1074053024 2748 pts/8 Sl Nov02 0:05 git-annex find --include=* --json
dandi 1025280 0.0 0.0 24976 1876 pts/8 S Nov02 0:00 git --git-dir=.git --work-tree=. --literal-pathspecs ls-files --stage -z --error-unmatch --
dandi 1025281 0.0 0.0 55924 792 pts/8 S Nov02 0:00 git --git-dir=.git --work-tree=. --literal-pathspecs cat-file --batch-check=%(objectname) %(objecttype) %(objectsize) --buffer
dandi 1025282 0.0 0.0 55924 924 pts/8 S Nov02 0:00 git --git-dir=.git --work-tree=. --literal-pathspecs cat-file --batch=%(objectname) %(objecttype) %(objectsize) --buffer
dandi 1025721 0.0 0.0 1074053024 4820 pts/8 Sl Nov02 0:05 git-annex find --include=* --json
dandi 1025728 0.0 0.0 24976 10372 pts/8 S Nov02 0:00 git --git-dir=.git --work-tree=. --literal-pathspecs ls-files --stage -z --error-unmatch --
dandi 1025729 0.0 0.0 53088 1220 pts/8 S Nov02 0:00 git --git-dir=.git --work-tree=. --literal-pathspecs cat-file --batch-check=%(objectname) %(objecttype) %(objectsize) --buffer
dandi 1025731 0.0 0.0 53088 1392 pts/8 S Nov02 0:00 git --git-dir=.git --work-tree=. --literal-pathspecs cat-file --batch=%(objectname) %(objecttype) %(objectsize) --buffer
dandi 1026364 0.0 0.0 1074053024 2348 pts/8 Sl Nov02 0:04 git-annex find --include=* --json
dandi 1026371 0.0 0.0 24976 10696 pts/8 S Nov02 0:00 git --git-dir=.git --work-tree=. --literal-pathspecs ls-files --stage -z --error-unmatch --
dandi 1026372 0.0 0.0 45864 52 pts/8 S Nov02 0:00 git --git-dir=.git --work-tree=. --literal-pathspecs cat-file --batch-check=%(objectname) %(objecttype) %(objectsize) --buffer
dandi 1026373 0.0 0.0 45864 8 pts/8 S Nov02 0:00 git --git-dir=.git --work-tree=. --literal-pathspecs cat-file --batch=%(objectname) %(objecttype) %(objectsize) --buffer
dandi 1031092 0.0 0.0 1074053024 8156 pts/8 Sl Nov02 0:01 git-annex find --include=* --json
dandi 1031099 0.0 0.0 24480 3940 pts/8 S Nov02 0:00 git --git-dir=.git --work-tree=. --literal-pathspecs ls-files --stage -z --error-unmatch --
dandi 1031100 0.0 0.0 46304 1564 pts/8 S Nov02 0:00 git --git-dir=.git --work-tree=. --literal-pathspecs cat-file --batch-check=%(objectname) %(objecttype) %(objectsize) --buffer
dandi 1031101 0.0 0.0 46304 1668 pts/8 S Nov02 0:00 git --git-dir=.git --work-tree=. --literal-pathspecs cat-file --batch=%(objectname) %(objecttype) %(objectsize) --buffer
dandi 1032723 0.0 0.0 1074053024 2304 pts/8 Sl Nov02 0:00 git-annex find --include=* --json
dandi 1032730 0.0 0.0 24480 3460 pts/8 S Nov02 0:00 git --git-dir=.git --work-tree=. --literal-pathspecs ls-files --stage -z --error-unmatch --
dandi 1032731 0.0 0.0 52248 992 pts/8 S Nov02 0:00 git --git-dir=.git --work-tree=. --literal-pathspecs cat-file --batch-check=%(objectname) %(objecttype) %(objectsize) --buffer
dandi 1032732 0.0 0.0 52248 1244 pts/8 S Nov02 0:00 git --git-dir=.git --work-tree=. --literal-pathspecs cat-file --batch=%(objectname) %(objecttype) %(objectsize) --buffer
dandi 4515 0.0 0.0 1074053024 16248 pts/8 Sl Nov03 0:12 git-annex find --include=* --json
dandi 4522 0.0 0.0 0 0 pts/8 Z Nov03 0:00 [git] <defunct>
dandi 4523 0.0 0.0 52232 1176 pts/8 S Nov03 0:00 git --git-dir=.git --work-tree=. --literal-pathspecs cat-file --batch-check=%(objectname) %(objecttype) %(objectsize) --buffer
dandi 4524 0.0 0.0 52232 1796 pts/8 S Nov03 0:00 git --git-dir=.git --work-tree=. --literal-pathspecs cat-file --batch=%(objectname) %(objecttype) %(objectsize) --buffer
dandi 7620 0.0 0.0 1074053024 14392 pts/8 Sl Nov03 0:08 git-annex find --include=* --json
dandi 7627 0.0 0.0 24976 15964 pts/8 S Nov03 0:00 git --git-dir=.git --work-tree=. --literal-pathspecs ls-files --stage -z --error-unmatch --
dandi 7628 0.0 0.0 59532 1376 pts/8 S Nov03 0:00 git --git-dir=.git --work-tree=. --literal-pathspecs cat-file --batch-check=%(objectname) %(objecttype) %(objectsize) --buffer
dandi 7629 0.0 0.0 59532 268 pts/8 S Nov03 0:00 git --git-dir=.git --work-tree=. --literal-pathspecs cat-file --batch=%(objectname) %(objecttype) %(objectsize) --buffer
dandi 26207 0.0 0.0 1074053024 13056 pts/8 Sl Nov03 0:04 git-annex find --include=* --json
dandi 26214 0.0 0.0 24976 15720 pts/8 S Nov03 0:00 git --git-dir=.git --work-tree=. --literal-pathspecs ls-files --stage -z --error-unmatch --
dandi 26215 0.0 0.0 45864 756 pts/8 S Nov03 0:00 git --git-dir=.git --work-tree=. --literal-pathspecs cat-file --batch-check=%(objectname) %(objecttype) %(objectsize) --buffer
dandi 26216 0.0 0.0 45864 900 pts/8 S Nov03 0:00 git --git-dir=.git --work-tree=. --literal-pathspecs cat-file --batch=%(objectname) %(objecttype) %(objectsize) --buffer
dandi 26217 0.0 0.0 1074053024 12024 pts/8 Sl Nov03 0:04 git-annex find --include=* --json
dandi 26224 0.0 0.0 24976 16076 pts/8 S Nov03 0:00 git --git-dir=.git --work-tree=. --literal-pathspecs ls-files --stage -z --error-unmatch --
dandi 26225 0.0 0.0 53088 284 pts/8 S Nov03 0:00 git --git-dir=.git --work-tree=. --literal-pathspecs cat-file --batch-check=%(objectname) %(objecttype) %(objectsize) --buffer
dandi 26226 0.0 0.0 53088 300 pts/8 S Nov03 0:00 git --git-dir=.git --work-tree=. --literal-pathspecs cat-file --batch=%(objectname) %(objecttype) %(objectsize) --buffer
dandi 26227 0.0 0.0 1074053024 12464 pts/8 Sl Nov03 0:04 git-annex find --include=* --json
dandi 26234 0.0 0.0 24976 15992 pts/8 S Nov03 0:00 git --git-dir=.git --work-tree=. --literal-pathspecs ls-files --stage -z --error-unmatch --
dandi 26235 0.0 0.0 55924 2020 pts/8 S Nov03 0:00 git --git-dir=.git --work-tree=. --literal-pathspecs cat-file --batch-check=%(objectname) %(objecttype) %(objectsize) --buffer
dandi 26236 0.0 0.0 55924 2172 pts/8 S Nov03 0:00 git --git-dir=.git --work-tree=. --literal-pathspecs cat-file --batch=%(objectname) %(objecttype) %(objectsize) --buffer
dandi 26432 0.0 0.0 1074053024 13776 pts/8 Sl Nov03 0:03 git-annex find --include=* --json
dandi 26439 0.0 0.0 24480 15288 pts/8 S Nov03 0:00 git --git-dir=.git --work-tree=. --literal-pathspecs ls-files --stage -z --error-unmatch --
dandi 26440 0.0 0.0 46304 700 pts/8 S Nov03 0:00 git --git-dir=.git --work-tree=. --literal-pathspecs cat-file --batch-check=%(objectname) %(objecttype) %(objectsize) --buffer
dandi 26441 0.0 0.0 46304 644 pts/8 S Nov03 0:00 git --git-dir=.git --work-tree=. --literal-pathspecs cat-file --batch=%(objectname) %(objecttype) %(objectsize) --buffer
dandi 26618 0.0 0.0 1074053024 11736 pts/8 Sl Nov03 0:03 git-annex find --include=* --json
dandi 26625 0.0 0.0 24480 15304 pts/8 S Nov03 0:00 git --git-dir=.git --work-tree=. --literal-pathspecs ls-files --stage -z --error-unmatch --
dandi 26626 0.0 0.0 52248 284 pts/8 S Nov03 0:00 git --git-dir=.git --work-tree=. --literal-pathspecs cat-file --batch-check=%(objectname) %(objecttype) %(objectsize) --buffer
dandi 26627 0.0 0.0 52248 300 pts/8 S Nov03 0:00 git --git-dir=.git --work-tree=. --literal-pathspecs cat-file --batch=%(objectname) %(objecttype) %(objectsize) --buffer
dandi 27041 0.0 0.0 1074053024 11508 pts/8 Sl Nov03 0:02 git-annex find --include=* --json
dandi 27048 0.0 0.0 24480 15440 pts/8 S Nov03 0:00 git --git-dir=.git --work-tree=. --literal-pathspecs ls-files --stage -z --error-unmatch --
dandi 27049 0.0 0.0 55288 284 pts/8 S Nov03 0:00 git --git-dir=.git --work-tree=. --literal-pathspecs cat-file --batch-check=%(objectname) %(objecttype) %(objectsize) --buffer
dandi 27050 0.0 0.0 55288 300 pts/8 S Nov03 0:00 git --git-dir=.git --work-tree=. --literal-pathspecs cat-file --batch=%(objectname) %(objecttype) %(objectsize) --buffer
dandi 27507 0.0 0.0 1074053024 11888 pts/8 Sl Nov03 0:02 git-annex find --include=* --json
dandi 27514 0.0 0.0 24480 15300 pts/8 S Nov03 0:00 git --git-dir=.git --work-tree=. --literal-pathspecs ls-files --stage -z --error-unmatch --
dandi 27515 0.0 0.0 54860 280 pts/8 S Nov03 0:00 git --git-dir=.git --work-tree=. --literal-pathspecs cat-file --batch-check=%(objectname) %(objecttype) %(objectsize) --buffer
dandi 27516 0.0 0.0 54860 304 pts/8 S Nov03 0:00 git --git-dir=.git --work-tree=. --literal-pathspecs cat-file --batch=%(objectname) %(objecttype) %(objectsize) --buffer
dandi 27561 0.0 0.0 1074053024 11896 pts/8 Sl Nov03 0:02 git-annex find --include=* --json
dandi 27576 0.0 0.0 24480 15348 pts/8 S Nov03 0:00 git --git-dir=.git --work-tree=. --literal-pathspecs ls-files --stage -z --error-unmatch --
dandi 27577 0.0 0.0 54564 284 pts/8 S Nov03 0:00 git --git-dir=.git --work-tree=. --literal-pathspecs cat-file --batch-check=%(objectname) %(objecttype) %(objectsize) --buffer
dandi 27578 0.0 0.0 54564 300 pts/8 S Nov03 0:00 git --git-dir=.git --work-tree=. --literal-pathspecs cat-file --batch=%(objectname) %(objecttype) %(objectsize) --buffer
dandi 27762 0.0 0.0 1074053024 11868 pts/8 Sl Nov03 0:02 git-annex find --include=* --json
dandi 27769 0.0 0.0 24480 15348 pts/8 S Nov03 0:00 git --git-dir=.git --work-tree=. --literal-pathspecs ls-files --stage -z --error-unmatch --
dandi 27770 0.0 0.0 52888 284 pts/8 S Nov03 0:00 git --git-dir=.git --work-tree=. --literal-pathspecs cat-file --batch-check=%(objectname) %(objecttype) %(objectsize) --buffer
dandi 27771 0.0 0.0 52888 304 pts/8 S Nov03 0:00 git --git-dir=.git --work-tree=. --literal-pathspecs cat-file --batch=%(objectname) %(objecttype) %(objectsize) --buffer
dandi 28340 0.0 0.0 1074053024 12576 pts/8 Sl Nov03 0:01 git-annex find --include=* --json
dandi 28347 0.0 0.0 24480 15352 pts/8 S Nov03 0:00 git --git-dir=.git --work-tree=. --literal-pathspecs ls-files --stage -z --error-unmatch --
dandi 28348 0.0 0.0 51756 856 pts/8 S Nov03 0:00 git --git-dir=.git --work-tree=. --literal-pathspecs cat-file --batch-check=%(objectname) %(objecttype) %(objectsize) --buffer
dandi 28350 0.0 0.0 51756 824 pts/8 S Nov03 0:00 git --git-dir=.git --work-tree=. --literal-pathspecs cat-file --batch=%(objectname) %(objecttype) %(objectsize) --buffer
dandi 28997 0.0 0.0 1074053024 9744 pts/8 Sl Nov03 0:00 git-annex find --include=* --json
dandi 29004 0.0 0.0 24480 15444 pts/8 S Nov03 0:00 git --git-dir=.git --work-tree=. --literal-pathspecs ls-files --stage -z --error-unmatch --
dandi 29005 0.0 0.0 47764 280 pts/8 S Nov03 0:00 git --git-dir=.git --work-tree=. --literal-pathspecs cat-file --batch-check=%(objectname) %(objecttype) %(objectsize) --buffer
dandi 29006 0.0 0.0 47764 2140 pts/8 S Nov03 0:00 git --git-dir=.git --work-tree=. --literal-pathspecs cat-file --batch=%(objectname) %(objecttype) %(objectsize) --buffer
--
dandi 3616062 0.7 0.0 1074053024 36472 pts/8 Sl 07:31 0:11 git-annex find --include=* --json
dandi 3616069 0.0 0.0 25048 18052 pts/8 S 07:31 0:00 git --git-dir=.git --work-tree=. --literal-pathspecs ls-files --stage -z --error-unmatch --
dandi 3616070 0.0 0.0 40604 11644 pts/8 S 07:31 0:00 git --git-dir=.git --work-tree=. --literal-pathspecs cat-file --batch-check=%(objectname) %(objecttype) %(objectsize) --buffer
dandi 3616071 0.0 0.0 40604 11528 pts/8 S 07:31 0:00 git --git-dir=.git --work-tree=. --literal-pathspecs cat-file --batch=%(objectname) %(objecttype) %(objectsize) --buffer
dandi 3616142 0.7 0.0 1074053024 34376 pts/8 Sl 07:31 0:12 git-annex find --include=* --json
dandi 3616149 0.0 0.0 0 0 pts/8 Z 07:31 0:00 [git] <defunct>
dandi 3616150 0.0 0.0 60064 24028 pts/8 S 07:31 0:00 git --git-dir=.git --work-tree=. --literal-pathspecs cat-file --batch-check=%(objectname) %(objecttype) %(objectsize) --buffer
dandi 3616151 0.0 0.0 60064 19888 pts/8 S 07:31 0:00 git --git-dir=.git --work-tree=. --literal-pathspecs cat-file --batch=%(objectname) %(objecttype) %(objectsize) --buffer
dandi 3616220 0.7 0.0 1074053024 34692 pts/8 Sl 07:31 0:11 git-annex find --include=* --json
dandi 3616227 0.0 0.0 0 0 pts/8 Z 07:31 0:00 [git] <defunct>
dandi 3616228 0.0 0.0 55664 19908 pts/8 S 07:31 0:00 git --git-dir=.git --work-tree=. --literal-pathspecs cat-file --batch-check=%(objectname) %(objecttype) %(objectsize) --buffer
dandi 3616229 0.0 0.0 55664 20032 pts/8 S 07:31 0:00 git --git-dir=.git --work-tree=. --literal-pathspecs cat-file --batch=%(objectname) %(objecttype) %(objectsize) --buffer
dandi 3616299 0.7 0.0 1074053024 34720 pts/8 Sl 07:31 0:11 git-annex find --include=* --json
dandi 3616306 0.0 0.0 24976 17852 pts/8 S 07:31 0:00 git --git-dir=.git --work-tree=. --literal-pathspecs ls-files --stage -z --error-unmatch --
dandi 3616307 0.0 0.0 51312 20100 pts/8 S 07:31 0:00 git --git-dir=.git --work-tree=. --literal-pathspecs cat-file --batch-check=%(objectname) %(objecttype) %(objectsize) --buffer
dandi 3616308 0.0 0.0 51312 15704 pts/8 S 07:31 0:00 git --git-dir=.git --work-tree=. --literal-pathspecs cat-file --batch=%(objectname) %(objecttype) %(objectsize) --buffer
dandi 3616682 0.7 0.0 1074053024 35300 pts/8 Sl 07:31 0:11 git-annex find --include=* --json
dandi 3616689 0.0 0.0 24976 18012 pts/8 S 07:31 0:00 git --git-dir=.git --work-tree=. --literal-pathspecs ls-files --stage -z --error-unmatch --
dandi 3616690 0.0 0.0 43496 15976 pts/8 S 07:31 0:00 git --git-dir=.git --work-tree=. --literal-pathspecs cat-file --batch-check=%(objectname) %(objecttype) %(objectsize) --buffer
dandi 3616691 0.0 0.0 43496 15924 pts/8 S 07:31 0:00 git --git-dir=.git --work-tree=. --literal-pathspecs cat-file --batch=%(objectname) %(objecttype) %(objectsize) --buffer
dandi 3617046 0.6 0.0 1074053024 34468 pts/8 Sl 07:32 0:10 git-annex find --include=* --json
dandi 3617053 0.0 0.0 24976 18088 pts/8 S 07:32 0:00 git --git-dir=.git --work-tree=. --literal-pathspecs ls-files --stage -z --error-unmatch --
dandi 3617054 0.0 0.0 43508 11756 pts/8 S 07:32 0:00 git --git-dir=.git --work-tree=. --literal-pathspecs cat-file --batch-check=%(objectname) %(objecttype) %(objectsize) --buffer
dandi 3617055 0.0 0.0 43508 11408 pts/8 S 07:32 0:00 git --git-dir=.git --work-tree=. --literal-pathspecs cat-file --batch=%(objectname) %(objecttype) %(objectsize) --buffer
dandi 3617298 0.5 0.0 1074053024 36844 pts/8 Sl 07:32 0:09 git-annex find --include=* --json
dandi 3617305 0.0 0.0 24976 18044 pts/8 S 07:32 0:00 git --git-dir=.git --work-tree=. --literal-pathspecs ls-files --stage -z --error-unmatch --
dandi 3617306 0.0 0.0 52232 15760 pts/8 S 07:32 0:00 git --git-dir=.git --work-tree=. --literal-pathspecs cat-file --batch-check=%(objectname) %(objecttype) %(objectsize) --buffer
dandi 3617307 0.0 0.0 52232 15744 pts/8 S 07:32 0:00 git --git-dir=.git --work-tree=. --literal-pathspecs cat-file --batch=%(objectname) %(objecttype) %(objectsize) --buffer
dandi 3617796 0.5 0.0 1074053024 34628 pts/8 Sl 07:33 0:07 git-annex find --include=* --json
dandi 3617803 0.0 0.0 24976 17988 pts/8 S 07:33 0:00 git --git-dir=.git --work-tree=. --literal-pathspecs ls-files --stage -z --error-unmatch --
dandi 3617804 0.0 0.0 56708 15716 pts/8 S 07:33 0:00 git --git-dir=.git --work-tree=. --literal-pathspecs cat-file --batch-check=%(objectname) %(objecttype) %(objectsize) --buffer
dandi 3617805 0.0 0.0 56708 15748 pts/8 S 07:33 0:00 git --git-dir=.git --work-tree=. --literal-pathspecs cat-file --batch=%(objectname) %(objecttype) %(objectsize) --buffer
dandi 3617970 0.5 0.0 1074053024 34524 pts/8 Sl 07:33 0:07 git-annex find --include=* --json
dandi 3617977 0.0 0.0 24976 17980 pts/8 S 07:33 0:00 git --git-dir=.git --work-tree=. --literal-pathspecs ls-files --stage -z --error-unmatch --
dandi 3617978 0.0 0.0 58948 15936 pts/8 S 07:33 0:00 git --git-dir=.git --work-tree=. --literal-pathspecs cat-file --batch-check=%(objectname) %(objecttype) %(objectsize) --buffer
dandi 3617979 0.0 0.0 58948 15792 pts/8 S 07:33 0:00 git --git-dir=.git --work-tree=. --literal-pathspecs cat-file --batch=%(objectname) %(objecttype) %(objectsize) --buffer
dandi 3618449 0.4 0.0 1074053024 34440 pts/8 Sl 07:34 0:06 git-annex find --include=* --json
dandi 3618456 0.0 0.0 24976 18064 pts/8 S 07:34 0:00 git --git-dir=.git --work-tree=. --literal-pathspecs ls-files --stage -z --error-unmatch --
dandi 3618457 0.0 0.0 60564 16148 pts/8 S 07:34 0:00 git --git-dir=.git --work-tree=. --literal-pathspecs cat-file --batch-check=%(objectname) %(objecttype) %(objectsize) --buffer
dandi 3618458 0.0 0.0 60564 15952 pts/8 S 07:34 0:00 git --git-dir=.git --work-tree=. --literal-pathspecs cat-file --batch=%(objectname) %(objecttype) %(objectsize) --buffer
dandi 3620441 0.2 0.0 1074053024 34208 pts/8 Sl 07:35 0:03 git-annex find --include=* --json
dandi 3620448 0.0 0.0 24976 18192 pts/8 S 07:35 0:00 git --git-dir=.git --work-tree=. --literal-pathspecs ls-files --stage -z --error-unmatch --
dandi 3620449 0.0 0.0 59532 15880 pts/8 S 07:35 0:00 git --git-dir=.git --work-tree=. --literal-pathspecs cat-file --batch-check=%(objectname) %(objecttype) %(objectsize) --buffer
dandi 3620450 0.0 0.0 59532 15736 pts/8 S 07:35 0:00 git --git-dir=.git --work-tree=. --literal-pathspecs cat-file --batch=%(objectname) %(objecttype) %(objectsize) --buffer
dandi 3620982 0.1 0.0 1074053024 34376 pts/8 Sl 07:36 0:02 git-annex find --include=* --json
dandi 3620989 0.0 0.0 24976 18120 pts/8 S 07:36 0:00 git --git-dir=.git --work-tree=. --literal-pathspecs ls-files --stage -z --error-unmatch --
dandi 3620990 0.0 0.0 55924 11924 pts/8 S 07:36 0:00 git --git-dir=.git --work-tree=. --literal-pathspecs cat-file --batch-check=%(objectname) %(objecttype) %(objectsize) --buffer
dandi 3620991 0.0 0.0 55924 11512 pts/8 S 07:36 0:00 git --git-dir=.git --work-tree=. --literal-pathspecs cat-file --batch=%(objectname) %(objecttype) %(objectsize) --buffer
dandi 3621445 0.0 0.0 1074053024 34520 pts/8 Sl 07:37 0:00 git-annex find --include=* --json
dandi 3621452 0.0 0.0 24976 18052 pts/8 S 07:37 0:00 git --git-dir=.git --work-tree=. --literal-pathspecs ls-files --stage -z --error-unmatch --
dandi 3621453 0.0 0.0 53088 11892 pts/8 S 07:37 0:00 git --git-dir=.git --work-tree=. --literal-pathspecs cat-file --batch-check=%(objectname) %(objecttype) %(objectsize) --buffer
dandi 3621454 0.0 0.0 53088 11624 pts/8 S 07:37 0:00 git --git-dir=.git --work-tree=. --literal-pathspecs cat-file --batch=%(objectname) %(objecttype) %(objectsize) --buffer
(venv) (base) dandi@drogon:/mnt/backup/dandi/dandisets/.git/dandi/backups2datalad$
here are the zarrs with counts of processes for each(venv) (base) dandi@drogon:/mnt/backup/dandi/dandisets/.git/dandi/backups2datalad$ ps auxw -H | grep 'git-annex find' | awk '{print $2;}' | while read p; do readlink /proc/$p/cwd; done | sort | uniq -c | sort -n
1 /mnt/backup/dandi/dandizarrs/1fba437e-d693-4571-bd67-f84126c20792
1 /mnt/backup/dandi/dandizarrs/3516f9b2-e61b-4d2c-a3d8-7c3dd31433a6
1 /mnt/backup/dandi/dandizarrs/3f8b02a9-dea4-4811-87a9-ae1ea358c372
1 /mnt/backup/dandi/dandizarrs/3fcfa4ae-0456-4432-821b-570d5b9dd603
1 /mnt/backup/dandi/dandizarrs/464df854-b8d8-4da9-9e8d-d2de848156dc
1 /mnt/backup/dandi/dandizarrs/83bfd77b-e3b3-4b2f-ab1b-b10856634f8d
1 /mnt/backup/dandi/dandizarrs/8b3b3222-9ad9-43ff-8d38-0076e9dff4c6
1 /mnt/backup/dandi/dandizarrs/a2f939e7-9c38-4267-8c5a-373ccb9663f7
1 /mnt/backup/dandi/dandizarrs/b5c86d5c-82fc-4a04-ad4d-64dc0bece535
1 /mnt/backup/dandi/dandizarrs/c2b1efb6-27c5-4747-9b79-7141ed98a513
1 /mnt/backup/dandi/dandizarrs/cba6df7e-d1a7-4ecf-821f-496df9d6ffd5
1 /mnt/backup/dandi/dandizarrs/d0238e53-3c40-4fa5-b347-7285e35f19d3
1 /mnt/backup/dandi/dandizarrs/de17fdd7-65ab-4041-ba8a-23c581c8be43
1 /mnt/backup/dandi/dandizarrs/e162e9bf-19d8-4e16-b98c-8da3c4eed08c
1 /mnt/backup/dandi/dandizarrs/e5bc3c38-fb9a-4784-a5ab-9ebba8de0267
1 /mnt/backup/dandi/dandizarrs/eedba4a8-781e-4c68-a1d0-ccfd9d6a171f
2 /mnt/backup/dandi/dandizarrs/239da6c9-fecb-4d9c-86a5-b9d48ae43367
2 /mnt/backup/dandi/dandizarrs/3c716d0a-2071-4f2e-a6d8-deea33636857
2 /mnt/backup/dandi/dandizarrs/3ff0dc45-3711-475c-848c-3a5e8d18e237
2 /mnt/backup/dandi/dandizarrs/61f67b6c-b201-4881-8fc3-9449c4e662ec
2 /mnt/backup/dandi/dandizarrs/87581439-b257-4276-97f7-6b5d236a331f
2 /mnt/backup/dandi/dandizarrs/b16720b1-c7ac-418e-a415-795facb05d55
2 /mnt/backup/dandi/dandizarrs/cc2dcb99-61da-4b21-a3ed-55b0daabf738
2 /mnt/backup/dandi/dandizarrs/fa81584a-bb9b-4a26-b66e-5694f57efee0
3 /mnt/backup/dandi/dandizarrs/368567e1-5b70-4707-892a-922998919cfc
3 /mnt/backup/dandi/dandizarrs/4d121041-6ee4-4c6f-8f02-a5e0e204cb4d
3 /mnt/backup/dandi/dandizarrs/91eedea3-38d9-455e-918e-305c5792c50c
3 /mnt/backup/dandi/dandizarrs/a0b98e7b-bdcc-4ddc-b660-3480204714b8 they were all clean(venv) (base) dandi@drogon:/mnt/backup/dandi/dandisets/.git/dandi/backups2datalad$ ps auxw -H | grep 'git-annex find' | awk '{print $2;}' | while read p; do readlink /proc/$p/cwd; done | sort | uniq | while read z; do (cd $z; git status; git clean -dfx; git reset --hard; ); done
On branch draft
Your branch is up to date with 'github/draft'.
nothing to commit, working tree clean
HEAD is now at 34b9dbf703 [backups2datalad] 98959 files added, checksum updated
On branch draft
Your branch is up to date with 'github/draft'.
nothing to commit, working tree clean
HEAD is now at 45126f0ef [backups2datalad] 102787 files added, checksum updated
On branch draft
Your branch is up to date with 'github/draft'.
nothing to commit, working tree clean
HEAD is now at 2934d294d5 [backups2datalad] 98959 files added, checksum updated
On branch draft
Your branch is up to date with 'github/draft'.
nothing to commit, working tree clean
HEAD is now at 663ade3155 [backups2datalad] 102787 files added, checksum updated
On branch draft
Your branch is up to date with 'github/draft'.
nothing to commit, working tree clean
HEAD is now at 3fd0abbd21 [backups2datalad] 98959 files added, checksum updated
On branch draft
Your branch is up to date with 'github/draft'.
nothing to commit, working tree clean
HEAD is now at ecb9bdd20a [backups2datalad] 102787 files added, checksum updated
On branch draft
Your branch is up to date with 'github/draft'.
nothing to commit, working tree clean
HEAD is now at 744d9ac049 [backups2datalad] checksum updated
On branch draft
Your branch is up to date with 'github/draft'.
nothing to commit, working tree clean
HEAD is now at c215a9e79e [backups2datalad] 102787 files added, checksum updated
On branch draft
Your branch is up to date with 'github/draft'.
nothing to commit, working tree clean
HEAD is now at dbd1f6cad6 [backups2datalad] checksum updated
On branch draft
Your branch is up to date with 'github/draft'.
nothing to commit, working tree clean
HEAD is now at 37f5726c87 [backups2datalad] 102787 files added, checksum updated
On branch draft
Your branch is up to date with 'github/draft'.
nothing to commit, working tree clean
HEAD is now at 1ffa12c26 [backups2datalad] 102787 files added, checksum updated
On branch draft
Your branch is up to date with 'github/draft'.
nothing to commit, working tree clean
HEAD is now at 2427625afa [backups2datalad] checksum updated
On branch draft
Your branch is up to date with 'github/draft'.
nothing to commit, working tree clean
HEAD is now at dd3bc600b0 [backups2datalad] 102787 files added, checksum updated
On branch draft
Your branch is up to date with 'github/draft'.
nothing to commit, working tree clean
HEAD is now at ad73de7fb2 [backups2datalad] 98959 files added, checksum updated
On branch draft
Your branch is up to date with 'github/draft'.
nothing to commit, working tree clean
HEAD is now at 86f93f16f0 [backups2datalad] 102787 files added, checksum updated
On branch draft
Your branch is up to date with 'github/draft'.
nothing to commit, working tree clean
HEAD is now at c5897677ee [backups2datalad] 102787 files added, checksum updated
On branch draft
Your branch is up to date with 'github/draft'.
nothing to commit, working tree clean
HEAD is now at 39ae78ff7 [backups2datalad] 103386 files added, checksum updated
On branch draft
Your branch is up to date with 'github/draft'.
nothing to commit, working tree clean
HEAD is now at 3129672e8a [backups2datalad] 102787 files added, checksum updated
On branch draft
Your branch is up to date with 'github/draft'.
nothing to commit, working tree clean
HEAD is now at 807c918b3f [backups2datalad] checksum updated
On branch draft
Your branch is up to date with 'github/draft'.
nothing to commit, working tree clean
HEAD is now at 4ea903f3c6 [backups2datalad] 98959 files added, checksum updated
On branch draft
Your branch is up to date with 'github/draft'.
nothing to commit, working tree clean
HEAD is now at ae636da6d0 [backups2datalad] 98959 files added, checksum updated
On branch draft
Your branch is up to date with 'github/draft'.
nothing to commit, working tree clean
HEAD is now at 97bafe1f84 [backups2datalad] 98959 files added, checksum updated
On branch draft
Your branch is up to date with 'github/draft'.
nothing to commit, working tree clean
HEAD is now at 0411a0b5b1 [backups2datalad] 102787 files added, checksum updated
On branch draft
Your branch is up to date with 'github/draft'.
nothing to commit, working tree clean
HEAD is now at e701b6ce71 [backups2datalad] 102787 files added, checksum updated
On branch draft
Your branch is up to date with 'github/draft'.
nothing to commit, working tree clean
HEAD is now at 8e83e20450 [backups2datalad] checksum updated
On branch draft
Your branch is up to date with 'github/draft'.
nothing to commit, working tree clean
HEAD is now at b3ad264add [backups2datalad] 98959 files added, checksum updated
On branch draft
Your branch is up to date with 'github/draft'.
nothing to commit, working tree clean
HEAD is now at ce48d69c5 [backups2datalad] checksum updated
On branch draft
Your branch is up to date with 'github/draft'.
nothing to commit, working tree clean
HEAD is now at 40ced110f8 [backups2datalad] 102787 files added, checksum updated
|
@yarikoptic I'm not sure what exactly is happening anymore. However, I've done my best to kill any & all subprocess on error in #295. |
After #295 we get something "fresh":
full log - /mnt/backup/dandi/dandisets/.git/dandi/backups2datalad/2022.11.07.18.36.17Z.log .
(dandisets) dandi@drogon:/mnt/backup/dandi/dandisets/.git/dandi/backups2datalad$ grep RuntimeError /mnt/backup/dandi/dandisets/.git/dandi/backups2datalad/2022.11.01.19.31.16Z.log /mnt/backup/dandi/dandisets/.git/dandi/backups2datalad/2022.11.02.16.24.18Z.log
/mnt/backup/dandi/dandisets/.git/dandi/backups2datalad/2022.11.01.19.31.16Z.log: raise RuntimeError(
/mnt/backup/dandi/dandisets/.git/dandi/backups2datalad/2022.11.01.19.31.16Z.log:RuntimeError: Zarr eb14aa4a-af93-41be-a62a-aa0dd573d581: local checksum 'fe61f9e261c479d7e9d250e88a7659f1-103386--149262577373' differs from remote checksum 'c72911f5e683930da80c90e8bcf3104a-4000--5785686453' after backup, and no change on server was detected
/mnt/backup/dandi/dandisets/.git/dandi/backups2datalad/2022.11.01.19.31.16Z.log: raise RuntimeError(
/mnt/backup/dandi/dandisets/.git/dandi/backups2datalad/2022.11.01.19.31.16Z.log:RuntimeError: Backups for 1 Dandiset failed
/mnt/backup/dandi/dandisets/.git/dandi/backups2datalad/2022.11.01.19.31.16Z.log: raise RuntimeError(
/mnt/backup/dandi/dandisets/.git/dandi/backups2datalad/2022.11.01.19.31.16Z.log:RuntimeError: Backups for 1 Dandiset failed
/mnt/backup/dandi/dandisets/.git/dandi/backups2datalad/2022.11.01.19.31.16Z.log: raise RuntimeError(
/mnt/backup/dandi/dandisets/.git/dandi/backups2datalad/2022.11.01.19.31.16Z.log:RuntimeError: Backups for 1 Dandiset failed
/mnt/backup/dandi/dandisets/.git/dandi/backups2datalad/2022.11.01.19.31.16Z.log: raise RuntimeError(
/mnt/backup/dandi/dandisets/.git/dandi/backups2datalad/2022.11.01.19.31.16Z.log:RuntimeError: Backups for 1 Dandiset failed
is our logic for this error message on zarrs just means that zarr computed on the server differs from what we get from S3, and zarr should be reingested?
|
finished running that grep
so some old logs didn't have it but recent did, I will run sync for that zarr alone for now and see where we get. It was redigested:
so we have that one matching! I will reset recent dirty zarrs again and restart backup process. @jwodder do you think it would be possible to delay crashing the entire process due to that RuntimeError until all zarrs/assets are considered i.e. to accumulate the errors and then spit out some "composite exception"? I know that it sounds cumbersome, but I am afraid that it would be more than one of such zarrs in the archive and given that we go through them in the same order each time and it takes hours to reach the next failing |
Exactly what do you want to happen instead of the current behavior? If a checksum mismatch is detected after backing up a Zarr, should the Zarr backup just conclude normally aside from the delayed error? Exactly when should the delayed error be raised? |
it was more of a feasibility question, but could be expressed in primitive python as exceptions = []
for zarr in zarrs:
try:
dowhatever(zarr)
except Exception as exc:
lgr.exception("Delaying raising for exception while processing zarrs")
exceptions.append(exc)
if exceptions:
raise RuntimeError(f"{len(exceptions)} exceptions while dowhatevering zarrs happened. Details were logged") Also WDYT about the question above
|
@yarikoptic Is that code intended as a direct answer to my questions? Catching exceptions raised while processing Zarrs isn't really an option, as the Zarrs are processed concurrently with the rest of the assets in a task group.
Possibly. |
no, it is not the direct answer - it was just formulation in Python. But there is then answers to your questions:
I am not sure what other steps would lead to "conclude" but really it should not be considered "normal course of backup". The effect I wanted is to delay erroring the entire process out until we are done with all zarrs first, where some might "conclude normally" since they would be perfectly ok, and some known to be problematic. And then if there was any problematic zarr -- raise and exception and refer to some prior logs (ideally with tracebacks etc) with details on what was problematic in those zarrs.
e.g. after finishing a loop on zarrs if any zarr is problematic. |
@yarikoptic The only Zarr backup steps that happen after the point where the RuntimeError is currently raised are: updating the checksum file and updating the
There is no "loop on zarrs". The code loops through all assets, starts up asynchronous tasks for any Zarrs, and then after all tasks are done, the results of the Zarr tasks are retrieved. |
I haven't seen those for quite a while although we do get other errors etc! may be indeed #299 resolved them?! let's let it RiP |
continuation of #260 saga. Happened on the recent run for 000108 of
so workers was 4. The log is /mnt/backup/dandi/dandisets/.git/dandi/backups2datalad/2022.11.01.19.31.16Z.log just in case . 000108 was left in
state and I am not sure if any commit was done
I will reset and rerun with --workers 2 now. I might check some recently modified zarrs if all good first
The text was updated successfully, but these errors were encountered: