Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Multiple dirs in --ignore-dirs not working as expected #196

Closed
edrozenberg opened this issue Dec 11, 2023 · 8 comments
Closed

Multiple dirs in --ignore-dirs not working as expected #196

edrozenberg opened this issue Dec 11, 2023 · 8 comments
Labels
bug Something isn't working help wanted Extra attention is needed

Comments

@edrozenberg
Copy link

edrozenberg commented Dec 11, 2023

I'm comparing gdu results (correct) with dua results (incorrect) when excluding multiple dirs on MacOS. I'm excluding these specific dirs in order to avoid the double-counting caused by how Apple has decided to mount, alias, and symlink various items in /System.

My Mac drive is a 1 TB with ~ 330 GB free atm, so gdu is correct showing ~600 GB used, while dua shows ~1.2TB used due to the double counting.

My GDU command:
sudo gdu-go -i /System/Volumes/Data/Volumes,/System/Volumes/Data /

GDU Results - Correct

My DUA command:
sudo dua -i /System/Volumes/Data/Volumes,/System/Volumes/Data interactive /

DUA Results - Incorrect

And reported disk use from df:

Filesystem                       Size   Used  Avail Capacity  Mounted on
/dev/disk1s1s1                  932Gi   21Gi  332Gi     6%    /
devfs                           194Ki  194Ki    0Bi   100%    /dev
/dev/disk1s4                    932Gi  1.0Gi  332Gi     1%    /System/Volumes/VM
/dev/disk1s2                    932Gi  2.1Gi  332Gi     1%    /System/Volumes/Preboot
/dev/disk1s6                    932Gi   70Mi  332Gi     1%    /System/Volumes/Update
/dev/disk1s5                    932Gi  574Gi  332Gi    64%    /System/Volumes/Data
map auto_home                     0Bi    0Bi    0Bi   100%    /System/Volumes/Data/home
/dev/disk1s1                    932Gi   21Gi  332Gi     6%    /System/Volumes/Update/mnt1
/dev/disk3s1                    3.8Gi  3.6Gi  220Mi    95%    /Library/Developer/CoreSimulator/Volumes/watchOS_20S75
@Byron Byron added bug Something isn't working help wanted Extra attention is needed labels Dec 11, 2023
@Byron
Copy link
Owner

Byron commented Dec 11, 2023

Thanks a lot for letting me know!

Indeed, it doesn't seem to work at all and I could reproduce it well. If it's that broken, and was at some point deemed working, maybe a test is missing for it as well.

@edrozenberg
Copy link
Author

I've never tried ignore before with dua so do not know if it ever worked :). Thanks for looking into it!

@gosuwachu
Copy link
Contributor

@edrozenberg This looks to work as expected. To pass multiple ignore directories you have to specify them separately, like this:

dua -i /System/Volumes/Data/Volumes -i /System/Volumes/Data interactive /

@edrozenberg
Copy link
Author

@edrozenberg This looks to work as expected. To pass multiple ignore directories you have to specify them separately, like this:

dua -i /System/Volumes/Data/Volumes -i /System/Volumes/Data interactive /

Thanks but this does not work as expected, still getting an incorrect double-counted result:

sudo dua -i /System/Volumes/Data/Volumes -i /System/Volumes/Data interactive /

dua results

@gosuwachu
Copy link
Contributor

gosuwachu commented Dec 12, 2023

I think I know where this bug is coming from. Try this instead:

sudo dua -i System/Volumes/Data/Volumes -i System/Volumes/Data interactive /

dua doesn't include the starting / in the path which was causing your ignored directories not to match. However on my mac dua still shows incorrect results - but probably for a different reason.

@edrozenberg
Copy link
Author

edrozenberg commented Dec 12, 2023

I think I know where this bug is coming from. Try this instead:

sudo dua -i System/Volumes/Data/Volumes -i System/Volumes/Data interactive /

dua doesn't include the starting / in the path which was causing your ignored directories not to match. However on my mac dua still shows incorrect results - but probably for a different reason.

Thanks this seems to work fine, but it shouldn't because if I don't include the / for each path it should not find them if my current directory is not in the root of the drive. But it does find them somehow. [edit] I see it finds them based on the starting directory given, /.

[~]$ sudo dua -i System/Volumes/Data/Volumes -i System/Volumes/Data interactive /

Somehow works

@Byron
Copy link
Owner

Byron commented Dec 13, 2023

Thanks everyone for chiming in and for helping to solve this puzzle!

I can confirm that sudo dua -i System/Volumes/Data/Volumes -i System/Volumes/Data interactive / works for me.

I think in order to resolve this, dua should properly transform the paths to the format that it needs to make them work for it. That way, -i target/ , -i $PWD/target or -i target/ will all work the same.

Byron added a commit that referenced this issue Jan 3, 2024
Previously they would need to be specified as relative to the traversal root, which
was unintuitive and would lead to ignores not working for many.

Even though this was done for performance to avoid canonicalization, we do now
perform a more performance version of canonicalization so the overall performance
should be acceptable nonetheless.

Also note that ignored directories are now logged when using a `--log-file`.
@Byron
Copy link
Owner

Byron commented Jan 3, 2024

This should be fixed with the latest release, and I wasn't able to reproduce the issue anymore with the new version either.

Please let us know if the issue persists.

@Byron Byron closed this as completed Jan 3, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working help wanted Extra attention is needed
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants