Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

QDirStat doesn't scan Btrfs subvolumes #39

Closed
shundhammer opened this issue Dec 6, 2016 · 15 comments
Closed

QDirStat doesn't scan Btrfs subvolumes #39

shundhammer opened this issue Dec 6, 2016 · 15 comments
Labels

Comments

@shundhammer
Copy link
Owner

shundhammer commented Dec 6, 2016

If you use QDirStat to scan a Btrfs partition, any subvolumes of that partition are not scanned: Btrfs subvolumes are treated just like ordinary mount points (which, to all intents and purposes, they are). So you might wonder why the df command shows your 40 GB root filesystem as 97% full, yet QDirStat shows only about 7 GB. The rest might be hidden in subvolumes.

QDirStat stops reading at mount points - which only makes sense because normally you want to know what eats up the disk space on that one partition that is filling up, not on any others like /home that are mounted there. Unfortunately, a Btrfs subvolume is also just another mount point, and QDirStat will start reading there, too - at /var/log, at /var/spool, at /var/lib/libvirt etc.; a typical Btrfs root filesystem has about a dozen subvolumes, and all files in them are currently disregarded by QDirStat.

You can of course click on "Continue reading at mount point" individually in
QDirStat's directory tree for each one of them, but that's tedious.

@shundhammer
Copy link
Owner Author

shundhammer commented Dec 6, 2016

One approach would be to check if the current filesystem is Btrfs and list its subvolumes, but the Btrfs developers in their infinite wisdom decided that btrfs subvolume list <path> is a privileged operation, so QDirStat would have to use sudo with it and prompt for the root password (at which point I as a user would terminate the program and not use it any more).

This is broken by design. (but then, so are many things about Btrfs).
A simple info command like that should not require root privileges.

@shundhammer
Copy link
Owner Author

The stat shell command shows that a Btrfs subvolume has a different major/minor device ID than its parent filesystem.

@shundhammer
Copy link
Owner Author

Question to the world:

How do I find out without root privileges what subvolumes a Btrfs filesystem has?

@shundhammer
Copy link
Owner Author

It looks like a Btrfs subvolume always has i-node ID 256:

http://stackoverflow.com/questions/25908149/how-to-test-if-location-is-a-btrfs-subvolume

Tested on a sample Btrfs. Need to verify that this is reliable - and preferably documented.

@shundhammer
Copy link
Owner Author

The content of /proc/mounts can at least identify filesystems as Btrfs. The problem is just that even the toplevel mount point shows up as a mere subvolume there (which technically it is, but that's pretty irrelevant for the normal use case):

grep -i subvol /proc/mounts | column -t

/dev/vda6  /                        btrfs  rw,relatime,space_cache,subvolid=259,subvol=/@/.snapshots/1/snapshot   0  0
/dev/vda6  /.snapshots              btrfs  rw,relatime,space_cache,subvolid=258,subvol=/@/.snapshots              0  0
/dev/vda6  /var/lib/named           btrfs  rw,relatime,space_cache,subvolid=273,subvol=/@/var/lib/named           0  0
/dev/vda6  /srv                     btrfs  rw,relatime,space_cache,subvolid=263,subvol=/@/srv                     0  0
/dev/vda6  /var/lib/libvirt/images  btrfs  rw,relatime,space_cache,subvolid=268,subvol=/@/var/lib/libvirt/images  0  0
/dev/vda6  /opt                     btrfs  rw,relatime,space_cache,subvolid=262,subvol=/@/opt                     0  0
/dev/vda6  /var/opt                 btrfs  rw,relatime,space_cache,subvolid=276,subvol=/@/var/opt                 0  0
/dev/vda6  /usr/local               btrfs  rw,relatime,space_cache,subvolid=265,subvol=/@/usr/local               0  0
/dev/vda6  /var/lib/mailman         btrfs  rw,relatime,space_cache,subvolid=270,subvol=/@/var/lib/mailman         0  0
/dev/vda6  /var/lib/mariadb         btrfs  rw,relatime,space_cache,subvolid=271,subvol=/@/var/lib/mariadb         0  0
/dev/vda6  /var/crash               btrfs  rw,relatime,space_cache,subvolid=267,subvol=/@/var/crash               0  0
/dev/vda6  /boot/grub2/x86_64-efi   btrfs  rw,relatime,space_cache,subvolid=261,subvol=/@/boot/grub2/x86_64-efi   0  0
/dev/vda6  /boot/grub2/i386-pc      btrfs  rw,relatime,space_cache,subvolid=260,subvol=/@/boot/grub2/i386-pc      0  0
/dev/vda6  /var/lib/mysql           btrfs  rw,relatime,space_cache,subvolid=272,subvol=/@/var/lib/mysql           0  0
/dev/vda6  /var/lib/pgsql           btrfs  rw,relatime,space_cache,subvolid=274,subvol=/@/var/lib/pgsql           0  0
/dev/vda6  /var/log                 btrfs  rw,relatime,space_cache,subvolid=275,subvol=/@/var/log                 0  0
/dev/vda6  /var/tmp                 btrfs  rw,relatime,space_cache,subvolid=278,subvol=/@/var/tmp                 0  0
/dev/vda6  /var/spool               btrfs  rw,relatime,space_cache,subvolid=277,subvol=/@/var/spool               0  0
/dev/vda6  /tmp                     btrfs  rw,relatime,space_cache,subvolid=264,subvol=/@/tmp                     0  0
/dev/vda6  /var/lib/machines        btrfs  rw,relatime,space_cache,subvolid=269,subvol=/@/var/lib/machines        0  0
/dev/vda6  /var/cache               btrfs  rw,relatime,space_cache,subvolid=266,subvol=/@/var/cache               0  0

Now the question is how to tell the difference between a subvolume mount point and the mount point of a full-fledged Btrfs into another Btrfs: /home on a separate partition might be a Btrfs as well, so we couldn't tell the difference between /home and any subvolume of /.

This output shows /dev/vda6 as the device for all of those Btrfs subvolumes, but the output of the stat() syscall or the stat shell command shows another major/minor device ID for each of the subvolumes.

@shundhammer
Copy link
Owner Author

Once I know that I am on one of those Btrfs partitions, I can do string comparisons: Am I still on /dev/vda6, or is another (Btrfs?) partition mounted there, too (say, /dev/vda7)? But how do I find out in the beginning that I am on /dev/vda6? The minor/major device ID returned from lstat() is not the one from /dev/vda6, but from some Btrfs device that does not show up below /dev (or does it? I couldn't find it).

@shundhammer
Copy link
Owner Author

Btrfs:

e76:~ # df /
Filesystem     1K-blocks    Used Available Use% Mounted on
/dev/vda6       40434688 3407732  36598220   9% /
e76:~ # stat /
  File: '/'
  Size: 156       	Blocks: 0          IO Block: 4096   directory
Device: 26h/38d	Inode: 256         Links: 1
Access: (0755/drwxr-xr-x)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2016-12-06 11:43:00.000000000 +0100
Modify: 2016-12-06 11:40:46.700000000 +0100
Change: 2016-12-06 11:40:46.700000000 +0100
 Birth: -
e76:~ # ls -l /dev/vda6
brw-rw---- 1 root disk 253, 6 Dec  6 11:45 /dev/vda6

I.e. stat() returns major/minor device 0x0026, but the underlying partition /dev/vda6 has major/minor device 0xFD06.

XFS:

morgul:~ # df /
Filesystem     1K-blocks     Used Available Use% Mounted on
/dev/sda3       41924608 23577132  18347476  57% /
morgul:~ # stat /
  File: '/'
  Size: 4096      	Blocks: 8          IO Block: 4096   directory
Device: 803h/2051d	Inode: 96          Links: 36
Access: (0755/drwxr-xr-x)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2016-12-06 12:25:23.564113162 +0100
Modify: 2016-11-30 16:47:40.127400702 +0100
Change: 2016-11-30 16:47:40.127400702 +0100
 Birth: -
morgul:~ # ls -l /dev/sda3
brw-rw---- 1 root disk 8, 3 Nov 30 16:47 /dev/sda3

stat() returns 0x0803, and this is the underlying partition's (/dev/sda3) major/minor device ID.

@shundhammer
Copy link
Owner Author

@shundhammer
Copy link
Owner Author

shundhammer commented Dec 6, 2016

There is some more (but not much more) information in /proc/self/mountinfo:

e76:~ # grep btrfs /proc/self/mountinfo | column -t
59   0   0:34  /@/.snapshots/1/snapshot   /                        rw,relatime  shared:1   -  btrfs  /dev/vda6  rw,space_cache,subvolid=259,subvol=/@/.snapshots/1/snapshot
79   59  0:34  /@/.snapshots              /.snapshots              rw,relatime  shared:26  -  btrfs  /dev/vda6  rw,space_cache,subvolid=258,subvol=/@/.snapshots
69   59  0:34  /@/var/lib/named           /var/lib/named           rw,relatime  shared:27  -  btrfs  /dev/vda6  rw,space_cache,subvolid=273,subvol=/@/var/lib/named
71   59  0:34  /@/srv                     /srv                     rw,relatime  shared:28  -  btrfs  /dev/vda6  rw,space_cache,subvolid=263,subvol=/@/srv
73   59  0:34  /@/var/lib/libvirt/images  /var/lib/libvirt/images  rw,relatime  shared:29  -  btrfs  /dev/vda6  rw,space_cache,subvolid=268,subvol=/@/var/lib/libvirt/images
75   59  0:34  /@/opt                     /opt                     rw,relatime  shared:30  -  btrfs  /dev/vda6  rw,space_cache,subvolid=262,subvol=/@/opt
77   59  0:34  /@/var/opt                 /var/opt                 rw,relatime  shared:31  -  btrfs  /dev/vda6  rw,space_cache,subvolid=276,subvol=/@/var/opt
91   59  0:34  /@/usr/local               /usr/local               rw,relatime  shared:32  -  btrfs  /dev/vda6  rw,space_cache,subvolid=265,subvol=/@/usr/local
81   59  0:34  /@/var/lib/mailman         /var/lib/mailman         rw,relatime  shared:33  -  btrfs  /dev/vda6  rw,space_cache,subvolid=270,subvol=/@/var/lib/mailman
83   59  0:34  /@/var/lib/mariadb         /var/lib/mariadb         rw,relatime  shared:34  -  btrfs  /dev/vda6  rw,space_cache,subvolid=271,subvol=/@/var/lib/mariadb
85   59  0:34  /@/var/crash               /var/crash               rw,relatime  shared:35  -  btrfs  /dev/vda6  rw,space_cache,subvolid=267,subvol=/@/var/crash
87   59  0:34  /@/boot/grub2/x86_64-efi   /boot/grub2/x86_64-efi   rw,relatime  shared:36  -  btrfs  /dev/vda6  rw,space_cache,subvolid=261,subvol=/@/boot/grub2/x86_64-efi
89   59  0:34  /@/boot/grub2/i386-pc      /boot/grub2/i386-pc      rw,relatime  shared:37  -  btrfs  /dev/vda6  rw,space_cache,subvolid=260,subvol=/@/boot/grub2/i386-pc
80   59  0:34  /@/var/lib/mysql           /var/lib/mysql           rw,relatime  shared:38  -  btrfs  /dev/vda6  rw,space_cache,subvolid=272,subvol=/@/var/lib/mysql
95   59  0:34  /@/var/lib/pgsql           /var/lib/pgsql           rw,relatime  shared:39  -  btrfs  /dev/vda6  rw,space_cache,subvolid=274,subvol=/@/var/lib/pgsql
97   59  0:34  /@/var/log                 /var/log                 rw,relatime  shared:40  -  btrfs  /dev/vda6  rw,space_cache,subvolid=275,subvol=/@/var/log
99   59  0:34  /@/var/tmp                 /var/tmp                 rw,relatime  shared:41  -  btrfs  /dev/vda6  rw,space_cache,subvolid=278,subvol=/@/var/tmp
101  59  0:34  /@/var/spool               /var/spool               rw,relatime  shared:42  -  btrfs  /dev/vda6  rw,space_cache,subvolid=277,subvol=/@/var/spool
103  59  0:34  /@/tmp                     /tmp                     rw,relatime  shared:43  -  btrfs  /dev/vda6  rw,space_cache,subvolid=264,subvol=/@/tmp
105  59  0:34  /@/var/lib/machines        /var/lib/machines        rw,relatime  shared:44  -  btrfs  /dev/vda6  rw,space_cache,subvolid=269,subvol=/@/var/lib/machines
107  59  0:34  /@/var/cache               /var/cache               rw,relatime  shared:45  -  btrfs  /dev/vda6  rw,space_cache,subvolid=266,subvol=/@/var/cache

e76:~ # l /dev/vda6
brw-rw---- 1 root disk 253, 6 Dec  6 11:45 /dev/vda6

See also https://www.kernel.org/doc/Documentation/filesystems/proc.txt :

3.5 /proc/self/mountinfo - Information about mounts

This file contains lines of the form:

36 35 98:0 /mnt1 /mnt2 rw,noatime master:1 - ext3 /dev/root rw,errors=continue
(1)(2)(3) (4) (5) (6) (7) (8) (9) (10) (11)

(1) mount ID: unique identifier of the mount (may be reused after umount)
(2) parent ID: ID of parent (or of self for the top of the mount tree)
(3) major:minor: value of st_dev for files on filesystem
(4) root: root of the mount within the filesystem
(5) mount point: mount point relative to the process's root
(6) mount options: per mount options
(7) optional fields: zero or more fields of the form "tag[:value]"
(8) separator: marks the end of the optional fields
(9) filesystem type: name of filesystem of the form "type[.subtype]"
(10) mount source: filesystem specific information or "none"
(11) super options: per super block options

Parsers should ignore all unrecognised optional fields. Currently the
possible optional fields are:

shared:X mount is shared in peer group X
master:X mount is slave to peer group X
propagate_from:X mount is slave and receives propagation from peer group X (*)
unbindable mount is unbindable

@shundhammer
Copy link
Owner Author

Now working on a promising approach in branch huha-subvols:

Reading and storing /proc/mounts (or fallback /etc/mtab) and comparing the device names when a mount point is found.

The initial mount point of QDirStat's tree is found by canonicalizing the path (removing symlinks and and ".." parts) and comparing that path to any of the known mount points from /proc/mounts. If there is no corresponding mount point, the last path component is removed for the next lookup etc. up to /.

If no device name can be obtained for either the tree root or for a mount point that was found, the simple major/minor device number comparison has priority, i.e. it is considered a filesystem boundary when they are different. That's what QDirStat (and KDirStat) always did.

But for Btrfs, even though major/minor are different when a subvolume mount point is detected, if the device name is still the same (both directories on, say, /dev/sda2), it is considered the same filesystem, and QDirStat keeps reading.

A nice side effect is that this is not limited to Btrfs: Other filesystem types that have a similar behaviour will benefit from that, too.

Right now the toplevel .snapshots directory is still excluded, but that is by sheer accident: There is a very old exclude rule for that name because some backup tools use it, too.

@shundhammer
Copy link
Owner Author

Tested on an openSUSE Tumbleweed VM with a root Btrfs and a /home Btrfs, and it appears to work great: When started to read /, it doesn't stop at all the subvolumes on that filesystem, but it does stop at all the other mount points, including /home (which is also a Btrfs, so this might have been problematic).

From the log:

shundhammer@d165:/tmp> egrep -i "(device|mount point)" qdirstat-1000.log
2016-12-08 11:03:54.988 [3027] <Info>    DirTree.cpp:98 startReading():  device: /dev/vda2
2016-12-08 11:03:54.999 [3027] <Info>    DirReadJob.cpp:111 crossingFileSystems():  Mount point /.snapshots is still on the same device /dev/vda2
2016-12-08 11:03:54.999 [3027] <Info>    DirReadJob.cpp:111 crossingFileSystems():  Mount point /opt is still on the same device /dev/vda2
2016-12-08 11:03:54.999 [3027] <Info>    DirReadJob.cpp:111 crossingFileSystems():  Mount point /srv is still on the same device /dev/vda2
2016-12-08 11:03:54.999 [3027] <Info>    DirReadJob.cpp:111 crossingFileSystems():  Mount point /tmp is still on the same device /dev/vda2
2016-12-08 11:03:54.999 [3027] <Debug>   DirReadJob.cpp:199 startReading():  Found mount point /home
2016-12-08 11:03:54.999 [3027] <Debug>   DirReadJob.cpp:199 startReading():  Found mount point /dev
2016-12-08 11:03:54.999 [3027] <Debug>   DirReadJob.cpp:199 startReading():  Found mount point /proc
2016-12-08 11:03:54.999 [3027] <Debug>   DirReadJob.cpp:199 startReading():  Found mount point /sys
2016-12-08 11:03:54.999 [3027] <Debug>   DirReadJob.cpp:199 startReading():  Found mount point /run
2016-12-08 11:03:55.067 [3027] <Info>    DirReadJob.cpp:111 crossingFileSystems():  Mount point /usr/local is still on the same device /dev/vda2
2016-12-08 11:03:55.067 [3027] <Info>    DirReadJob.cpp:111 crossingFileSystems():  Mount point /var/cache is still on the same device /dev/vda2
2016-12-08 11:03:55.067 [3027] <Info>    DirReadJob.cpp:111 crossingFileSystems():  Mount point /var/crash is still on the same device /dev/vda2
2016-12-08 11:03:55.067 [3027] <Info>    DirReadJob.cpp:111 crossingFileSystems():  Mount point /var/log is still on the same device /dev/vda2
2016-12-08 11:03:55.067 [3027] <Info>    DirReadJob.cpp:111 crossingFileSystems():  Mount point /var/opt is still on the same device /dev/vda2
2016-12-08 11:03:55.067 [3027] <Info>    DirReadJob.cpp:111 crossingFileSystems():  Mount point /var/spool is still on the same device /dev/vda2
2016-12-08 11:03:55.067 [3027] <Info>    DirReadJob.cpp:111 crossingFileSystems():  Mount point /var/tmp is still on the same device /dev/vda2
2016-12-08 11:03:55.077 [3027] <Info>    DirReadJob.cpp:111 crossingFileSystems():  Mount point /boot/grub2/i386-pc is still on the same device /dev/vda2
2016-12-08 11:03:55.077 [3027] <Info>    DirReadJob.cpp:111 crossingFileSystems():  Mount point /boot/grub2/x86_64-efi is still on the same device /dev/vda2
2016-12-08 11:03:55.100 [3027] <Info>    DirReadJob.cpp:111 crossingFileSystems():  Mount point /var/lib/machines is still on the same device /dev/vda2
2016-12-08 11:03:55.100 [3027] <Info>    DirReadJob.cpp:111 crossingFileSystems():  Mount point /var/lib/mailman is still on the same device /dev/vda2
2016-12-08 11:03:55.100 [3027] <Info>    DirReadJob.cpp:111 crossingFileSystems():  Mount point /var/lib/mariadb is still on the same device /dev/vda2
2016-12-08 11:03:55.100 [3027] <Info>    DirReadJob.cpp:111 crossingFileSystems():  Mount point /var/lib/mysql is still on the same device /dev/vda2
2016-12-08 11:03:55.100 [3027] <Info>    DirReadJob.cpp:111 crossingFileSystems():  Mount point /var/lib/named is still on the same device /dev/vda2
2016-12-08 11:03:55.100 [3027] <Info>    DirReadJob.cpp:111 crossingFileSystems():  Mount point /var/lib/pgsql is still on the same device /dev/vda2
2016-12-08 11:03:55.178 [3027] <Info>    DirReadJob.cpp:111 crossingFileSystems():  Mount point /var/lib/libvirt/images is still on the same device /dev/vda2

Subvolumes on both Btrfs partitions:

shundhammer@d165:/tmp> sudo btrfs subvolume list /
ID 257 gen 32 top level 5 path @
ID 258 gen 524 top level 257 path @/.snapshots
ID 259 gen 533 top level 258 path @/.snapshots/1/snapshot
ID 260 gen 529 top level 257 path @/boot/grub2/i386-pc
ID 261 gen 529 top level 257 path @/boot/grub2/x86_64-efi
ID 262 gen 529 top level 257 path @/opt
ID 263 gen 529 top level 257 path @/srv
ID 264 gen 533 top level 257 path @/tmp
ID 265 gen 529 top level 257 path @/usr/local
ID 266 gen 533 top level 257 path @/var/cache
ID 267 gen 529 top level 257 path @/var/crash
ID 268 gen 529 top level 257 path @/var/lib/libvirt/images
ID 269 gen 34 top level 257 path @/var/lib/machines
ID 270 gen 529 top level 257 path @/var/lib/mailman
ID 271 gen 529 top level 257 path @/var/lib/mariadb
ID 272 gen 529 top level 257 path @/var/lib/mysql
ID 273 gen 529 top level 257 path @/var/lib/named
ID 274 gen 529 top level 257 path @/var/lib/pgsql
ID 275 gen 533 top level 257 path @/var/log
ID 276 gen 529 top level 257 path @/var/opt
ID 277 gen 533 top level 257 path @/var/spool
ID 278 gen 533 top level 257 path @/var/tmp
ID 283 gen 523 top level 258 path @/.snapshots/2/snapshot


shundhammer@d165:/tmp> sudo btrfs subvolume list /home
ID 257 gen 16 top level 5 path @

df output for a (reasonably) good overview of all the mount points:

shundhammer@d165:/tmp> df
Filesystem     1K-blocks    Used Available Use% Mounted on
devtmpfs         1010564       0   1010564   0% /dev
tmpfs            1024132     108   1024024   1% /dev/shm
tmpfs            1024132    1372   1022760   1% /run
tmpfs            1024132       0   1024132   0% /sys/fs/cgroup
/dev/vda2       16346112 3448028  11090660  24% /
/dev/vda2       16346112 3448028  11090660  24% /var/log
/dev/vda2       16346112 3448028  11090660  24% /boot/grub2/i386-pc
/dev/vda2       16346112 3448028  11090660  24% /var/lib/named
/dev/vda2       16346112 3448028  11090660  24% /var/lib/mailman
/dev/vda2       16346112 3448028  11090660  24% /opt
/dev/vda2       16346112 3448028  11090660  24% /var/opt
/dev/vda2       16346112 3448028  11090660  24% /boot/grub2/x86_64-efi
/dev/vda2       16346112 3448028  11090660  24% /srv
/dev/vda2       16346112 3448028  11090660  24% /.snapshots
/dev/vda2       16346112 3448028  11090660  24% /var/lib/mysql
/dev/vda2       16346112 3448028  11090660  24% /var/crash
/dev/vda2       16346112 3448028  11090660  24% /var/lib/pgsql
/dev/vda2       16346112 3448028  11090660  24% /var/lib/mariadb
/dev/vda2       16346112 3448028  11090660  24% /var/lib/machines
/dev/vda2       16346112 3448028  11090660  24% /tmp
/dev/vda2       16346112 3448028  11090660  24% /var/lib/libvirt/images
/dev/vda2       16346112 3448028  11090660  24% /var/spool
/dev/vda2       16346112 3448028  11090660  24% /usr/local
/dev/vda2       16346112 3448028  11090660  24% /var/tmp
/dev/vda2       16346112 3448028  11090660  24% /var/cache
/dev/vda3       23492608   17328  21377520   1% /home
tmpfs             204828      16    204812   1% /run/user/1000

Finally, all the mount points:

shundhammer@d165:/tmp> mount | column -t
sysfs       on  /sys                             type  sysfs            (rw,nosuid,nodev,noexec,relatime)
proc        on  /proc                            type  proc             (rw,nosuid,nodev,noexec,relatime)
devtmpfs    on  /dev                             type  devtmpfs         (rw,nosuid,size=1010564k,nr_inodes=252641,mode=755)
securityfs  on  /sys/kernel/security             type  securityfs       (rw,nosuid,nodev,noexec,relatime)
tmpfs       on  /dev/shm                         type  tmpfs            (rw,nosuid,nodev)
devpts      on  /dev/pts                         type  devpts           (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)
tmpfs       on  /run                             type  tmpfs            (rw,nosuid,nodev,mode=755)
tmpfs       on  /sys/fs/cgroup                   type  tmpfs            (ro,nosuid,nodev,noexec,mode=755)
cgroup      on  /sys/fs/cgroup/systemd           type  cgroup           (rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd)
pstore      on  /sys/fs/pstore                   type  pstore           (rw,nosuid,nodev,noexec,relatime)
cgroup      on  /sys/fs/cgroup/devices           type  cgroup           (rw,nosuid,nodev,noexec,relatime,devices)
cgroup      on  /sys/fs/cgroup/net_cls,net_prio  type  cgroup           (rw,nosuid,nodev,noexec,relatime,net_cls,net_prio)
cgroup      on  /sys/fs/cgroup/cpu,cpuacct       type  cgroup           (rw,nosuid,nodev,noexec,relatime,cpu,cpuacct)
cgroup      on  /sys/fs/cgroup/pids              type  cgroup           (rw,nosuid,nodev,noexec,relatime,pids)
cgroup      on  /sys/fs/cgroup/blkio             type  cgroup           (rw,nosuid,nodev,noexec,relatime,blkio)
cgroup      on  /sys/fs/cgroup/cpuset            type  cgroup           (rw,nosuid,nodev,noexec,relatime,cpuset)
cgroup      on  /sys/fs/cgroup/freezer           type  cgroup           (rw,nosuid,nodev,noexec,relatime,freezer)
cgroup      on  /sys/fs/cgroup/hugetlb           type  cgroup           (rw,nosuid,nodev,noexec,relatime,hugetlb)
cgroup      on  /sys/fs/cgroup/perf_event        type  cgroup           (rw,nosuid,nodev,noexec,relatime,perf_event)
cgroup      on  /sys/fs/cgroup/memory            type  cgroup           (rw,nosuid,nodev,noexec,relatime,memory)
/dev/vda2   on  /                                type  btrfs            (rw,relatime,space_cache,subvolid=259,subvol=/@/.snapshots/1/snapshot)
systemd-1   on  /proc/sys/fs/binfmt_misc         type  autofs           (rw,relatime,fd=22,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=13138)
hugetlbfs   on  /dev/hugepages                   type  hugetlbfs        (rw,relatime)
mqueue      on  /dev/mqueue                      type  mqueue           (rw,relatime)
debugfs     on  /sys/kernel/debug                type  debugfs          (rw,relatime)
/dev/vda2   on  /var/log                         type  btrfs            (rw,relatime,space_cache,subvolid=275,subvol=/@/var/log)
/dev/vda2   on  /boot/grub2/i386-pc              type  btrfs            (rw,relatime,space_cache,subvolid=260,subvol=/@/boot/grub2/i386-pc)
/dev/vda2   on  /var/lib/named                   type  btrfs            (rw,relatime,space_cache,subvolid=273,subvol=/@/var/lib/named)
/dev/vda2   on  /var/lib/mailman                 type  btrfs            (rw,relatime,space_cache,subvolid=270,subvol=/@/var/lib/mailman)
/dev/vda2   on  /opt                             type  btrfs            (rw,relatime,space_cache,subvolid=262,subvol=/@/opt)
/dev/vda2   on  /var/opt                         type  btrfs            (rw,relatime,space_cache,subvolid=276,subvol=/@/var/opt)
/dev/vda2   on  /boot/grub2/x86_64-efi           type  btrfs            (rw,relatime,space_cache,subvolid=261,subvol=/@/boot/grub2/x86_64-efi)
/dev/vda2   on  /srv                             type  btrfs            (rw,relatime,space_cache,subvolid=263,subvol=/@/srv)
/dev/vda2   on  /.snapshots                      type  btrfs            (rw,relatime,space_cache,subvolid=258,subvol=/@/.snapshots)
/dev/vda2   on  /var/lib/mysql                   type  btrfs            (rw,relatime,space_cache,subvolid=272,subvol=/@/var/lib/mysql)
/dev/vda2   on  /var/crash                       type  btrfs            (rw,relatime,space_cache,subvolid=267,subvol=/@/var/crash)
/dev/vda2   on  /var/lib/pgsql                   type  btrfs            (rw,relatime,space_cache,subvolid=274,subvol=/@/var/lib/pgsql)
/dev/vda2   on  /var/lib/mariadb                 type  btrfs            (rw,relatime,space_cache,subvolid=271,subvol=/@/var/lib/mariadb)
/dev/vda2   on  /var/lib/machines                type  btrfs            (rw,relatime,space_cache,subvolid=269,subvol=/@/var/lib/machines)
/dev/vda2   on  /tmp                             type  btrfs            (rw,relatime,space_cache,subvolid=264,subvol=/@/tmp)
/dev/vda2   on  /var/lib/libvirt/images          type  btrfs            (rw,relatime,space_cache,subvolid=268,subvol=/@/var/lib/libvirt/images)
/dev/vda2   on  /var/spool                       type  btrfs            (rw,relatime,space_cache,subvolid=277,subvol=/@/var/spool)
/dev/vda2   on  /usr/local                       type  btrfs            (rw,relatime,space_cache,subvolid=265,subvol=/@/usr/local)
/dev/vda2   on  /var/tmp                         type  btrfs            (rw,relatime,space_cache,subvolid=278,subvol=/@/var/tmp)
/dev/vda2   on  /var/cache                       type  btrfs            (rw,relatime,space_cache,subvolid=266,subvol=/@/var/cache)
/dev/vda3   on  /home                            type  btrfs            (rw,relatime,space_cache,subvolid=257,subvol=/@)
tmpfs       on  /run/user/1000                   type  tmpfs            (rw,nosuid,nodev,relatime,size=204828k,mode=700,uid=1000,gid=100)
fusectl     on  /sys/fs/fuse/connections         type  fusectl          (rw,relatime)
gvfsd-fuse  on  /run/user/1000/gvfs              type  fuse.gvfsd-fuse  (rw,nosuid,nodev,relatime,user_id=1000,group_id=100)
tracefs     on  /sys/kernel/debug/tracing        type  tracefs          (rw,relatime)

@shundhammer
Copy link
Owner Author

This concludes the C++ part of this issue. The Perl part in qdirstat-cache-writer still remains to be done; it's equally important there.

@shundhammer
Copy link
Owner Author

Also tested successfully on a setup with Btrfs on LVM:

shundhammer@g138:/tmp> egrep -i "(device|mount point)" qdirstat-0.log 
2016-12-08 13:33:01.889 [27685] <Info>    DirTree.cpp:98 startReading():  device: /dev/mapper/system-root
2016-12-08 13:33:01.905 [27685] <Info>    DirReadJob.cpp:111 crossingFileSystems():  Mount point /.snapshots is still on the same device /dev/mapper/system-root
2016-12-08 13:33:01.905 [27685] <Info>    DirReadJob.cpp:111 crossingFileSystems():  Mount point /opt is still on the same device /dev/mapper/system-root
2016-12-08 13:33:01.905 [27685] <Info>    DirReadJob.cpp:111 crossingFileSystems():  Mount point /srv is still on the same device /dev/mapper/system-root
2016-12-08 13:33:01.905 [27685] <Info>    DirReadJob.cpp:111 crossingFileSystems():  Mount point /tmp is still on the same device /dev/mapper/system-root
2016-12-08 13:33:01.905 [27685] <Debug>   DirReadJob.cpp:199 startReading():  Found mount point /home
2016-12-08 13:33:01.905 [27685] <Debug>   DirReadJob.cpp:199 startReading():  Found mount point /dev
2016-12-08 13:33:01.905 [27685] <Debug>   DirReadJob.cpp:199 startReading():  Found mount point /proc
2016-12-08 13:33:01.905 [27685] <Debug>   DirReadJob.cpp:199 startReading():  Found mount point /sys
2016-12-08 13:33:01.905 [27685] <Debug>   DirReadJob.cpp:199 startReading():  Found mount point /run
2016-12-08 13:33:01.987 [27685] <Info>    DirReadJob.cpp:111 crossingFileSystems():  Mount point /usr/local is still on the same device /dev/mapper/system-root
2016-12-08 13:33:01.988 [27685] <Info>    DirReadJob.cpp:111 crossingFileSystems():  Mount point /var/cache is still on the same device /dev/mapper/system-root
2016-12-08 13:33:01.988 [27685] <Info>    DirReadJob.cpp:111 crossingFileSystems():  Mount point /var/crash is still on the same device /dev/mapper/system-root
2016-12-08 13:33:01.988 [27685] <Info>    DirReadJob.cpp:111 crossingFileSystems():  Mount point /var/log is still on the same device /dev/mapper/system-root
2016-12-08 13:33:01.988 [27685] <Info>    DirReadJob.cpp:111 crossingFileSystems():  Mount point /var/opt is still on the same device /dev/mapper/system-root
2016-12-08 13:33:01.988 [27685] <Info>    DirReadJob.cpp:111 crossingFileSystems():  Mount point /var/spool is still on the same device /dev/mapper/system-root
2016-12-08 13:33:01.988 [27685] <Info>    DirReadJob.cpp:111 crossingFileSystems():  Mount point /var/tmp is still on the same device /dev/mapper/system-root
2016-12-08 13:33:01.996 [27685] <Debug>   DirReadJob.cpp:199 startReading():  Found mount point /.snapshots/1/snapshot
2016-12-08 13:33:01.996 [27685] <Debug>   DirReadJob.cpp:199 startReading():  Found mount point /.snapshots/2/snapshot
2016-12-08 13:33:01.996 [27685] <Debug>   DirReadJob.cpp:199 startReading():  Found mount point /.snapshots/3/snapshot
2016-12-08 13:33:01.996 [27685] <Debug>   DirReadJob.cpp:199 startReading():  Found mount point /.snapshots/4/snapshot
2016-12-08 13:33:01.996 [27685] <Debug>   DirReadJob.cpp:199 startReading():  Found mount point /.snapshots/5/snapshot
2016-12-08 13:33:01.997 [27685] <Debug>   DirReadJob.cpp:199 startReading():  Found mount point /.snapshots/6/snapshot
2016-12-08 13:33:01.997 [27685] <Debug>   DirReadJob.cpp:199 startReading():  Found mount point /.snapshots/7/snapshot
2016-12-08 13:33:01.997 [27685] <Debug>   DirReadJob.cpp:199 startReading():  Found mount point /.snapshots/8/snapshot
2016-12-08 13:33:01.997 [27685] <Info>    DirReadJob.cpp:111 crossingFileSystems():  Mount point /boot/grub2/i386-pc is still on the same device /dev/mapper/system-root
2016-12-08 13:33:01.997 [27685] <Info>    DirReadJob.cpp:111 crossingFileSystems():  Mount point /boot/grub2/x86_64-efi is still on the same device /dev/mapper/system-root
2016-12-08 13:33:02.017 [27685] <Info>    DirReadJob.cpp:111 crossingFileSystems():  Mount point /var/lib/machines is still on the same device /dev/mapper/system-root
2016-12-08 13:33:02.017 [27685] <Info>    DirReadJob.cpp:111 crossingFileSystems():  Mount point /var/lib/mailman is still on the same device /dev/mapper/system-root
2016-12-08 13:33:02.017 [27685] <Info>    DirReadJob.cpp:111 crossingFileSystems():  Mount point /var/lib/mariadb is still on the same device /dev/mapper/system-root
2016-12-08 13:33:02.017 [27685] <Info>    DirReadJob.cpp:111 crossingFileSystems():  Mount point /var/lib/mysql is still on the same device /dev/mapper/system-root
2016-12-08 13:33:02.017 [27685] <Info>    DirReadJob.cpp:111 crossingFileSystems():  Mount point /var/lib/named is still on the same device /dev/mapper/system-root
2016-12-08 13:33:02.017 [27685] <Info>    DirReadJob.cpp:111 crossingFileSystems():  Mount point /var/lib/pgsql is still on the same device /dev/mapper/system-root
2016-12-08 13:33:02.070 [27685] <Info>    DirReadJob.cpp:111 crossingFileSystems():  Mount point /var/lib/libvirt/images is still on the same device /dev/mapper/system-root

df output:

shundhammer@g138:/tmp> df -h
Filesystem               Size  Used Avail Use% Mounted on
devtmpfs                 986M     0  986M   0% /dev
tmpfs                   1001M  100K 1001M   1% /dev/shm
tmpfs                   1001M  1.4M  999M   1% /run
tmpfs                   1001M     0 1001M   0% /sys/fs/cgroup
/dev/mapper/system-root   16G  3.6G   12G  25% /
/dev/mapper/system-root   16G  3.6G   12G  25% /var/opt
/dev/mapper/system-root   16G  3.6G   12G  25% /var/lib/named
/dev/mapper/system-root   16G  3.6G   12G  25% /var/cache
/dev/mapper/system-root   16G  3.6G   12G  25% /var/lib/machines
/dev/mapper/system-root   16G  3.6G   12G  25% /usr/local
/dev/mapper/system-root   16G  3.6G   12G  25% /boot/grub2/i386-pc
/dev/mapper/system-root   16G  3.6G   12G  25% /var/crash
/dev/mapper/system-root   16G  3.6G   12G  25% /var/lib/mariadb
/dev/mapper/system-root   16G  3.6G   12G  25% /boot/grub2/x86_64-efi
/dev/mapper/system-root   16G  3.6G   12G  25% /var/lib/pgsql
/dev/mapper/system-root   16G  3.6G   12G  25% /var/spool
/dev/mapper/system-root   16G  3.6G   12G  25% /var/lib/mysql
/dev/mapper/system-root   16G  3.6G   12G  25% /opt
/dev/mapper/system-root   16G  3.6G   12G  25% /.snapshots
/dev/mapper/system-root   16G  3.6G   12G  25% /tmp
/dev/mapper/system-root   16G  3.6G   12G  25% /var/tmp
/dev/mapper/system-root   16G  3.6G   12G  25% /var/lib/mailman
/dev/mapper/system-root   16G  3.6G   12G  25% /var/lib/libvirt/images
/dev/mapper/system-root   16G  3.6G   12G  25% /srv
/dev/mapper/system-root   16G  3.6G   12G  25% /var/log
/dev/mapper/system-home   23G   52M   23G   1% /home
tmpfs                    201M   16K  201M   1% /run/user/1000

@shundhammer
Copy link
Owner Author

The Btrfs snapshots are conveniently filtered out, but that seems more like a byproduct of a Btrfs bug to me:

g138:/tmp # grep crossing qdirstat-0.log 
2016-12-08 14:36:26.525 [2664] <Info>    DirReadJob.cpp:118 crossingFileSystems():  Mount point /.snapshots is still on the same device /dev/mapper/system-root
2016-12-08 14:36:26.525 [2664] <Info>    DirReadJob.cpp:118 crossingFileSystems():  Mount point /opt is still on the same device /dev/mapper/system-root
2016-12-08 14:36:26.525 [2664] <Info>    DirReadJob.cpp:118 crossingFileSystems():  Mount point /srv is still on the same device /dev/mapper/system-root
2016-12-08 14:36:26.525 [2664] <Info>    DirReadJob.cpp:118 crossingFileSystems():  Mount point /tmp is still on the same device /dev/mapper/system-root
2016-12-08 14:36:26.525 [2664] <Info>    DirReadJob.cpp:114 crossingFileSystems():  Found mount point /home on device /dev/mapper/system-home
2016-12-08 14:36:26.525 [2664] <Info>    DirReadJob.cpp:114 crossingFileSystems():  Found mount point /dev on device devtmpfs
2016-12-08 14:36:26.525 [2664] <Info>    DirReadJob.cpp:114 crossingFileSystems():  Found mount point /proc on device proc
2016-12-08 14:36:26.525 [2664] <Info>    DirReadJob.cpp:114 crossingFileSystems():  Found mount point /sys on device sysfs
2016-12-08 14:36:26.525 [2664] <Info>    DirReadJob.cpp:114 crossingFileSystems():  Found mount point /run on device tmpfs
2016-12-08 14:36:26.616 [2664] <Info>    DirReadJob.cpp:118 crossingFileSystems():  Mount point /usr/local is still on the same device /dev/mapper/system-root
2016-12-08 14:36:26.616 [2664] <Info>    DirReadJob.cpp:118 crossingFileSystems():  Mount point /var/cache is still on the same device /dev/mapper/system-root
2016-12-08 14:36:26.616 [2664] <Info>    DirReadJob.cpp:118 crossingFileSystems():  Mount point /var/crash is still on the same device /dev/mapper/system-root
2016-12-08 14:36:26.616 [2664] <Info>    DirReadJob.cpp:118 crossingFileSystems():  Mount point /var/log is still on the same device /dev/mapper/system-root
2016-12-08 14:36:26.616 [2664] <Info>    DirReadJob.cpp:118 crossingFileSystems():  Mount point /var/opt is still on the same device /dev/mapper/system-root
2016-12-08 14:36:26.616 [2664] <Info>    DirReadJob.cpp:118 crossingFileSystems():  Mount point /var/spool is still on the same device /dev/mapper/system-root
2016-12-08 14:36:26.616 [2664] <Info>    DirReadJob.cpp:118 crossingFileSystems():  Mount point /var/tmp is still on the same device /dev/mapper/system-root
2016-12-08 14:36:26.625 [2664] <Info>    DirReadJob.cpp:112 crossingFileSystems():  Found mount point /.snapshots/1/snapshot
2016-12-08 14:36:26.625 [2664] <Info>    DirReadJob.cpp:112 crossingFileSystems():  Found mount point /.snapshots/2/snapshot
2016-12-08 14:36:26.625 [2664] <Info>    DirReadJob.cpp:112 crossingFileSystems():  Found mount point /.snapshots/3/snapshot
2016-12-08 14:36:26.625 [2664] <Info>    DirReadJob.cpp:112 crossingFileSystems():  Found mount point /.snapshots/4/snapshot
2016-12-08 14:36:26.625 [2664] <Info>    DirReadJob.cpp:112 crossingFileSystems():  Found mount point /.snapshots/5/snapshot
2016-12-08 14:36:26.625 [2664] <Info>    DirReadJob.cpp:112 crossingFileSystems():  Found mount point /.snapshots/6/snapshot
2016-12-08 14:36:26.625 [2664] <Info>    DirReadJob.cpp:112 crossingFileSystems():  Found mount point /.snapshots/7/snapshot
2016-12-08 14:36:26.625 [2664] <Info>    DirReadJob.cpp:112 crossingFileSystems():  Found mount point /.snapshots/8/snapshot
2016-12-08 14:36:26.625 [2664] <Info>    DirReadJob.cpp:118 crossingFileSystems():  Mount point /boot/grub2/i386-pc is still on the same device /dev/mapper/system-root
2016-12-08 14:36:26.625 [2664] <Info>    DirReadJob.cpp:118 crossingFileSystems():  Mount point /boot/grub2/x86_64-efi is still on the same device /dev/mapper/system-root
2016-12-08 14:36:26.646 [2664] <Info>    DirReadJob.cpp:118 crossingFileSystems():  Mount point /var/lib/machines is still on the same device /dev/mapper/system-root
2016-12-08 14:36:26.646 [2664] <Info>    DirReadJob.cpp:118 crossingFileSystems():  Mount point /var/lib/mailman is still on the same device /dev/mapper/system-root
2016-12-08 14:36:26.646 [2664] <Info>    DirReadJob.cpp:118 crossingFileSystems():  Mount point /var/lib/mariadb is still on the same device /dev/mapper/system-root
2016-12-08 14:36:26.646 [2664] <Info>    DirReadJob.cpp:118 crossingFileSystems():  Mount point /var/lib/mysql is still on the same device /dev/mapper/system-root
2016-12-08 14:36:26.646 [2664] <Info>    DirReadJob.cpp:118 crossingFileSystems():  Mount point /var/lib/named is still on the same device /dev/mapper/system-root
2016-12-08 14:36:26.646 [2664] <Info>    DirReadJob.cpp:118 crossingFileSystems():  Mount point /var/lib/pgsql is still on the same device /dev/mapper/system-root
2016-12-08 14:36:26.694 [2664] <Info>    DirReadJob.cpp:118 crossingFileSystems():  Mount point /var/lib/libvirt/images is still on the same device /dev/mapper/system-root

They don't show up in /proc/mounts or /etc/mtab or in the output of df:

g138:/tmp # grep -i snap /proc/mounts 
/dev/mapper/system-root / btrfs rw,relatime,space_cache,subvolid=259,subvol=/@/.snapshots/1/snapshot 0 0
/dev/mapper/system-root /.snapshots btrfs rw,relatime,space_cache,subvolid=258,subvol=/@/.snapshots 0 0

g138:/tmp # grep -i snap /etc/mtab
/dev/mapper/system-root / btrfs rw,relatime,space_cache,subvolid=259,subvol=/@/.snapshots/1/snapshot 0 0
/dev/mapper/system-root /.snapshots btrfs rw,relatime,space_cache,subvolid=258,subvol=/@/.snapshots 0 0

g138:/tmp # df -h | grep -i snap
/dev/mapper/system-root   16G  3.8G   12G  25% /.snapshots

And forcing a df on them returns a device name -:

g138:/tmp # df -h /.snapshots/*/snapshot
Filesystem      Size  Used Avail Use% Mounted on
-                16G  3.8G   12G  25% /.snapshots/1/snapshot
-                16G  3.8G   12G  25% /.snapshots/2/snapshot
-                16G  3.8G   12G  25% /.snapshots/3/snapshot
-                16G  3.8G   12G  25% /.snapshots/4/snapshot
-                16G  3.8G   12G  25% /.snapshots/5/snapshot
-                16G  3.8G   12G  25% /.snapshots/6/snapshot
-                16G  3.8G   12G  25% /.snapshots/7/snapshot
-                16G  3.8G   12G  25% /.snapshots/8/snapshot

This is convenient right now because adding snapshots to the directory sums would count every file that is also in a snapshot several times.

If a user still wants to scan a snapshot, he can simply use the normal "Continue reading at mount point" function from the context menu, just like with a normal mount point.

@shundhammer
Copy link
Owner Author

The qdirstat-cache-writer script now also checks the device names of a mount point and its parent directory, not only their major/minor device numbers; so now it will not stop at Btrfs subvolumes while scanning.

That script uses a more simplistic approach than QDirStat itself: It invokes the df command with that path and parses its output. If the path contains very weird special characters, this may fail, in which case that directory (which at that point is already known to have a different device major/minor number than its parent) is considered a filesystem boundary, and that branch is not scanned.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

1 participant