Very rarely and often by mistake, you can end up with a deleted file that is still used by some process, which is likely writing to it, so the used space cannot be freed.
The running process is still holding a reference to it, so the used disk space cannot be freed. You can kill this process right away or truncate the file to reclaim used disk space.
It will not fix the source of the problem. It is not safe but can give you enough time to properly fix this issue.
I will describe this specific case on Debian Squeeze, but the whole process is still relevant today.
$ lsb_release -a
No LSB modules are available. Distributor ID: Debian Description: Debian GNU/Linux 6.0.10 (squeeze) Release: 6.0.10 Codename: squeeze
Notice that glusterfs process is still writing to the deleted log file.
$ sudo lsof -p 7447
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME glusterfs 7447 root cwd DIR 8,1 4096 2 / glusterfs 7447 root rtd DIR 8,1 4096 2 / glusterfs 7447 root txt REG 8,1 57792 16030 /usr/sbin/glusterfsd glusterfs 7447 root mem REG 8,1 90504 15601 /lib/libgcc_s.so.1 glusterfs 7447 root mem REG 8,1 92640 127616 /usr/lib/glusterfs/3.2.7/xlator/debug/io-stats.so.0.0.0 glusterfs 7447 root mem REG 8,1 121472 127590 /usr/lib/glusterfs/3.2.7/xlator/performance/stat-prefetch.so.0.0.0 glusterfs 7447 root mem REG 8,1 86960 127586 /usr/lib/glusterfs/3.2.7/xlator/performance/quick-read.so.0.0.0 glusterfs 7447 root mem REG 8,1 71744 127588 /usr/lib/glusterfs/3.2.7/xlator/performance/io-cache.so.0.0.0 glusterfs 7447 root mem REG 8,1 45416 127577 /usr/lib/glusterfs/3.2.7/xlator/performance/read-ahead.so.0.0.0 glusterfs 7447 root mem REG 8,1 71752 127575 /usr/lib/glusterfs/3.2.7/xlator/performance/write-behind.so.0.0.0 glusterfs 7447 root mem REG 8,1 393760 127544 /usr/lib/glusterfs/3.2.7/xlator/cluster/afr.so.0.0.0 glusterfs 7447 root mem REG 8,1 234976 127571 /usr/lib/glusterfs/3.2.7/xlator/protocol/client.so.0.0.0 glusterfs 7447 root mem REG 8,1 80712 30248 /lib/libresolv-2.11.3.so glusterfs 7447 root mem REG 8,1 22928 34763 /lib/libnss_dns-2.11.3.so glusterfs 7447 root mem REG 8,1 51728 34761 /lib/libnss_files-2.11.3.so glusterfs 7447 root mem REG 8,1 76936 127535 /usr/lib/glusterfs/3.2.7/rpc-transport/socket.so.0.0.0 glusterfs 7447 root mem REG 8,1 142168 127592 /usr/lib/glusterfs/3.2.7/xlator/mount/fuse.so.0.0.0 glusterfs 7447 root mem REG 8,1 1478056 34754 /lib/libc-2.11.3.so glusterfs 7447 root mem REG 8,1 131261 34764 /lib/libpthread-2.11.3.so glusterfs 7447 root mem REG 8,1 14696 34760 /lib/libdl-2.11.3.so glusterfs 7447 root mem REG 8,1 91952 17982 /usr/lib/libgfxdr.so.0.0.0 glusterfs 7447 root mem REG 8,1 88584 17739 /usr/lib/libgfrpc.so.0.0.0 glusterfs 7447 root mem REG 8,1 400504 17697 /usr/lib/libglusterfs.so.0.0.0 glusterfs 7447 root mem REG 8,1 128744 34748 /lib/ld-2.11.3.so glusterfs 7447 root 0u CHR 1,3 0t0 2045 /dev/null glusterfs 7447 root 1u CHR 1,3 0t0 2045 /dev/null glusterfs 7447 root 2u CHR 1,3 0t0 2045 /dev/null glusterfs 7447 root 3u 0000 0,9 0 2042 anon_inode glusterfs 7447 root 4w REG 8,1 993079299 127832 /var/log/glusterfs/mnt-backup.log.1.old (deleted) glusterfs 7447 root 5u CHR 10,229 0t0 7324 /dev/fuse glusterfs 7447 root 6u IPv4 1112791139 0t0 TCP gfs-a.local:1022->gfs-a.local:24007 (ESTABLISHED) glusterfs 7447 root 7u IPv4 1095043971 0t0 TCP gfs-a.local:1023->gfs-a.local:24009 (ESTABLISHED) glusterfs 7447 root 8u REG 8,1 0 16722 /tmp/tmpf6nhOjW (deleted) glusterfs 7447 root 9u IPv4 1235641780 0t0 TCP gfs-a.local:1020->gfs-b.local:24009 (ESTABLISHED) glusterfs 7447 root 10r CHR 1,9 0t0 2050 /dev/urandom
Filter the data to display only open files that have been unlinked for a specific process.
$ sudo lsof -a +L1 -p 7447
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NLINK NODE NAME glusterfs 7447 root 4w REG 8,1 993079299 0 127832 /var/log/glusterfs/mnt-backup.log.1.old (deleted) glusterfs 7447 root 8u REG 8,1 0 0 16722 /tmp/tmpf6nhOjW (deleted)
The log file cannot be truncated directly as it is not available in the filesystem. Still, you can access it by using the file descriptor opened by this particular process.
$ sudo ls -l /proc/7447/fd
total 0 lrwx------ 1 root root 64 Nov 21 11:02 0 -> /dev/null lrwx------ 1 root root 64 Nov 21 11:02 1 -> /dev/null lr-x------ 1 root root 64 Nov 21 11:02 10 -> /dev/urandom lrwx------ 1 root root 64 Nov 21 11:02 2 -> /dev/null lrwx------ 1 root root 64 Nov 21 11:02 3 -> anon_inode:[eventpoll] l-wx------ 1 root root 64 Nov 21 11:02 4 -> /var/log/glusterfs/mnt-backup.log.1.old (deleted) lrwx------ 1 root root 64 Nov 21 11:02 5 -> /dev/fuse lrwx------ 1 root root 64 Nov 21 11:02 6 -> socket:[1112791139] lrwx------ 1 root root 64 Nov 21 11:02 7 -> socket:[1095043971] lrwx------ 1 root root 64 Nov 21 11:02 8 -> /tmp/tmpf6nhOjW (deleted) lrwx------ 1 root root 64 Nov 21 11:02 9 -> socket:[1235641780]
Display file descriptors pointing to deleted files.
$ sudo find /proc/7447/fd -ilname "*(deleted)"
/proc/7447/fd/4 /proc/7447/fd/8
Display file descriptors pointing to deleted files in a more useful way.
$ sudo find /proc/7447/fd -ilname "*(deleted)" -printf "%h/%f -> %l\n"
/proc/7447/fd/4 -> /var/log/glusterfs/mnt-backup.log.1.old (deleted) /proc/7447/fd/8 -> /tmp/tmpf6nhOjW (deleted)
Display file descriptor pointing to the particular deleted file.
$ sudo find /proc/7447/fd -lname "*mnt-backup.log.1.old (deleted)"
/proc/7447/fd/4
Truncate this file descriptor.
$ sudo truncate /proc/7447/fd/4 --size 0
Filter the data to display only open files that have been unlinked for a specific process.
$ sudo lsof -a +L1 -p 7447
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NLINK NODE NAME glusterfs 7447 root 4w REG 8,1 729 0 127832 /var/log/glusterfs/mnt-backup.log.1.old (deleted) glusterfs 7447 root 8u REG 8,1 0 0 16722 /tmp/tmpf6nhOjW (deleted)
This will give you some time.