Oh dear, df -h is showing that my filesystem is full up but du -sh * is only showing 8G out of 15G is used. Where has the rest gone? The most likely cause is that someone has deleted a file that is still open so the space hasn't been released. But how to find a deleted file?

The following command will find any deleted files that are still open:

 find /proc/*/fd -ls | grep  '(deleted)'

If you have lsof installed, that has some neat options.To find an open deleted file, the following can be used:

 lsof -nP +L1

The -nP options just inhibit the lookup of network names and port names so lsof runs quicker. The +L1 flag will display any files with a link count of less than 1 i.e. the deleted ones.

A really useful doc is the lsof Quickstart file distributed with the lsof package. If you're nor sure where that is, you can find it as follows:

 locate -i quickstart |grep lsof
 /usr/share/doc/lsof-4.82/00QUICKSTART
 head /usr/share/doc/lsof-4.82/00QUICKSTART
 
                      A Quick Start for Lsof
 
 1.  Introduction
 ================
 
Agreed, the lsof man page is dense and lsof has a plethora of
options.  There are examples, but the manual page format buries
them at the end.  How does one get started with lsof?

All the most regularly used lsof options are listed in the brief document.

If you do find that you need to reclaim space from a log file that's opened and the application using it can't be stopped, cat /dev/null > to the log file, e.g.

 
 tail -n200 some.log > some.log.2  # to save the last 200 lines
 cat /dev/null > some.log