Linux no space left on device issue

We have SUSE Linux OS:

$ uname -a
Linux 3.0.13-0.27-default #1 SMP Wed Feb 15 13:33:49 UTC 2012  x86_64 x86_64 x86_64 GNU/Linux

Only for a specific file the file creation fails with error no space left on device. We are able to create other files, what has gone wrong? In the current scenario:

  • Inode usage is only 47%

    $ df -i .
    Filesystem               Inodes    IUsed    IFree IUse% Mounted on
    /dev/mapper/appvg-dsfs 39321600 18218000 21103600   47% /dsfs
    
  • Disk usage is only 82%

    $ df -h .
    Filesystem              Size  Used Avail Use% Mounted on
    /dev/mapper/appvg-dsfs  591G  457G  105G  82% /dsfs
    
  • File creation of one specific file fails in this case JDE_GL_INCR_STG_201504_181.log.load.058

    $ echo "a" > JDE_GL_INCR_STG_201504_181.log.load.058
    -bash: JDE_GL_INCR_STG_201504_181.log.load.058: No space left on device
    echo "a" > testfile
    
    $ mv JDE_GL_INCR_STG_201504_181.log.load.058 tempfile
    mv: cannot stat `JDE_GL_INCR_STG_201504_181.log.load.058': No such file or directory
    
  • File creation of other files is possible


Maybe you run out of Semaphores. Try this as root

ipcs -s

and check if there are a lot of semaphores owned by the same user. If so run the following command

ipcs -s |grep username |awk '{print $2}'

in the above command replace the username withn the actual user.