Cannot delete Linux directory - infinite recursion
We have an NFS mount on a RHEL6 VM that supports our version control server - recently, one of the repositories went a bit crazy and this is what I found on the server:
ls -latri repo.git/refs/heads/
total 28
5551210 drwxr-xr-x. 2 git git 8192 Jun 1 21:21 .
5551210 drwxr-xr-x. 2 git git 8192 Jun 1 21:21
5551210 drwxr-xr-x. 2 git git 8192 Jun 1 21:21
5551209 drwxr-xr-x. 3 git git 4096 Jun 1 22:09 ..
When I run tree against the dir, it appears to be infinitely recursive - e.g.:
repo.git/refs/heads/
├──
│ ├──
│ │ ├──
│ │ │ ├──
│ │ │ │ ├──
│ │ │ │ │ ├──
│ │ │ │ │ │ ├──
│ │ │ │ │ │ │ ├──
│ │ │ │ │ │ │ │ ├──
│ │ │ │ │ │ │ │ │ ├──
│ │ │ │ │ │ │ │ │ │ ├──
│ │ │ │ │ │ │ │ │ │ │ ├──
│ │ │ │ │ │ │ │ │ │ │ │ ├──
│ │ │ │ │ │ │ │ │ │ │ │ │ ├──
│ │ │ │ │ │ │ │ │ │ │ │ │ │ ├──
│ │ │ │ │ │ │ │ │ │ │ │ │ │ │ ├──
│ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ ├──
I have attempted to delete the repository via its inode ref:
[root@node repo.git/refs]# ls -latri
total 16
5551210 drwxr-xr-x. 2 git git 8192 Jun 1 21:21 heads
[root@node repo.git/refs]# find . -inum 5551210 -exec rm -rf {} \;
rm: cannot remove `./refs/heads': Directory not empty
find: `./refs/heads/': No such file or directory
find: `./refs/heads/': No such file or directory
I am a bit at a loss what to do here - the inode info on the ls -latri command seems to indicate that there are 2 directories in the 'heads' directory that are hardlinks to the heads directory?
Any ideas on how to clean this up would be most welcome - think I have solved the application issue it was causing but the bigger issue with the filesystem needs to get sorted.
Thank you!
Edit: bit of additional output:
no hidden characters:
[root@node repo.git/refs]# ls -latrib heads/
total 28
5551210 drwxr-xr-x. 2 git git 8192 Jun 1 21:21 .
5551210 drwxr-xr-x. 2 git git 8192 Jun 1 21:21
5551210 drwxr-xr-x. 2 git git 8192 Jun 1 21:21
5551209 drwxr-xr-x. 3 git git 4096 Jun 1 22:09 ..
but here is some fun output when I'm actually in the heads dir:
[root@node repo.git/refs/heads]# ls -latrib
ls: cannot access : No such file or directory
ls: cannot access : No such file or directory
total 12
? -?????????? ? ? ? ? ?
? -?????????? ? ? ? ? ?
5551210 drwxr-xr-x. 2 git git 8192 Jun 1 21:21 .
5551209 drwxr-xr-x. 3 git git 4096 Jun 1 22:09 ..
Solution 1:
First: Git can neither be the cause nor the solution of a problem that manifests as nonsensical output from ls. Stop using Git or other tools on the filesystem and unmount it to avoid harm.
This looks like either a broken filesystem or a broken mount. Try unmounting and remounting the filesystem on the client. Try fully rebooting the client. Try doing the same mount on another client. Each time, check that ls output to see if it becomes normal. This will help you diagnose whether the problem is on the NFS server side. If the ls output continues to look the same, investigation and repair of the filesystem (fsck or whatever) and/or the NFS service (restarting NFS-related daemoons; reboot if nfsd is in-kernel) needs to take place on the server side.