Tuesday, May 21, 2013

Umount a stalled/frozen NFS mount point

NFS is known to be a little nasty when it comes to umount stalled shares.

Most of the time a simple umount doesn't work, which is a bit frustrating specially when it comes to production servers; The process just hangs and there is no way to interrupt...

Below two procedures to umount stalled  NFS shares. You should try method one before method two as it requires some network "hacks".

Method 1 :

Use a forced lazy umount, this method works 90% of the time :
 # umount -f -l /mnt/nfs_share
Note : Don't use bash auto-completion !!!  

Method 2:

This method is to be used only if method one failed.

The trick is to temporarily steal the NFS server IP adress on the NFS client (the one with stalled mount) so this client thinks that the NFS server is still alive.

Warning : Use method 1 above if your NFS server is still reachable from the NFS client. Otherwise you'll have an IP conflit and trust me you really don't want that to happen.

Let's assume the NFS server IP is
  1. Double check that the NFS server is down with ping or nmap.
  2. If your NFS client has very restrictive IPTables rules shut them down temporarily
  3. On the NFS client, set the NFS server IP as secondary address
  4.  # ifconfig eth0:0  
     Note : Adjust interface to your own needs  
  5. Umount the NFS with a forced lzay umount
  6.  # umount -f -l /mnt/nfs_share  
     Note : Don't use bash auto-completion !!! 
  7.  Check that the NFS mount is gone
  8. Remove secondary interface
     # ifconfig eth0:0 down  
     Note : Adjust interface to your own needs  
  9. Restart IPTables if needed
  10. Be happy
  11. Go to sleep, it's been a long day (or night)
 If you have multiple NFS clients stalled, you can set the secondary IP on one client only.
  • Client 1 : Step 1 to 5
  • Client 2 to n : Step 4 and 5
  • Client 1 : Step 6 to 9

This will only work if your NFS client can communicate between each others (watch for IPTables or any other filtering softwares/devices).

Hope that helps ! (that helped me a lot :)

No comments:

Post a Comment