XenServer General Backend Error while trying to detach an NFS Storage Repository

Ever had this error? Really annoying, and you're stuck basically until the NFS server comes up (if you're lucky).

It's more or less practice for some people (like me) to have their O/S images uploaded to and made available from an EC2 instance so that whenever a new VM is about to be installed, that instance can be turned on and accessed by XenServer as an NFS ISO Library.

Here's the catch though: EC2 instances have dynamic IPs, so if you forget to detach the SR from XenServer before you shut the instance down it becomes inaccessible and therefore XenServer can't detach it gracefully.

Of course that is not the only case where this error can occur. If your NFS server bites the dust, you're stuck with a Storage Repository that just won't die. The workaround is pretty straightforward though.

- SSH to your XenServer
- Find out the UUID of your NFS Storage (xe sr-list name-label=YOUR_NFS_SR_NAME)
- Find out the SR-UUID of the corresponding Physical Block Device (xe pbd-list sr-uuid=YOUR_NFS_STORAGE_UUID)
- Find out the location of the filesystem where the zombie NFS export is mounted on
- Unmount it using the "force" and "lazy" parameters
- Order XenServer to unplug the Physical Block Device through the command line (xe pbd-unplug uuid=UUID_OF_THE_PBD --although now this should be possible from the GUI as well)
- Order XenServer to forget the device altogether (xe sr-forget uuid=YOUR_NFS_STORAGE_UUID --although now this should be possible from the GUI as well)

For our purposes, our XenServer's IP address is 172.16.1.10, the remote NFS share is located at 54.217.159.179:/export and our NFS SR was named "NFS ISO library".

root@linux:~# ssh -C 172.16.1.10   
[root@xenserver ~]# xe sr-list name-label="NFS ISO library" 
uuid ( RO)                : 6e0b7373-a1a3-1edb-1a09-b72a2bebbc7d
          name-label ( RW): NFS ISO library
    name-description ( RW): NFS ISO Library [54.217.159.179:/export]
                host ( RO): xenserver
                type ( RO): iso
        content-type ( RO): iso


[root@xenserver ~]# xe pbd-list sr-uuid=6e0b7373-a1a3-1edb-1a09-b72a2bebbc7d
uuid ( RO)                  : accefe2d-8a22-d87d-23c6-53ab3110fc6b
             host-uuid ( RO): 66807c1f-39c3-418f-8d91-301380a05805
               sr-uuid ( RO): 6e0b7373-a1a3-1edb-1a09-b72a2bebbc7d
         device-config (MRO): type: nfs_iso; location: 54.217.159.179:/export
    currently-attached ( RO): true

[root@xenserver ~]# cat /etc/mtab
/dev/md0 / ext3 rw 0 0
none /proc proc rw 0 0
none /sys sysfs rw 0 0
none /dev/pts devpts rw 0 0
none /dev/shm tmpfs rw 0 0
/opt/xensource/packages/iso/XenCenter.iso /var/xen/xc-install iso9660 \
    ro,loop=/dev/loop0 0 0
none /proc/sys/fs/binfmt_misc binfmt_misc rw 0 0
sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw 0 0
54.217.159.179:/export /var/run/sr-mount/6e0b7373-a1a3-1edb-1a09-b72a2bebbc7d \
    nfs rw,soft,timeo=133,retrans=2147483647,tcp,actimeo=0,addr=54.217.159.179 0 0

[root@xenserver ~]# umount -f -l /var/run/sr-mount/6e0b7373-a1a3-1edb-1a09-b72a2bebbc7d
[root@xenserver ~]# xe pbd-unplug uuid=accefe2d-8a22-d87d-23c6-53ab3110fc6b
[root@xenserver ~]# xe sr-forget uuid=6e0b7373-a1a3-1edb-1a09-b72a2bebbc7d