Removing vg and lv after physical drive has been removed

We had a disk fail in a server and replaced it before removing the drive from LVM.

The server has 4 physical drives (PV's), each with it's own volume group (VG). Each VG has 2 or more logical volumes (LV's.) Now LVM is complaining about the missing drive. So we have a VG (vg04) with two LV's that have become orphans than we need to clear out of the system.

The problem is every time we run any LVM command we get these 'read failed' errors:

# lvscan
  /dev/vg04/swap: read failed after 0 of 4096 at 4294901760: Input/output error
  /dev/vg04/swap: read failed after 0 of 4096 at 4294959104: Input/output error
  /dev/vg04/swap: read failed after 0 of 4096 at 0: Input/output error
  /dev/vg04/swap: read failed after 0 of 4096 at 4096: Input/output error
  /dev/vg04/vz: read failed after 0 of 4096 at 995903864832: Input/output error
  /dev/vg04/vz: read failed after 0 of 4096 at 995903922176: Input/output error
  /dev/vg04/vz: read failed after 0 of 4096 at 0: Input/output error
  /dev/vg04/vz: read failed after 0 of 4096 at 4096: Input/output error

# vgreduce vg04 --removemissing --force
  /dev/vg04/swap: read failed after 0 of 4096 at 4294901760: Input/output error
  /dev/vg04/swap: read failed after 0 of 4096 at 4294959104: Input/output error
  /dev/vg04/swap: read failed after 0 of 4096 at 0: Input/output error
  /dev/vg04/swap: read failed after 0 of 4096 at 4096: Input/output error
  /dev/vg04/vz: read failed after 0 of 4096 at 995903864832: Input/output error
  /dev/vg04/vz: read failed after 0 of 4096 at 995903922176: Input/output error
  /dev/vg04/vz: read failed after 0 of 4096 at 0: Input/output error
  /dev/vg04/vz: read failed after 0 of 4096 at 4096: Input/output error
  Volume group "vg04" not found

# vgchange -a n /dev/vg04
  /dev/vg04/swap: read failed after 0 of 4096 at 4294901760: Input/output error
  /dev/vg04/swap: read failed after 0 of 4096 at 4294959104: Input/output error
  /dev/vg04/swap: read failed after 0 of 4096 at 0: Input/output error
  /dev/vg04/swap: read failed after 0 of 4096 at 4096: Input/output error
  /dev/vg04/vz: read failed after 0 of 4096 at 995903864832: Input/output error
  /dev/vg04/vz: read failed after 0 of 4096 at 995903922176: Input/output error
  /dev/vg04/vz: read failed after 0 of 4096 at 0: Input/output error
  /dev/vg04/vz: read failed after 0 of 4096 at 4096: Input/output error
  Volume group "vg04" not found

# lvchange -a n /dev/vg04/vz
  /dev/vg04/swap: read failed after 0 of 4096 at 4294901760: Input/output error
  /dev/vg04/swap: read failed after 0 of 4096 at 4294959104: Input/output error
  /dev/vg04/swap: read failed after 0 of 4096 at 0: Input/output error
  /dev/vg04/swap: read failed after 0 of 4096 at 4096: Input/output error
  /dev/vg04/vz: read failed after 0 of 4096 at 995903864832: Input/output error
  /dev/vg04/vz: read failed after 0 of 4096 at 995903922176: Input/output error
  /dev/vg04/vz: read failed after 0 of 4096 at 0: Input/output error
  /dev/vg04/vz: read failed after 0 of 4096 at 4096: Input/output error
  Volume group "vg04" not found
  Skipping volume group vg04

The missing VG and LV's are not important, we just want to remove them.

As you can see we've tried all the suggestions made, so far without luck.

Output from 'lvm dumpconfig' can be checked at http://pastebin.com/MHiBzrLJ

Solution 1:

The solution was to run dmsetup, in this case the two commands

dmsetup remove vg04-vz
dmsetup remove vg04-swap

Before doing this, I checked with the command 'dmsetup info' that the 'open count' for both LV's were zero.

WARNING: dmsetup can wreck serious havoc with your disks so anyone using this information in the future please make sure you read the man page.

Solution 2:

I experienced similar after removing a VMware disk /dev/sdb.

Use lsscsi to determine which device is being used (you may need to install the program) then run lsscsi:

# lsscsi
[1:0:0:0]    cd/dvd  NECVMWar VMware IDE CDR10 1.00  /dev/sr0
[2:0:0:0]    disk    VMware   Virtual disk     1.0   /dev/sda
[2:0:1:0]    disk    VMware   Virtual disk     1.0   /dev/sdb <== the removed physical disk
[2:0:2:0]    disk    VMware   Virtual disk     1.0   /dev/sdc

Now write the delete file with

# echo 1 > /sys/class/scsi_device/2\:0\:1\:0/device/delete

The device is immediately gone.

Solution 3:

vgchange -a n /dev/vg04

If you have logical volumes still active you may have to

lvchange -a n /dev/vg04/swap
lvchange -a n dev/vg04/vz

etc.