(LVM) Cannot restore Volume Group with 1 PVs marked as missing. Couldn't find device with uuid, but the pv exists and blkid finds the same uuid
This problem is on a kvm/qemu hypervisor with CentOS 7, where we host multiple virtual machines.
The Disk /dev/sda( 1918.2 GB) is a raid6 LSI MEGARAID.
[root@mia3 ~]# fdisk -l
Disk /dev/sda: 1918.2 GB, 1918239768576 bytes, 3746562048 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk label type: dos
Disk identifier: 0x51549cd4
Device Boot Start End Blocks Id System
/dev/sda1 * 2048 1026047 512000 83 Linux
/dev/sda2 1026048 135243775 67108864 82 Linux swap / Solaris
/dev/sda3 135243776 449816575 157286400 83 Linux
/dev/sda4 449816576 3746562047 1648372736 8e Linux LVM
This is our disk configuration: sda1 - /boot, sda2 - swap, sda3 - / (root), sda4 - PV for our LVM with thin provisioning for our VMs.
After a power failure, our data on the root partition got corrupted. We had to reinstall the OS in order to bring up the server. We didn't touch the /dev/sda4 partition. Before that, we mounted the root partition in rescue mode and made a backup for the /etc/lvm/ folder.
After reinstalling Centos we tried to recover LVM using vgcfgrestore, but we got the error:
[root@mia3 ~]# vgcfgrestore -f /etc/lvm/backup/vg_kvm vg_kvm -v
No devices found.
/dev/sda4: Checksum error at offset 808960
Couldn't read volume group metadata from /dev/sda4.
Metadata location on /dev/sda4 at 808960 has invalid summary for VG.
Failed to read metadata summary from /dev/sda4
Failed to scan VG from /dev/sda4
Couldn't find device with uuid HNB35U-ilI4-WWAK-1Q4D-Netj-t68p-Po4vCT.
Consider using option --force to restore Volume Group vg_kvm with thin volumes.
Restore failed.
Scanning all devices to initialize lvmetad.
/dev/sda4: Checksum error at offset 808960
Couldn't read volume group metadata from /dev/sda4.
Metadata location on /dev/sda4 at 808960 has invalid summary for VG.
Failed to read metadata summary from /dev/sda4
Failed to scan VG from /dev/sda4
[root@mia3 ~]#
So we tried to readd the PV and ovewrite the metadata, but without success:
[root@mia3 ~]# pvcreate -f /dev/sda4 --restorefile /root/lvm/backup/vg_kvm --uuid HNB35U-ilI4-WWAK-1Q4D-Netj-t68p-Po4vCT -vv
devices/global_filter not found in config: defaulting to global_filter = [ "a|.*/|" ]
global/lvmetad_update_wait_time not found in config: defaulting to 10
devices/filter not found in config: defaulting to filter = [ "a|.*/|" ]
devices/cache not found in config: defaulting to /etc/lvm/cache/.cache
metadata/record_lvs_history not found in config: defaulting to 0
File-based locking selected.
Couldn't find device with uuid HNB35U-ilI4-WWAK-1Q4D-Netj-t68p-Po4vCT.
Stack vg_kvm/lvpool:0[0] on LV vg_kvm/lvpool_tdata:0.
Adding vg_kvm/lvpool:0 as an user of vg_kvm/lvpool_tdata.
Adding vg_kvm/lvpool:0 as an user of vg_kvm/lvpool_tmeta.
Adding vg_kvm/vsv11999-dm9oryxunpw8po9p-1xwrkrotul6i5k3g:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv12001-dmzdbquhkhrydlnf-kqjcbeyvz1c5xwun:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv12034-dnadddzaxsxoq2rx-iwxjz6wr2vmqhsj3:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv12036-dbgt9zilw2tg9sqd-c0xa9h1uxigjuqg9:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv12037-dctrjirji3agbl9l-dtzyf8jrb8lip0m7:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv12038-dm5mulwohkvz5ara-kccgnknpjdi11tdy:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv12039-d1lnb9qcyjaqmodk-mnl22loaflo24kpm:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv12040-daejvderic4uvccj-wjnydccgzohyaekq:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv12046-dswief4uk4gthxa1-ie5s7up6vbf37p68:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv12056-du7hvvsp3lw7jgja-uap8l1jpe29llrpu:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv12080-dwydbngcjptnnryn-hzz0jimwfl6yau2x:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv12094-d8cmko7aio2f1fyo-warkvewdzfeedme5:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv12097-dllp7oepp4homdj9-jm6n2itmvy29k9xf:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv12098-dcj6mnnidej8c80u-r3bkk3pfu8g9sf3o:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv12103-dwtyfq1quemcij56-4rwivmmtzxwk1ydb:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv12106-dngrxbyoapwac5sm-bstrvsbxzrztefob:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv12107-dq3h0wgnzkasoles-hv12r8qmyobojubp:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv12109-ddq6cmtow9qwuaka-v9a1omyljy0xujk4:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv12110-dvaaktx2bng99yl9-f5xh28mlg7u3mo5r:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv12111-db0wlufykbcbb2mp-dxhgxxmivthvqwwp:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv12112-dyfnblgyajfpidqs-ny0xpnxhzfzwc1db:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv12116-dowxexkxa0ygsgki-e4dih6xcnybtdgx2:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv12117-dwd0weu8ig1dvukx-fly3n8dyseklhntv:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv12118-dz9hbvnbtsihufm5-paqhexr1tuld8bnd:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv12119-dof5n90vez6smo1j-h1xmccyzl7qotery:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv12120-d9q0y85z4shpw64o-6zx5tmxhajpcxtkv:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv12121-dac4vuig8akyryd0-0e4rf2snm04r5mdy:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv12122-d99yza6ssbhxxkru-16ptyp8btexq6wkb:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv12124-d4ud5tu2ueajr0vc-r9pv77cjusnjue21:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv12126-dqjmtfzubhqvheij-blecac71kpwsmy62:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv12127-d5sgcmdodjaqtvp9-oqbroobow5tzn0va:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv12130-dtf7rf4suf8bov9t-8lb0agstvcpann2e:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv12131-dwkveordvertdam3-3g4qwtf5ydhy0hug:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv12134-dgpxbewt3xlkxlqb-j5ad04pt2fndqqmw:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv12135-d55krjg3dotss3e4-dwbbjmo0shwwwzv2:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv12138-dyz9yl6swxbaxfrj-1pgri05suzqyagqq:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv12140-dykgcwqaauznoxrm-4ibft4womwuvpwkx:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv12142-dwtlkzoke0a05u1o-rqxjogzuvjvfttxs:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv12144-dn3gyu3uroppwgzg-wraejm2ynwnmtmd4:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv12162-dalw90et3l9vbath-eo5womktdromb5sn:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv12211-dyyizuuag0mz4eok-je7du6slil1jlfzp:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv12329-d5xmlj2iqsuekgkm-cg6qplkwk48yiyeo:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv12333-drqvvb7fktr4n52b-estbavvooqdljpvo:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv12334-dbdf0lvefbgpqbxa-bmpnzainjdfvuavv:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv12339-d3yfeg1upsfoxbd3-nbxbccudjdtcgztt:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv12354-d1ctzl88dwv9znri-pzfp6weunhpdhvgn:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv12361-dbug4gr7r8mbsnva-yamgras92xuqin66:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv12088-dqphwqqeuwxtmm7y-4rurc6jm0lvcj57q:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv12089-dbhimrcqvjkaxvxc-3vpljlh2xkojhb9e:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv12090-dndlzsmpj8qgm2gb-wm4sg3d6n2cq3tpm:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv12091-dyyxcz5cf4w4cu6z-gja2kph6egpjvyzh:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv12092-dnllln6x7uud75ss-um9zelymhc04ohd6:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv12442-dyf2cgwdgaluli0s-cahwc74lrstt61gl:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv12456-dmiuo7guhru4snsf-yt55qun0wx5mtfkg:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv12087-dxbtb8w1g3tx5akk-bdqcsnlathemkrig:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv12234-dq82wyoapvxvh7zz-maaekbfd7h9ixks8:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv12588-das5kos8qt1ziktl-aky0pxwg8ecv9b48:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv12771-djaikmniei1vxia5-jhqasdwc763s3pmb:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv12784-d3k6aplejbvjrlcw-uwo81r0wkmeiptso:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv12139-dpnhdmg2odfabvfn-zqgiw42wfsgbbkrc:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv12137-dxtiqptvdwcrrcwa-22fkxorf6hl44evf:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv12866-d37htmu48aopiafv-sli9m0lt3dz0yeyl:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv12096-dz9pfno05wqq8vbg-sq8i5na5z8lrnm6n:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv12363-dtlh9w3tza99quoj-dlicfndz6e3pbfu0:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv12890-dmaaydjldw3c13jk-iskdygh5pczveo3r:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv12320-dqmecms9nep6muns-xmmkziafa8mz1hta:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv12936-dhdfa36ovcdzfafp-ultnlmom4scugsw1:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv12964-dctbp9xgo2gl8w6i-kbw9cinhoieea8av:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv12984-dwqjvkqc3aznhb4o-criyexpxvdndoscq:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv12143-d3s1toagczaiq4ma-oupwnk3eg27r41ze:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv13019-d5yrsi5a58bdzalc-i6mbum8zthk2arhs:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv13022-djszrasae5w7xegh-ehqsutbhrwd7qgsi:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv13027-dq6pg99jd2fphljg-u15x8sonbwukaree:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv12041-dlsmtmfa6fp4wcuc-bfhftkdhwfqa7sqb:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv13039-djtjimojqi3iaufp-csbinqtsvebzzmqx:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv13061-duocptl9qdtvwubp-feryqeejht4j24ql:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv13064-dexwkomdh62by80s-aymbuetu6p0btn4d:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv13076-drd4nrhy4u7fq1tn-f1kt8ubk2jch4ol3:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv13082-ds9gjikqyg9nftua-rrhk4dlrr8cp3la7:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv12759-du4yducr4ghi8eun-bzivqxabwaltaddm:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv12123-dzb9rktypp0jqinx-adtlzed02wpccxoa:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv13118-d0lj9gpvqfmox63i-9jstjcjkmemszrla:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv13121-du0r39n6wvnrclox-bviz76ig9vvexota:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv13133-d2fk6lg6bsppbkcs-hwk5bmsvqxa0y0bs:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv13204-dl6hzsmswetitsb4-bais6lpjbbhfkexn:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv13230-dtiyrmijg8mde5xy-vobukov1qxg8bx5j:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv13254-d3sdhrrgcvbwrsyp-yxaozxmt3clqzxxc:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv13271-drutw4ihq774xhs6-ecpwfmzcrqy0vvsm:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv13266-dmuj5cfeu2hrgfxw-2tpflycnwoedyidb:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv13380-d99ciziibe7rt3es-7fkh56myy79oushw:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv13487-d30yxx4wgfxcb9te-chc2bxojs7tvcwz3:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv13502-dc2oivdodzo94jj3-1qmpba9ti4zzbfeg:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv13277-deudjco25w9cd3rv-re1ha5qkozses9i3:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv12208-dfqhoqmtuwzktwqm-n9s5daovhtnyvyeh:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv13481-dthcwgpolombhfed-dkmesvsvmt1vem0d:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv13585-ddvwhm2vcqpq66uy-1brs98bctgr6bcq9:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv13586-dadacaexxn8nkuy3-fyx8zp5mursbqhns:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv13587-dc0fmskwdbkmz7mg-ra53wgfksiv7tsqk:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv13569-dk6v0pyvz8eoohl5-qyndiwn8zriir75s:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv13603-d66mz2amabmd63fj-rkcqmtvfnlfmywou:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv13613-dykuo32mnkgf9nce-hhkhn118dhsedoc8:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv13623-dksoizdtoglihaax-23o0zltutliyoo2q:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv13629-dxbpqqernfvufxrz-iv6jujdxm78hxjbn:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv13610-drrehnh36oipacdf-frnumdhlt823mntg:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv13654-dmlnhhqilztxff5a-4gawkysgtkze4st7:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv13669-dk3ukgidvj0l4x4x-9d3vfao5qejzecdn:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv13672-d1r1608gu96b0ujo-nn7ejmxkbfce40m4:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv13689-dhpzh8w7jlwi0zza-aqizb6tbeuk9k1ca:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv12477-dejmu0gihtllnj97-gaxwgicndwp25unx:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv13710-dv2bktawped5fz44-0bkdtrkp8tk2vynu:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv12125-d5d9pebts1rue7jt-vwnkapewjkfszked:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv13718-dmimkn9h1ax3t4am-8tdnpznya2reek8g:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv13545-dzaq2rjqfiyf5azl-kw35xomndhltjnrs:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv13723-daleipfjwu7zm7aj-s8jo6yhxwz4efl2y:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv13726-dpdqvmnzkvxkiz4y-cfudhed6m12hnlmj:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv12136-d0qvvamyx5qq1g6o-awtlh6pbotvlacyc:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv13258-du2kuwgkwhypcfzz-i6h00rd0exzw4b69:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv13755-dqglblzy3akyohph-wtev5tlrs5lkkejd:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv13758-dhfrybljxildcky1-ktotkou3mpvxfjnq:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv13759-dxqbq7phpjgoa1ed-dwtly1dkuorvlyli:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv13762-di54bo3us4p8gefa-34s2uhcye1rbdbvw:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv13767-dwtsyqxhqbp8fa9w-us8xvwm9q6nqkmzx:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv13768-dadnlkrau2cwhlbd-cj8liz4hfolpmjfa:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv13769-dt4nwlcmpfqyenob-jvpfd5jut8kvcm8u:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv13781-dvckr0uqfwrsvmc6-zklf1olhlhzzfppp:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv12133-dp55dttaolpbd6n8-rlhoezgituqkyrov:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv12042-dnncyzlyhfxyaxkt-metx0ei0zlz3d2sn:0 as an user of vg_kvm/lvpool.
metadata/pvmetadataignore not found in config: defaulting to 0
metadata/pvmetadatasize not found in config: defaulting to 255
metadata/pvmetadatacopies not found in config: defaulting to 1
report/output_format not found in config: defaulting to basic
log/report_command_log not found in config: defaulting to 0
Locking /run/lock/lvm/P_orphans WB
/dev/sda4: size is 3296745472 sectors
/dev/sda4: using cached size 3296745472 sectors
/dev/sda: size is 3746562048 sectors
/dev/sda1: size is 1024000 sectors
/dev/sda1: using cached size 1024000 sectors
/dev/sda2: size is 134217728 sectors
/dev/sda2: using cached size 134217728 sectors
/dev/sda3: size is 314572800 sectors
/dev/sda3: using cached size 314572800 sectors
/dev/sda4: using cached size 3296745472 sectors
/dev/sda4: using cached size 3296745472 sectors
Locking #orphans_lvm1 already done
Reading VG #orphans_lvm1
Locking #orphans_pool already done
Reading VG #orphans_pool
Locking #orphans_lvm2 already done
Reading VG #orphans_lvm2
Processing device /dev/sda1.
Processing device /dev/sda2.
Processing device /dev/sda3.
Processing device /dev/sda4.
/dev/sda4: lvm2 label detected at sector 1
/dev/sda4: Checksum error at offset 808960
Couldn't read volume group metadata from /dev/sda4.
Metadata location on /dev/sda4 at 808960 has invalid summary for VG.
Failed to read metadata summary from /dev/sda4
Failed to scan VG from /dev/sda4
Unlocking /run/lock/lvm/P_orphans
[root@mia3 ~]#
Note the error:
Couldn't find device with uuid HNB35U-ilI4-WWAK-1Q4D-Netj-t68p-Po4vCT.
The PV (/dev/sda4) has the same UUID:
[root@mia3 ~]# blkid
/dev/sda1: LABEL="boot" UUID="087755bd-9434-4597-953b-c056fd9cb285" TYPE="xfs"
/dev/sda2: LABEL="swap" UUID="c7fd2030-0441-4a4e-8462-0aeffc272bfa" TYPE="swap"
/dev/sda3: LABEL="root" UUID="3ee19cf6-87b5-40b5-a7dd-13d507efa734" TYPE="xfs"
/dev/sda4: UUID="HNB35U-ilI4-WWAK-1Q4D-Netj-t68p-Po4vCT" TYPE="LVM2_member"
[root@mia3 ~]#
Here is a small part of the content for /etc/lvm/backup/vg_kvm:
vg_kvm {
id = "i3usZz-vdt5-9yaU-Kbcl-XDWJ-OFr1-UhZRBa"
seqno = 1349
format = "lvm2" # informational
status = ["RESIZEABLE", "READ", "WRITE"]
flags = []
extent_size = 8192 # 4 Megabytes
max_lv = 0
max_pv = 0
metadata_copies = 0
physical_volumes {
pv0 {
id = "HNB35U-ilI4-WWAK-1Q4D-Netj-t68p-Po4vCT"
device = "/dev/sda4" # Hint only
status = ["ALLOCATABLE"]
flags = []
dev_size = 3296745472 # 1.53517 Terabytes
pe_start = 2048
pe_count = 402434 # 1.53516 Terabytes
}
}
logical_volumes {
lvpool {
id = "DHqs7X-95DA-ADNm-WTd9-cXDf-HR5a-MoHLRx"
status = ["READ", "WRITE", "VISIBLE"]
flags = []
creation_time = 1575901618 # 2019-12-09 16:26:58 +0200
creation_host = "mia3"
segment_count = 1
segment1 {
start_extent = 0
extent_count = 401081 # 1.53 Terabytes
type = "thin-pool"
metadata = "lvpool_tmeta"
pool = "lvpool_tdata"
transaction_id = 667
chunk_size = 2048 # 1024 Kilobytes
discards = "passdown"
zero_new_blocks = 1
}
}
vsv11999-dm9oryxunpw8po9p-1xwrkrotul6i5k3g {
id = "vBco77-gPno-S0cu-QZdx-9Nnk-twXR-cBBAzW"
status = ["READ", "WRITE", "VISIBLE"]
flags = []
creation_time = 1575902920 # 2019-12-09 16:48:40 +0200
creation_host = "mia3"
segment_count = 1
segment1 {
start_extent = 0
extent_count = 6400 # 25 Gigabytes
type = "thin"
thin_pool = "lvpool"
transaction_id = 0
device_id = 1
}
}
vsv12001-dmzdbquhkhrydlnf-kqjcbeyvz1c5xwun {
id = "xqDzS2-iHXO-2zEi-LRHO-0FDR-hvcM-oRdFeD"
status = ["READ", "WRITE", "VISIBLE"]
flags = []
creation_time = 1575904408 # 2019-12-09 17:13:28 +0200
creation_host = "mia3"
segment_count = 1
segment1 {
start_extent = 0
extent_count = 3840 # 15 Gigabytes
type = "thin"
thin_pool = "lvpool"
transaction_id = 1
device_id = 2
}
}
vsv12034-dnadddzaxsxoq2rx-iwxjz6wr2vmqhsj3 {
id = "VIDuPN-SN4Z-V1qm-gzI9-sSnV-WGcp-NBvgUa"
status = ["READ", "WRITE", "VISIBLE"]
flags = []
creation_time = 1575986509 # 2019-12-10 16:01:49 +0200
creation_host = "mia3"
segment_count = 1
segment1 {
start_extent = 0
extent_count = 3840 # 15 Gigabytes
type = "thin"
thin_pool = "lvpool"
transaction_id = 6
device_id = 5
}
}
vsv12036-dbgt9zilw2tg9sqd-c0xa9h1uxigjuqg9 {
id = "XnnwPm-e3jE-VU9W-7QIz-3Sz5-HDSM-sEe7NY"
status = ["READ", "WRITE", "VISIBLE"]
flags = []
creation_time = 1575988785 # 2019-12-10 16:39:45 +0200
creation_host = "mia3"
segment_count = 1
segment1 {
start_extent = 0
extent_count = 3840 # 15 Gigabytes
type = "thin"
thin_pool = "lvpool"
transaction_id = 8
device_id = 7
}
}
vsv12037-dctrjirji3agbl9l-dtzyf8jrb8lip0m7 {
id = "ysHM90-Vmof-Mrwj-Nb1s-HPER-g3v5-hdutBz"
status = ["READ", "WRITE", "VISIBLE"]
flags = []
creation_time = 1575989058 # 2019-12-10 16:44:18 +0200
creation_host = "mia3"
segment_count = 1
segment1 {
start_extent = 0
extent_count = 3840 # 15 Gigabytes
type = "thin"
thin_pool = "lvpool"
transaction_id = 9
device_id = 8
}
}
All the data (root partitions for our VMs) are still there, we have checked with disk recovery tools like testdisk, but we can't recover the lvm. What are we doing wrong? Is there a possibility of recovering?
#UPDATE
Using Centos6 CD and rescue mode I have managed to restore the PV and VG. However, I can't reactivate the VG and LV pool, we're getting the error:
Executing: /usr/sbin/thin_check -q --clear-needs-check-flag /dev/mapper/vg_kvm-lvpool_tmeta
/usr/sbin/thin_check failed: 1
Check of pool vg_kvm/lvpool failed (status:1). Manual repair required!
It appears Red Hat 7 and 8 have similar issues when trying to use vgcfgrestore: The pvcreate command fails with "Metadata location has invalid summary for VG" error . CentOS shares a base with Red Hat, so...
One answer there was "We have a solution for this problem upstream and in RHEL8, Unfortunately, it is too invasive to consider back porting to RHEL7 at this stage." Another person in that bug said they fixed it by booting Red Hat 6, and then vgcfgrestore worked.
Perhaps you can boot a new sysrescue CD from a USB stick and do it?
And as I said in a comment, either backup the whole volume with dd, or at least do something like dd if=/dev/sda4 of=/somewere/backup-first100MB.img bs=1M count=100 so that you have the header info.
It worked with Centos6 rescue. After restoring, I booted into normal centos 7 and the vg was there. However, now I can't activate the lv (thin pool):
[root@mia3 ~]# lvchange -ay /dev/vg_kvm/lvpool
Check of pool vg_kvm/lvpool failed (status:1). Manual repair required!
[root@mia3 ~]#
Here is the command again with -vvv
[root@mia3 ~]# lvchange -ay /dev/vg_kvm/lvpool -vv
devices/global_filter not found in config: defaulting to global_filter = [ "a|.*/|" ]
global/lvmetad_update_wait_time not found in config: defaulting to 10
Setting devices/filter to filter = [ "a|/dev/sda.*|", "a|/dev/disk/by-id/dm-uuid-.*mpath-.*|", "r|.*|" ]
devices/cache not found in config: defaulting to /etc/lvm/cache/.cache
metadata/record_lvs_history not found in config: defaulting to 0
File-based locking selected.
report/output_format not found in config: defaulting to basic
log/report_command_log not found in config: defaulting to 0
Obtaining the complete list of VGs before processing their LVs
Processing VG vg_kvm i3usZz-vdt5-9yaU-Kbcl-XDWJ-OFr1-UhZRBa
Locking /run/lock/lvm/V_vg_kvm RB
Reading VG vg_kvm i3usZz-vdt5-9yaU-Kbcl-XDWJ-OFr1-UhZRBa
Stack vg_kvm/lvpool:0[0] on LV vg_kvm/lvpool_tdata:0.
Adding vg_kvm/lvpool:0 as an user of vg_kvm/lvpool_tdata.
Adding vg_kvm/lvpool:0 as an user of vg_kvm/lvpool_tmeta.
Adding vg_kvm/vsv11999-dm9oryxunpw8po9p-1xwrkrotul6i5k3g:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv12001-dmzdbquhkhrydlnf-kqjcbeyvz1c5xwun:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv12034-dnadddzaxsxoq2rx-iwxjz6wr2vmqhsj3:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv12036-dbgt9zilw2tg9sqd-c0xa9h1uxigjuqg9:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv12037-dctrjirji3agbl9l-dtzyf8jrb8lip0m7:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv12038-dm5mulwohkvz5ara-kccgnknpjdi11tdy:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv12039-d1lnb9qcyjaqmodk-mnl22loaflo24kpm:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv12040-daejvderic4uvccj-wjnydccgzohyaekq:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv12046-dswief4uk4gthxa1-ie5s7up6vbf37p68:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv12056-du7hvvsp3lw7jgja-uap8l1jpe29llrpu:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv12080-dwydbngcjptnnryn-hzz0jimwfl6yau2x:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv12094-d8cmko7aio2f1fyo-warkvewdzfeedme5:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv12097-dllp7oepp4homdj9-jm6n2itmvy29k9xf:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv12098-dcj6mnnidej8c80u-r3bkk3pfu8g9sf3o:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv12103-dwtyfq1quemcij56-4rwivmmtzxwk1ydb:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv12106-dngrxbyoapwac5sm-bstrvsbxzrztefob:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv12107-dq3h0wgnzkasoles-hv12r8qmyobojubp:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv12109-ddq6cmtow9qwuaka-v9a1omyljy0xujk4:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv12110-dvaaktx2bng99yl9-f5xh28mlg7u3mo5r:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv12111-db0wlufykbcbb2mp-dxhgxxmivthvqwwp:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv12112-dyfnblgyajfpidqs-ny0xpnxhzfzwc1db:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv12116-dowxexkxa0ygsgki-e4dih6xcnybtdgx2:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv12117-dwd0weu8ig1dvukx-fly3n8dyseklhntv:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv12118-dz9hbvnbtsihufm5-paqhexr1tuld8bnd:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv12119-dof5n90vez6smo1j-h1xmccyzl7qotery:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv12120-d9q0y85z4shpw64o-6zx5tmxhajpcxtkv:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv12121-dac4vuig8akyryd0-0e4rf2snm04r5mdy:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv12122-d99yza6ssbhxxkru-16ptyp8btexq6wkb:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv12124-d4ud5tu2ueajr0vc-r9pv77cjusnjue21:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv12126-dqjmtfzubhqvheij-blecac71kpwsmy62:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv12127-d5sgcmdodjaqtvp9-oqbroobow5tzn0va:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv12130-dtf7rf4suf8bov9t-8lb0agstvcpann2e:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv12131-dwkveordvertdam3-3g4qwtf5ydhy0hug:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv12134-dgpxbewt3xlkxlqb-j5ad04pt2fndqqmw:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv12135-d55krjg3dotss3e4-dwbbjmo0shwwwzv2:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv12138-dyz9yl6swxbaxfrj-1pgri05suzqyagqq:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv12140-dykgcwqaauznoxrm-4ibft4womwuvpwkx:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv12142-dwtlkzoke0a05u1o-rqxjogzuvjvfttxs:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv12144-dn3gyu3uroppwgzg-wraejm2ynwnmtmd4:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv12162-dalw90et3l9vbath-eo5womktdromb5sn:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv12211-dyyizuuag0mz4eok-je7du6slil1jlfzp:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv12329-d5xmlj2iqsuekgkm-cg6qplkwk48yiyeo:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv12333-drqvvb7fktr4n52b-estbavvooqdljpvo:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv12334-dbdf0lvefbgpqbxa-bmpnzainjdfvuavv:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv12339-d3yfeg1upsfoxbd3-nbxbccudjdtcgztt:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv12354-d1ctzl88dwv9znri-pzfp6weunhpdhvgn:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv12361-dbug4gr7r8mbsnva-yamgras92xuqin66:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv12088-dqphwqqeuwxtmm7y-4rurc6jm0lvcj57q:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv12089-dbhimrcqvjkaxvxc-3vpljlh2xkojhb9e:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv12090-dndlzsmpj8qgm2gb-wm4sg3d6n2cq3tpm:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv12091-dyyxcz5cf4w4cu6z-gja2kph6egpjvyzh:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv12092-dnllln6x7uud75ss-um9zelymhc04ohd6:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv12442-dyf2cgwdgaluli0s-cahwc74lrstt61gl:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv12456-dmiuo7guhru4snsf-yt55qun0wx5mtfkg:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv12087-dxbtb8w1g3tx5akk-bdqcsnlathemkrig:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv12234-dq82wyoapvxvh7zz-maaekbfd7h9ixks8:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv12588-das5kos8qt1ziktl-aky0pxwg8ecv9b48:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv12771-djaikmniei1vxia5-jhqasdwc763s3pmb:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv12784-d3k6aplejbvjrlcw-uwo81r0wkmeiptso:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv12139-dpnhdmg2odfabvfn-zqgiw42wfsgbbkrc:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv12137-dxtiqptvdwcrrcwa-22fkxorf6hl44evf:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv12866-d37htmu48aopiafv-sli9m0lt3dz0yeyl:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv12096-dz9pfno05wqq8vbg-sq8i5na5z8lrnm6n:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv12363-dtlh9w3tza99quoj-dlicfndz6e3pbfu0:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv12890-dmaaydjldw3c13jk-iskdygh5pczveo3r:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv12320-dqmecms9nep6muns-xmmkziafa8mz1hta:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv12936-dhdfa36ovcdzfafp-ultnlmom4scugsw1:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv12964-dctbp9xgo2gl8w6i-kbw9cinhoieea8av:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv12984-dwqjvkqc3aznhb4o-criyexpxvdndoscq:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv12143-d3s1toagczaiq4ma-oupwnk3eg27r41ze:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv13019-d5yrsi5a58bdzalc-i6mbum8zthk2arhs:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv13022-djszrasae5w7xegh-ehqsutbhrwd7qgsi:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv13027-dq6pg99jd2fphljg-u15x8sonbwukaree:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv12041-dlsmtmfa6fp4wcuc-bfhftkdhwfqa7sqb:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv13039-djtjimojqi3iaufp-csbinqtsvebzzmqx:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv13061-duocptl9qdtvwubp-feryqeejht4j24ql:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv13064-dexwkomdh62by80s-aymbuetu6p0btn4d:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv13076-drd4nrhy4u7fq1tn-f1kt8ubk2jch4ol3:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv13082-ds9gjikqyg9nftua-rrhk4dlrr8cp3la7:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv12759-du4yducr4ghi8eun-bzivqxabwaltaddm:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv12123-dzb9rktypp0jqinx-adtlzed02wpccxoa:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv13118-d0lj9gpvqfmox63i-9jstjcjkmemszrla:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv13121-du0r39n6wvnrclox-bviz76ig9vvexota:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv13133-d2fk6lg6bsppbkcs-hwk5bmsvqxa0y0bs:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv13204-dl6hzsmswetitsb4-bais6lpjbbhfkexn:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv13230-dtiyrmijg8mde5xy-vobukov1qxg8bx5j:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv13254-d3sdhrrgcvbwrsyp-yxaozxmt3clqzxxc:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv13271-drutw4ihq774xhs6-ecpwfmzcrqy0vvsm:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv13266-dmuj5cfeu2hrgfxw-2tpflycnwoedyidb:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv13380-d99ciziibe7rt3es-7fkh56myy79oushw:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv13487-d30yxx4wgfxcb9te-chc2bxojs7tvcwz3:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv13502-dc2oivdodzo94jj3-1qmpba9ti4zzbfeg:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv13277-deudjco25w9cd3rv-re1ha5qkozses9i3:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv12208-dfqhoqmtuwzktwqm-n9s5daovhtnyvyeh:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv13481-dthcwgpolombhfed-dkmesvsvmt1vem0d:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv13585-ddvwhm2vcqpq66uy-1brs98bctgr6bcq9:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv13586-dadacaexxn8nkuy3-fyx8zp5mursbqhns:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv13587-dc0fmskwdbkmz7mg-ra53wgfksiv7tsqk:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv13569-dk6v0pyvz8eoohl5-qyndiwn8zriir75s:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv13603-d66mz2amabmd63fj-rkcqmtvfnlfmywou:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv13613-dykuo32mnkgf9nce-hhkhn118dhsedoc8:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv13623-dksoizdtoglihaax-23o0zltutliyoo2q:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv13629-dxbpqqernfvufxrz-iv6jujdxm78hxjbn:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv13610-drrehnh36oipacdf-frnumdhlt823mntg:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv13654-dmlnhhqilztxff5a-4gawkysgtkze4st7:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv13669-dk3ukgidvj0l4x4x-9d3vfao5qejzecdn:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv13672-d1r1608gu96b0ujo-nn7ejmxkbfce40m4:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv13689-dhpzh8w7jlwi0zza-aqizb6tbeuk9k1ca:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv12477-dejmu0gihtllnj97-gaxwgicndwp25unx:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv13710-dv2bktawped5fz44-0bkdtrkp8tk2vynu:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv12125-d5d9pebts1rue7jt-vwnkapewjkfszked:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv13718-dmimkn9h1ax3t4am-8tdnpznya2reek8g:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv13545-dzaq2rjqfiyf5azl-kw35xomndhltjnrs:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv13723-daleipfjwu7zm7aj-s8jo6yhxwz4efl2y:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv13726-dpdqvmnzkvxkiz4y-cfudhed6m12hnlmj:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv12136-d0qvvamyx5qq1g6o-awtlh6pbotvlacyc:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv13258-du2kuwgkwhypcfzz-i6h00rd0exzw4b69:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv13755-dqglblzy3akyohph-wtev5tlrs5lkkejd:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv13758-dhfrybljxildcky1-ktotkou3mpvxfjnq:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv13759-dxqbq7phpjgoa1ed-dwtly1dkuorvlyli:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv13762-di54bo3us4p8gefa-34s2uhcye1rbdbvw:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv13767-dwtsyqxhqbp8fa9w-us8xvwm9q6nqkmzx:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv13768-dadnlkrau2cwhlbd-cj8liz4hfolpmjfa:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv13769-dt4nwlcmpfqyenob-jvpfd5jut8kvcm8u:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv13781-dvckr0uqfwrsvmc6-zklf1olhlhzzfppp:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv12133-dp55dttaolpbd6n8-rlhoezgituqkyrov:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv12042-dnncyzlyhfxyaxkt-metx0ei0zlz3d2sn:0 as an user of vg_kvm/lvpool.
/dev/sda4: size is 3296745472 sectors
Adding vg_kvm/lvpool to the list of LVs to be processed.
Processing LV lvpool in VG vg_kvm.
Activating logical volume vg_kvm/lvpool exclusively.
Locking /run/lock/lvm/A_vg_kvm WB
vg_kvm/lvpool is not active
vg_kvm/vsv11999-dm9oryxunpw8po9p-1xwrkrotul6i5k3g is not active
vg_kvm/vsv12001-dmzdbquhkhrydlnf-kqjcbeyvz1c5xwun is not active
vg_kvm/vsv12034-dnadddzaxsxoq2rx-iwxjz6wr2vmqhsj3 is not active
vg_kvm/vsv12036-dbgt9zilw2tg9sqd-c0xa9h1uxigjuqg9 is not active
vg_kvm/vsv12037-dctrjirji3agbl9l-dtzyf8jrb8lip0m7 is not active
vg_kvm/vsv12038-dm5mulwohkvz5ara-kccgnknpjdi11tdy is not active
vg_kvm/vsv12039-d1lnb9qcyjaqmodk-mnl22loaflo24kpm is not active
vg_kvm/vsv12040-daejvderic4uvccj-wjnydccgzohyaekq is not active
vg_kvm/vsv12046-dswief4uk4gthxa1-ie5s7up6vbf37p68 is not active
vg_kvm/vsv12056-du7hvvsp3lw7jgja-uap8l1jpe29llrpu is not active
vg_kvm/vsv12080-dwydbngcjptnnryn-hzz0jimwfl6yau2x is not active
vg_kvm/vsv12094-d8cmko7aio2f1fyo-warkvewdzfeedme5 is not active
vg_kvm/vsv12097-dllp7oepp4homdj9-jm6n2itmvy29k9xf is not active
vg_kvm/vsv12098-dcj6mnnidej8c80u-r3bkk3pfu8g9sf3o is not active
vg_kvm/vsv12103-dwtyfq1quemcij56-4rwivmmtzxwk1ydb is not active
vg_kvm/vsv12106-dngrxbyoapwac5sm-bstrvsbxzrztefob is not active
vg_kvm/vsv12107-dq3h0wgnzkasoles-hv12r8qmyobojubp is not active
vg_kvm/vsv12109-ddq6cmtow9qwuaka-v9a1omyljy0xujk4 is not active
vg_kvm/vsv12110-dvaaktx2bng99yl9-f5xh28mlg7u3mo5r is not active
vg_kvm/vsv12111-db0wlufykbcbb2mp-dxhgxxmivthvqwwp is not active
vg_kvm/vsv12112-dyfnblgyajfpidqs-ny0xpnxhzfzwc1db is not active
vg_kvm/vsv12116-dowxexkxa0ygsgki-e4dih6xcnybtdgx2 is not active
vg_kvm/vsv12117-dwd0weu8ig1dvukx-fly3n8dyseklhntv is not active
vg_kvm/vsv12118-dz9hbvnbtsihufm5-paqhexr1tuld8bnd is not active
vg_kvm/vsv12119-dof5n90vez6smo1j-h1xmccyzl7qotery is not active
vg_kvm/vsv12120-d9q0y85z4shpw64o-6zx5tmxhajpcxtkv is not active
vg_kvm/vsv12121-dac4vuig8akyryd0-0e4rf2snm04r5mdy is not active
vg_kvm/vsv12122-d99yza6ssbhxxkru-16ptyp8btexq6wkb is not active
vg_kvm/vsv12124-d4ud5tu2ueajr0vc-r9pv77cjusnjue21 is not active
vg_kvm/vsv12126-dqjmtfzubhqvheij-blecac71kpwsmy62 is not active
vg_kvm/vsv12127-d5sgcmdodjaqtvp9-oqbroobow5tzn0va is not active
vg_kvm/vsv12130-dtf7rf4suf8bov9t-8lb0agstvcpann2e is not active
vg_kvm/vsv12131-dwkveordvertdam3-3g4qwtf5ydhy0hug is not active
vg_kvm/vsv12134-dgpxbewt3xlkxlqb-j5ad04pt2fndqqmw is not active
vg_kvm/vsv12135-d55krjg3dotss3e4-dwbbjmo0shwwwzv2 is not active
vg_kvm/vsv12138-dyz9yl6swxbaxfrj-1pgri05suzqyagqq is not active
vg_kvm/vsv12140-dykgcwqaauznoxrm-4ibft4womwuvpwkx is not active
vg_kvm/vsv12142-dwtlkzoke0a05u1o-rqxjogzuvjvfttxs is not active
vg_kvm/vsv12144-dn3gyu3uroppwgzg-wraejm2ynwnmtmd4 is not active
vg_kvm/vsv12162-dalw90et3l9vbath-eo5womktdromb5sn is not active
vg_kvm/vsv12211-dyyizuuag0mz4eok-je7du6slil1jlfzp is not active
vg_kvm/vsv12329-d5xmlj2iqsuekgkm-cg6qplkwk48yiyeo is not active
vg_kvm/vsv12333-drqvvb7fktr4n52b-estbavvooqdljpvo is not active
vg_kvm/vsv12334-dbdf0lvefbgpqbxa-bmpnzainjdfvuavv is not active
vg_kvm/vsv12339-d3yfeg1upsfoxbd3-nbxbccudjdtcgztt is not active
vg_kvm/vsv12354-d1ctzl88dwv9znri-pzfp6weunhpdhvgn is not active
vg_kvm/vsv12361-dbug4gr7r8mbsnva-yamgras92xuqin66 is not active
vg_kvm/vsv12088-dqphwqqeuwxtmm7y-4rurc6jm0lvcj57q is not active
vg_kvm/vsv12089-dbhimrcqvjkaxvxc-3vpljlh2xkojhb9e is not active
vg_kvm/vsv12090-dndlzsmpj8qgm2gb-wm4sg3d6n2cq3tpm is not active
vg_kvm/vsv12091-dyyxcz5cf4w4cu6z-gja2kph6egpjvyzh is not active
vg_kvm/vsv12092-dnllln6x7uud75ss-um9zelymhc04ohd6 is not active
vg_kvm/vsv12442-dyf2cgwdgaluli0s-cahwc74lrstt61gl is not active
vg_kvm/vsv12456-dmiuo7guhru4snsf-yt55qun0wx5mtfkg is not active
vg_kvm/vsv12087-dxbtb8w1g3tx5akk-bdqcsnlathemkrig is not active
vg_kvm/vsv12234-dq82wyoapvxvh7zz-maaekbfd7h9ixks8 is not active
vg_kvm/vsv12588-das5kos8qt1ziktl-aky0pxwg8ecv9b48 is not active
vg_kvm/vsv12771-djaikmniei1vxia5-jhqasdwc763s3pmb is not active
vg_kvm/vsv12784-d3k6aplejbvjrlcw-uwo81r0wkmeiptso is not active
vg_kvm/vsv12139-dpnhdmg2odfabvfn-zqgiw42wfsgbbkrc is not active
vg_kvm/vsv12137-dxtiqptvdwcrrcwa-22fkxorf6hl44evf is not active
vg_kvm/vsv12866-d37htmu48aopiafv-sli9m0lt3dz0yeyl is not active
vg_kvm/vsv12096-dz9pfno05wqq8vbg-sq8i5na5z8lrnm6n is not active
vg_kvm/vsv12363-dtlh9w3tza99quoj-dlicfndz6e3pbfu0 is not active
vg_kvm/vsv12890-dmaaydjldw3c13jk-iskdygh5pczveo3r is not active
vg_kvm/vsv12320-dqmecms9nep6muns-xmmkziafa8mz1hta is not active
vg_kvm/vsv12936-dhdfa36ovcdzfafp-ultnlmom4scugsw1 is not active
vg_kvm/vsv12964-dctbp9xgo2gl8w6i-kbw9cinhoieea8av is not active
vg_kvm/vsv12984-dwqjvkqc3aznhb4o-criyexpxvdndoscq is not active
vg_kvm/vsv12143-d3s1toagczaiq4ma-oupwnk3eg27r41ze is not active
vg_kvm/vsv13019-d5yrsi5a58bdzalc-i6mbum8zthk2arhs is not active
vg_kvm/vsv13022-djszrasae5w7xegh-ehqsutbhrwd7qgsi is not active
vg_kvm/vsv13027-dq6pg99jd2fphljg-u15x8sonbwukaree is not active
vg_kvm/vsv12041-dlsmtmfa6fp4wcuc-bfhftkdhwfqa7sqb is not active
vg_kvm/vsv13039-djtjimojqi3iaufp-csbinqtsvebzzmqx is not active
vg_kvm/vsv13061-duocptl9qdtvwubp-feryqeejht4j24ql is not active
vg_kvm/vsv13064-dexwkomdh62by80s-aymbuetu6p0btn4d is not active
vg_kvm/vsv13076-drd4nrhy4u7fq1tn-f1kt8ubk2jch4ol3 is not active
vg_kvm/vsv13082-ds9gjikqyg9nftua-rrhk4dlrr8cp3la7 is not active
vg_kvm/vsv12759-du4yducr4ghi8eun-bzivqxabwaltaddm is not active
vg_kvm/vsv12123-dzb9rktypp0jqinx-adtlzed02wpccxoa is not active
vg_kvm/vsv13118-d0lj9gpvqfmox63i-9jstjcjkmemszrla is not active
vg_kvm/vsv13121-du0r39n6wvnrclox-bviz76ig9vvexota is not active
vg_kvm/vsv13133-d2fk6lg6bsppbkcs-hwk5bmsvqxa0y0bs is not active
vg_kvm/vsv13204-dl6hzsmswetitsb4-bais6lpjbbhfkexn is not active
vg_kvm/vsv13230-dtiyrmijg8mde5xy-vobukov1qxg8bx5j is not active
vg_kvm/vsv13254-d3sdhrrgcvbwrsyp-yxaozxmt3clqzxxc is not active
vg_kvm/vsv13271-drutw4ihq774xhs6-ecpwfmzcrqy0vvsm is not active
vg_kvm/vsv13266-dmuj5cfeu2hrgfxw-2tpflycnwoedyidb is not active
vg_kvm/vsv13380-d99ciziibe7rt3es-7fkh56myy79oushw is not active
vg_kvm/vsv13487-d30yxx4wgfxcb9te-chc2bxojs7tvcwz3 is not active
vg_kvm/vsv13502-dc2oivdodzo94jj3-1qmpba9ti4zzbfeg is not active
vg_kvm/vsv13277-deudjco25w9cd3rv-re1ha5qkozses9i3 is not active
vg_kvm/vsv12208-dfqhoqmtuwzktwqm-n9s5daovhtnyvyeh is not active
vg_kvm/vsv13481-dthcwgpolombhfed-dkmesvsvmt1vem0d is not active
vg_kvm/vsv13585-ddvwhm2vcqpq66uy-1brs98bctgr6bcq9 is not active
vg_kvm/vsv13586-dadacaexxn8nkuy3-fyx8zp5mursbqhns is not active
vg_kvm/vsv13587-dc0fmskwdbkmz7mg-ra53wgfksiv7tsqk is not active
vg_kvm/vsv13569-dk6v0pyvz8eoohl5-qyndiwn8zriir75s is not active
vg_kvm/vsv13603-d66mz2amabmd63fj-rkcqmtvfnlfmywou is not active
vg_kvm/vsv13613-dykuo32mnkgf9nce-hhkhn118dhsedoc8 is not active
vg_kvm/vsv13623-dksoizdtoglihaax-23o0zltutliyoo2q is not active
vg_kvm/vsv13629-dxbpqqernfvufxrz-iv6jujdxm78hxjbn is not active
vg_kvm/vsv13610-drrehnh36oipacdf-frnumdhlt823mntg is not active
vg_kvm/vsv13654-dmlnhhqilztxff5a-4gawkysgtkze4st7 is not active
vg_kvm/vsv13669-dk3ukgidvj0l4x4x-9d3vfao5qejzecdn is not active
vg_kvm/vsv13672-d1r1608gu96b0ujo-nn7ejmxkbfce40m4 is not active
vg_kvm/vsv13689-dhpzh8w7jlwi0zza-aqizb6tbeuk9k1ca is not active
vg_kvm/vsv12477-dejmu0gihtllnj97-gaxwgicndwp25unx is not active
vg_kvm/vsv13710-dv2bktawped5fz44-0bkdtrkp8tk2vynu is not active
vg_kvm/vsv12125-d5d9pebts1rue7jt-vwnkapewjkfszked is not active
vg_kvm/vsv13718-dmimkn9h1ax3t4am-8tdnpznya2reek8g is not active
vg_kvm/vsv13545-dzaq2rjqfiyf5azl-kw35xomndhltjnrs is not active
vg_kvm/vsv13723-daleipfjwu7zm7aj-s8jo6yhxwz4efl2y is not active
vg_kvm/vsv13726-dpdqvmnzkvxkiz4y-cfudhed6m12hnlmj is not active
vg_kvm/vsv12136-d0qvvamyx5qq1g6o-awtlh6pbotvlacyc is not active
vg_kvm/vsv13258-du2kuwgkwhypcfzz-i6h00rd0exzw4b69 is not active
vg_kvm/vsv13755-dqglblzy3akyohph-wtev5tlrs5lkkejd is not active
vg_kvm/vsv13758-dhfrybljxildcky1-ktotkou3mpvxfjnq is not active
vg_kvm/vsv13759-dxqbq7phpjgoa1ed-dwtly1dkuorvlyli is not active
vg_kvm/vsv13762-di54bo3us4p8gefa-34s2uhcye1rbdbvw is not active
vg_kvm/vsv13767-dwtsyqxhqbp8fa9w-us8xvwm9q6nqkmzx is not active
vg_kvm/vsv13768-dadnlkrau2cwhlbd-cj8liz4hfolpmjfa is not active
vg_kvm/vsv13769-dt4nwlcmpfqyenob-jvpfd5jut8kvcm8u is not active
vg_kvm/vsv13781-dvckr0uqfwrsvmc6-zklf1olhlhzzfppp is not active
vg_kvm/vsv12133-dp55dttaolpbd6n8-rlhoezgituqkyrov is not active
vg_kvm/vsv12042-dnncyzlyhfxyaxkt-metx0ei0zlz3d2sn is not active
vg_kvm/lvpool is not active
vg_kvm/lvpool_tmeta is not active
vg_kvm/lvpool_tdata is not active
Locking LV i3usZzvdt59yaUKbclXDWJOFr1UhZRBaDHqs7X95DAADNmWTd9cXDfHR5aMoHLRx (EX)
activation/volume_list configuration setting not defined: Checking only host tags for vg_kvm/lvpool.
/dev/sda4: read_ahead is 256 sectors
Getting driver version
Getting target version for thin-pool
Found thin-pool target v1.21.0.
Getting target version for thin-pool
Found thin-pool target v1.21.0.
Getting target version for linear
Found linear target v1.3.0.
Getting target version for striped
Found striped target v1.6.0.
Creating vg_kvm-lvpool_tmeta
Loading table for vg_kvm-lvpool_tmeta (253:0).
Resuming vg_kvm-lvpool_tmeta (253:0).
Creating vg_kvm-lvpool_tdata
Loading table for vg_kvm-lvpool_tdata (253:1).
Resuming vg_kvm-lvpool_tdata (253:1).
global/thin_check_executable not found in config: defaulting to /usr/sbin/thin_check
global/thin_check_options not found in config: defaulting to thin_check_options = [ "-q", "--clear-needs-check-flag" ]
Executing: /usr/sbin/thin_check -q --clear-needs-check-flag /dev/mapper/vg_kvm-lvpool_tmeta
/usr/sbin/thin_check failed: 1
Check of pool vg_kvm/lvpool failed (status:1). Manual repair required!
Removing vg_kvm-lvpool_tdata (253:1)
Removing vg_kvm-lvpool_tmeta (253:0)
Unlocking /run/lock/lvm/A_vg_kvm
Unlocking /run/lock/lvm/V_vg_kvm
You can wipe the first 2k of the PV with the error before restoring your VG meta backup. The default metadatasize of LVM is 1MB, so this shouldn't reach your data. The dd will invalidate the header so LVM will think its an empty disk, then you can restore the PV and VG without it complaining about the state of the existing (invalid) PV/VG meta:
~]# dd if=/dev/zero bs=1k count=2 of=/dev/sdX
~]# sync # force to disk, LVM might use directio
~]# pvcreate -ff --uuid 06kvOm-xN2Y-iX1y-Tt69-LfFh-ltkG-TtRsgy --restorefile data /dev/bcache0
Physical volume "/dev/sdX" successfully created.
~]# vgcfgrestore --force data
Restored volume group data
It worked for me, be careful with the dd command!