LV Path is Missing

I have a pacemaker HA cluster with two nodes. Those two nodes use iSCSI disks and they can successfully connect iSCSI disks.

However, recently an unknown error happened on my cluster: all the nodes can detect iSCSI disks, however, LVs can't be imported.

I can see /dev/sdb - /dev/sdd have been detected:

[root@web1 ~]#  iscsiadm --mode node --targetname iqn.1991-05.com.microsoft:smb-web-target --login

Logging in to [iface: default, target: iqn.1991-05.com.microsoft:smb-web-target, portal: 192.168.0.18,3260] (multiple)
Login to [iface: default, target: iqn.1991-05.com.microsoft:smb-web-target, portal: 192.168.0.18,3260] successful.

[root@web1 ~]# lsblk

NAME        MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
sda           8:0    0   127G  0 disk
├─sda1        8:1    0   200M  0 part /boot/efi
├─sda2        8:2    0     1G  0 part /boot
└─sda3        8:3    0 125.8G  0 part
  ├─cl-root 253:0    0    50G  0 lvm  /
  ├─cl-swap 253:1    0     2G  0 lvm  [SWAP]
  └─cl-home 253:2    0  73.8G  0 lvm  /home
sdb           8:16   0    10G  0 disk
sdc           8:32   0   512M  0 disk
sdd           8:48   0     4G  0 disk
├─sdd1        8:49   0     2G  0 part
└─sdd2        8:50   0     2G  0 part
sr0          11:0    1  1024M  0 rom

All the LVM information have been recognized:

[root@web1 ~]# pvs
  PV         VG     Fmt  Attr PSize   PFree
  /dev/sda3  cl     lvm2 a--  125.80g  4.00m
  /dev/sdb   web_vg lvm2 a--  <10.00g <3.00g
  /dev/sdd1  db_vg  lvm2 a--   <2.00g     0
  /dev/sdd2  db_vg  lvm2 a--   <2.00g     0
[root@web1 ~]# vgs
  VG     #PV #LV #SN Attr   VSize   VFree
  cl       1   3   0 wz--n- 125.80g  4.00m
  db_vg    2   1   0 wz--n-   3.99g     0
  web_vg   1   3   0 wz--n- <10.00g <3.00g
[root@web1 ~]# lvs
  LV       VG     Attr       LSize   Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
  home     cl     -wi-ao---- <73.80g
  root     cl     -wi-ao----  50.00g
  swap     cl     -wi-ao----   2.00g
  mysql_lv db_vg  -wi-------   3.99g
  db_lv    web_vg -wi-------   3.00g
  lv_drbd  web_vg -wi-------   1.00g
  wp_lv    web_vg -wi-------   3.00g

However, there is no VG or LV information under /dev directory:

[root@web1 ~]# ll /dev/web_vg
ls: cannot access /dev/web_vg: No such file or directory
[root@web1 ~]# ll /dev/db_vg
ls: cannot access /dev/db_vg: No such file or directory
[root@web1 ~]# ll /dev/mapper/
total 0
lrwxrwxrwx. 1 root root       7 Oct 21 19:17 cl-home -> ../dm-2
lrwxrwxrwx. 1 root root       7 Oct 21 19:17 cl-root -> ../dm-0
lrwxrwxrwx. 1 root root       7 Oct 21 19:17 cl-swap -> ../dm-1
crw-------. 1 root root 10, 236 Oct 21 19:17 control

And all the resources have be halted due to the dev path is not reachable:

[root@web1 ~]# pcs resource show MySQL_fs
 Resource: MySQL_fs (class=ocf provider=heartbeat type=Filesystem)
  Attributes: device=/dev/db_vg/mysql_lv directory=/var/lib/mysql fstype=ext4
  Operations: monitor interval=20 timeout=40 (MySQL_fs-monitor-interval-20)
              notify interval=0s timeout=60 (MySQL_fs-notify-interval-0s)
              start interval=0s timeout=60 (MySQL_fs-start-interval-0s)
              stop interval=0s timeout=60 (MySQL_fs-stop-interval-0s)

I tried to import VG, however, no luck:

[root@web1 ~]# vgchange -ay web_vg
  0 logical volume(s) in volume group "web_vg" now active

Does anyone have any idea on how to address this issue? Thank you.

I found the root cause. We must disable lvm2-lvmetad.socket serive to make sure lvm are manage by pcs.

systemctl disable lvm2-lvmetad.socket

LV Path is Missing

Related

Recent Posts