<div dir="ltr">More info : server is actually running CentOS 7.6 (one of the few that didn't recently get updated)<div><br></div><div>System has 5 disk configured in and md RAID5 set as md126</div><div><font face="monospace">md126 : active raid5 sdf[4] sdb[0] sde[3] sdc[1] sdd[2]<br> 11720536064 blocks super 1.2 level 5, 512k chunk, algorithm 2 [5/5] [UUUUU]<br> bitmap: 6/22 pages [24KB], 65536KB chunk</font><br>LVM filter excludes the sd : filter = [ "r|^/dev/sd[bcdef]|" ]</div><div><br></div><div>boot.log has complaints about 5 dm disks </div><div><font face="monospace">[FAILED] Failed to start LVM2 PV scan on device 253:55.<br>[FAILED] Failed to start LVM2 PV scan on device 253:47.<br>[FAILED] Failed to start LVM2 PV scan on device 253:50.<br>[FAILED] Failed to start LVM2 PV scan on device 253:56.<br>[FAILED] Failed to start LVM2 PV scan on device 253:34.<br></font></div><div><br></div><div>Typical message :</div><div>[<font face="monospace">FAILED] Failed to start LVM2 PV scan on device 253:47.<br>See 'systemctl status lvm2-pvscan@253:47.service' for details.<br></font></div><div><br></div><div>output of systemctl status:</div><div><font face="monospace">systemctl status lvm2-pvscan@253:55.service<br>● lvm2-pvscan@253:55.service - LVM2 PV scan on device 253:55<br> Loaded: loaded (/usr/lib/systemd/system/lvm2-pvscan@.service; static; vendor preset: disabled)<br> Active: failed (Result: exit-code) since Thu 2020-09-24 09:26:58 EDT; 5h 44min ago<br> Docs: man:pvscan(8)<br> Process: 17395 ExecStart=/usr/sbin/lvm pvscan --cache --activate ay %i (code=exited, status=5)<br> Main PID: 17395 (code=exited, status=5)<br><br>Sep 24 09:26:58 stack3 systemd[1]: Starting LVM2 PV scan on device 253:55...<br>Sep 24 09:26:58 stack3 lvm[17395]: Multiple VGs found with the same name: skipping encrypted_vg<br>Sep 24 09:26:58 stack3 lvm[17395]: Use --select vg_uuid=<uuid> in place of the VG name.<br>Sep 24 09:26:58 stack3 systemd[1]: lvm2-pvscan@253:55.service: main process exited, code=exited, status=5/NOTINSTALLED<br>Sep 24 09:26:58 stack3 systemd[1]: Failed to start LVM2 PV scan on device 253:55.<br>Sep 24 09:26:58 stack3 systemd[1]: Unit lvm2-pvscan@253:55.service entered failed state.<br>Sep 24 09:26:58 stack3 systemd[1]: lvm2-pvscan@253:55.service failed.</font><br></div><div><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Thu, Sep 24, 2020 at 2:07 PM Alan Davis <<a href="mailto:alan.davis@apogee-research.com">alan.davis@apogee-research.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">This morning my CentOS 7.7 RDO packstack installation of Rocky hung. On reboot some of the VMs won't start. This is a primary system and I need to find the most expedient way to recover without losing data. I'm not using LVM thin volumes.<div><br></div><div>Any help is appreciated.</div><div><br><div>Looking at nova-compute.log I see errors trying to find LUN 0 during the sysfs stage.</div><div><br></div><div>Several machines won't boot because their root disk entries in LVM are seen as PV and booting them doesn't see them in the DM subsystem.</div><div>Other machines boot but there attached disks throw LVM errors about duplicate PV and preferring the cinder-volumes VG version.</div><div><br></div><div>LVM is showing LVs that have both "bare" entries as well as entries in cinder-volumes and it's complaining about duplicate PVs, not using lvmetad and preferring some entries because they are in the dm subsystem.</div><div>I've verified that, so far, I haven't lost any data. The "bare" LV not being used as part of the DM subsystem because it's server won't boot can be mounted on the openstack host and all data on it is accessible.</div><div><br></div><div>This host has rebooted cleanly multiple times in the past. This is the first time it's shown any problems.</div><div><br></div><div>Am I missing an LVM filter? (unlikely since it wasn't neede before)</div><div>How can I reset the LVM configuration and convince it that it's not seeing duplicate PV?</div><div>How do I ensure that openstack sees the right UUID and volume ID?</div><div><br></div><div>Excerpts from error log and output of lvs :</div><div>--- nova-compute.log --- during VM start</div><div>2020-09-24 11:15:27.091 13953 INFO os_brick.initiator.connectors.iscsi [req-8d15fb6a-6324-471e-9497-587885eef8f6 396aeda6552f44fdac5f878b90325ee1 54af92f2bb494355b96024076184d1c8<br> - default default] Trying to connect to iSCSI portal <a href="http://172.10.0.40:3260" target="_blank">172.10.0.40:3260</a><br>2020-09-24 11:15:29.721 13953 WARNING nova.compute.manager [req-fd32e16f-c879-402f-a32c-6be45a943c34 48af9a366301467d9fec912fd1c072c6 f9fc7b412a8446d083da1356aa370eb4 - default d<br>efault] [instance: de7d740c-786a-4aa2-aa09-d447ae7e14b6] Received unexpected event network-vif-unplugged-79aff403-d2e4-4266-bd88-d7bd19d501a9 for instance with vm_state stopped a<br>nd task_state powering-on.<br>2020-09-24 11:16:21.361 13953 WARNING os_brick.initiator.connectors.iscsi [req-8d15fb6a-6324-471e-9497-587885eef8f6 396aeda6552f44fdac5f878b90325ee1 54af92f2bb494355b96024076184d<br>1c8 - default default] LUN 0 on iSCSI portal <a href="http://172.10.0.40:3260" target="_blank">172.10.0.40:3260</a> not found on sysfs after logging in.<br>2020-09-24 11:16:23.482 13953 INFO os_brick.initiator.connectors.iscsi [req-8d15fb6a-6324-471e-9497-587885eef8f6 396aeda6552f44fdac5f878b90325ee1 54af92f2bb494355b96024076184d1c8<br> - default default] Trying to connect to iSCSI portal <a href="http://172.10.0.40:3260" target="_blank">172.10.0.40:3260</a><br>2020-09-24 11:17:17.741 13953 WARNING os_brick.initiator.connectors.iscsi [req-8d15fb6a-6324-471e-9497-587885eef8f6 396aeda6552f44fdac5f878b90325ee1 54af92f2bb494355b96024076184d<br>1c8 - default default] LUN 0 on iSCSI portal <a href="http://172.10.0.40:3260" target="_blank">172.10.0.40:3260</a> not found on sysfs after logging in.: VolumeDeviceNotFound: Volume device not found at .<br>2020-09-24 11:17:21.864 13953 INFO os_brick.initiator.connectors.iscsi [req-8d15fb6a-6324-471e-9497-587885eef8f6 396aeda6552f44fdac5f878b90325ee1 54af92f2bb494355b96024076184d1c8<br> - default default] Trying to connect to iSCSI portal <a href="http://172.10.0.40:3260" target="_blank">172.10.0.40:3260</a><br>2020-09-24 11:18:16.113 13953 WARNING os_brick.initiator.connectors.iscsi [req-8d15fb6a-6324-471e-9497-587885eef8f6 396aeda6552f44fdac5f878b90325ee1 54af92f2bb494355b96024076184d<br>1c8 - default default] LUN 0 on iSCSI portal <a href="http://172.10.0.40:3260" target="_blank">172.10.0.40:3260</a> not found on sysfs after logging in.: VolumeDeviceNotFound: Volume device not found at .<br>2020-09-24 11:18:17.252 13953 INFO nova.compute.manager [req-8d15fb6a-6324-471e-9497-587885eef8f6 396aeda6552f44fdac5f878b90325ee1 54af92f2bb494355b96024076184d1c8 - default defa<br>ult] [instance: de7d740c-786a-4aa2-aa09-d447ae7e14b6] Successfully reverted task state from powering-on on failure for instance.<br>2020-09-24 11:18:17.279 13953 ERROR oslo_messaging.rpc.server [req-8d15fb6a-6324-471e-9497-587885eef8f6 396aeda6552f44fdac5f878b90325ee1 54af92f2bb494355b96024076184d1c8 - defaul<br>t default] Exception during message handling: VolumeDeviceNotFound: Volume device not found at .<br>2020-09-24 11:18:17.279 13953 ERROR oslo_messaging.rpc.server Traceback (most recent call last):<br>2020-09-24 11:18:17.279 13953 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 163, in _process_incoming<br>2020-09-24 11:18:17.279 13953 ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message)<br><div><br></div><div> </div><div><font face="monospace">--- lvs output ---</font></div><div><font face="monospace">I've annotated 1 machine's disks to illustrate the relationship between the volume-*** cinder-volumes vg entries and the "bare" lv seen as directly accessible from the host.<br>There are 3 servers that won't boot, they are the one's who's home/vg_home and encrypted_home/encrypted_vg entries are shown.<br><br> WARNING: Not using lvmetad because duplicate PVs were found.<br> WARNING: Use multipath or vgimportclone to resolve duplicate PVs?<br> WARNING: After duplicates are resolved, run "pvscan --cache" to enable lvmetad.<br> WARNING: Not using device /dev/sdu for PV yZy8Xk-foKT-ovjV-0EZv-VxEM-GqiP-WH7k53. == backup_lv/encrypted_vg<br> WARNING: Not using device /dev/sdv for PV tHA9ui-eSIO-MDmI-RM3u-3Bf4-Dznb-Ha3XfP. == varoptgitlab/encrypted_vg<br> WARNING: Not using device /dev/sdm for PV 5eoyCa-sMO4-b7O4-jIfh-byZE-L5pS-3lOu0D.<br> WARNING: Not using device /dev/sdp for PV 3BI0nV-TP0k-rgPC-PrjH-FT7z-reMe-ec1spj.<br> WARNING: Not using device /dev/sdt for PV ILdbcY-VFCm-fnH6-Y3jc-pdWZ-fnl8-PH3TPe. == storage_lv/encrypted_vg<br> WARNING: Not using device /dev/sdr for PV zowU2N-oaBh-r4cO-cxgX-YYiq-Kf3q-mqlHfK.<br> WARNING: PV yZy8Xk-foKT-ovjV-0EZv-VxEM-GqiP-WH7k53 prefers device /dev/cinder-volumes/volume-c8da1abf-7143-422c-9ee5-b2724a71c8ff because device is in dm subsystem.<br> WARNING: PV tHA9ui-eSIO-MDmI-RM3u-3Bf4-Dznb-Ha3XfP prefers device /dev/cinder-volumes/volume-0a12012f-8c2e-41fb-aa0c-a7ae99c62487 because device is in dm subsystem.<br> WARNING: PV 5eoyCa-sMO4-b7O4-jIfh-byZE-L5pS-3lOu0D prefers device /dev/cinder-volumes/volume-990a057c-46cc-4a81-ba02-28b72c34791d because device is in dm subsystem.<br> WARNING: PV 3BI0nV-TP0k-rgPC-PrjH-FT7z-reMe-ec1spj prefers device /dev/cinder-volumes/volume-b6a9da6e-1958-46ea-90b4-ac1aebed8c04 because device is in dm subsystem.<br> WARNING: PV ILdbcY-VFCm-fnH6-Y3jc-pdWZ-fnl8-PH3TPe prefers device /dev/cinder-volumes/volume-302dd53b-7d05-4f6d-9ada-8f2ed6e1d4c6 because device is in dm subsystem.<br> WARNING: PV zowU2N-oaBh-r4cO-cxgX-YYiq-Kf3q-mqlHfK prefers device /dev/cinder-volumes/volume-df006472-be7a-4957-972a-1db4463f5d67 because device is in dm subsystem.<br> LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert<br> home centos_stack3 -wi-ao---- 4.00g <br> root centos_stack3 -wi-ao---- 50.00g <br> swap centos_stack3 -wi-ao---- 4.00g <br> _snapshot-05b1e46b-1ae3-4cd0-9117-3fb53a6d94b0 cinder-volumes swi-a-s--- 20.00g volume-1d0ff5d5-93a3-44e8-8bfa-a9290765c8c6 0.00 <br> lv_filestore cinder-volumes -wi-ao---- 1.00t <br>...<br> volume-c8da1abf-7143-422c-9ee5-b2724a71c8ff cinder-volumes -wi-ao---- 100.00g <br> volume-0a12012f-8c2e-41fb-aa0c-a7ae99c62487 cinder-volumes -wi-ao---- 60.00g <br> volume-990a057c-46cc-4a81-ba02-28b72c34791d cinder-volumes -wi-ao---- 200.00g <br> volume-b6a9da6e-1958-46ea-90b4-ac1aebed8c04 cinder-volumes -wi-ao---- 30.00g <br> volume-302dd53b-7d05-4f6d-9ada-8f2ed6e1d4c6 cinder-volumes -wi-ao---- 60.00g <br> volume-df006472-be7a-4957-972a-1db4463f5d67 cinder-volumes -wi-ao---- 250.00g <br>...<br> volume-f3250e15-bb9c-43d1-989d-8a8f6635a416 cinder-volumes -wi-ao---- 20.00g <br> volume-fc1d5fcb-fda1-456b-a89d-582b7f94fb04 cinder-volumes -wi-ao---- 300.00g <br> volume-fc50a717-0857-4da3-93cb-a55292f7ed6d cinder-volumes -wi-ao---- 20.00g <br> volume-ff94e2d6-449b-495d-82e6-0debd694c1dd cinder-volumes -wi-ao---- 20.00g <br> data2 data2_vg -wi-a----- <300.00g <br> data data_vg -wi-a----- 1.79t <br> backup_lv encrypted_vg -wi------- <100.00g == ...WH7k53 <br> storage_lv encrypted_vg -wi------- <60.00g == ...PH3TPe <br> varoptgitlab_lv encrypted_vg -wi------- <200.00g <br> varoptgitlab_lv encrypted_vg -wi------- <30.00g <br> varoptgitlab_lv encrypted_vg -wi------- <60.00g == ...Ha3XfP<br> encrypted_home home_vg -wi-a----- <40.00g <br> encrypted_home home_vg -wi------- <60.00g <br> pub pub_vg -wi-a----- <40.00g <br> pub_lv pub_vg -wi------- <250.00g <br> rpms repo -wi-a----- 499.99g <br> home vg_home -wi-a----- <40.00g <br> gtri_pub vg_pub -wi-a----- 20.00g <br> pub vg_pub -wi-a----- <40.00g <br></font></div><font face="monospace">-- </font><br><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr">Alan Davis<div>Principal System Administrator</div><div>Apogee Research LLC</div><div><br></div></div></div></div></div></div></div></div>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr" class="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr">Alan Davis<div>Principal System Administrator</div><div>Apogee Research LLC</div><div>Office : 571.384.8941 x26</div><div>Cell : 410.701.0518</div></div></div></div></div></div></div>