<div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div class="gmail_default" style="font-family:verdana,sans-serif;color:rgb(102,102,102)">Hi Grant, what storage are you using here? Is the instance in an apparently "working" state before you try and reboot it? </div><div class="gmail_default" style="font-family:verdana,sans-serif;color:rgb(102,102,102)"><br></div><div class="gmail_default" style="font-family:verdana,sans-serif;color:rgb(102,102,102)">Have you looked into `/usr/bin/rescan-scsi-bus.sh` ? Please see this reference link in the first instance: [1] "When ‘rescan-scsi-bus.sh -i’ is run, script execute as well a LIP_RESET (ISSUE_LIP) which may cause a disruption in I/O on the server and even cause an outage in case of a system running on heavy load." </div><div class="gmail_default" style="font-family:verdana,sans-serif;color:rgb(102,102,102)"><br></div><div class="gmail_default"><font color="#666666" face="verdana, sans-serif">Are you using multipath? Some helpful commands:</font></div><div class="gmail_default"><font color="#666666" face="verdana, sans-serif"><br></font></div><div class="gmail_default"><font color="#666666" face="verdana, sans-serif">`tail -f /var/log/messages | grep multipath`<br></font></div><div class="gmail_default"><font color="#666666" face="verdana, sans-serif"><br></font></div><div class="gmail_default"><font color="#666666" face="verdana, sans-serif">`multipathd -k` = will go into mutipath cli. Then while in the cli:<br><div class="gmail_default">show config</div><div class="gmail_default">show paths<br></div><div class="gmail_default"><br></div><div class="gmail_default">If the cli is accessible then you're likely using multipath even if 1 path. Then the multipath.conf is taking effect even if it's a default config. </div></font></div><div class="gmail_default"><font color="#666666" face="verdana, sans-serif"><br></font></div><div class="gmail_default"><font color="#666666" face="verdana, sans-serif">Config files relating to iscsi storage: <br>/etc/iscsi/iscsid.conf<br></font></div><div class="gmail_default"><font color="#666666" face="verdana, sans-serif">/etc/multipath/multipath.conf </font></div><div class="gmail_default"><font color="#666666" face="verdana, sans-serif"><br></font></div><div class="gmail_default"><font color="#666666" face="verdana, sans-serif">[1] <a href="https://www.thegeekdiary.com/when-to-use-rescan-scsi-bus-sh-i-lip-flag-in-centos-rhel/">https://www.thegeekdiary.com/when-to-use-rescan-scsi-bus-sh-i-lip-flag-in-centos-rhel/</a></font><br></div><div class="gmail_default" style="font-family:verdana,sans-serif;color:rgb(102,102,102)"><br></div><div><div dir="ltr" class="gmail_signature"><div dir="ltr"><div class="gmail_default" style="font-family:verdana,sans-serif;color:rgb(102,102,102)">Regards,</div><br></div><div dir="ltr">Tony Pearce<br><br></div></div></div><br></div></div></div></div></div></div></div></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, 28 Oct 2020 at 03:39, Grant Morley <<a href="mailto:grant@civo.com">grant@civo.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div>
<p>Hi all,</p>
<p>We are seeing some oddness on a couple of our compute hosts that
seems to be related to iSCSI. On a couple of our hosts I am seeing
this error in the nova compute logs:</p>
<p>2020-10-27 18:56:14.814 31490 WARNING
os_brick.initiator.connectors.iscsi
[req-8613ae69-1661-49cf-8bdc-6fec875d01ba - - - - -] Couldn't find
iscsi sessions because iscsiadm err: iscsiadm: could not read
session targetname: 5<br>
iscsiadm: could not find session info for session1707</p>
<p>That seems to also stop any instance on the compute host from
being able to reboot. Reboots seem to get accepted but the
instance never completes and gets stuck in the reboot state:</p>
<p>2020-10-27 19:11:58.891 48612 INFO nova.compute.manager [-]
[instance: c8079e85-4777-4615-9d5a-3d1151e11984] During
sync_power_state the instance has a pending task
(reboot_started_hard). Skip.<br>
2020-10-27 19:11:58.891 48612 INFO nova.compute.manager [-]
[instance: 31128f26-910d-411f-98e0-c95dd36f4f0f] During
sync_power_state the instance has a pending task
(reboot_started_hard). Skip.<br>
</p>
<p>Does anyone know of a way to resolve this without rebooting the
entire compute host? I can't see any other issues other than the
fact there is this iSCSI error which in turn seems to stop nova
from processing anything for any instance.</p>
<p>Any advice would be much appreciated.</p>
<p>Regards,<br>
</p>
<div>-- <br>
<div dir="auto">
<table width="100%" cellspacing="0" cellpadding="0">
<tbody>
<tr>
<td>
<div style="line-height:15px;color:rgb(109,190,255);margin-bottom:5px"><span style="font-weight:bold;font-size:110%;color:rgb(35,157,255)">Grant
Morley</span></div>
<div style="line-height:15px;color:rgb(136,136,136);margin-bottom:10px">Cloud Engineer, <span style="color:rgb(35,157,255)">Civo Ltd</span></div>
<div style="margin-right:110px;line-height:15px;color:rgb(136,136,136);margin-bottom:10px"><span>Unit
H-K, Gateway 1000, Whittle Way<br>
Stevenage, Herts, SG1 2FP, UK</span></div>
</td>
<td valign="top" align="right"><img src="https://www.civo.com/images/email-logo.jpg" style="width: 100px; color: rgb(168, 216, 255); line-height: 15px; float: none; margin-top: 0px; margin-right: 10px;"></td>
</tr>
</tbody>
</table>
<table width="100%" cellspacing="0" cellpadding="0" border="0">
<tbody>
<tr>
<td style="padding:4px;background-color:rgb(35,157,255);color:white" width="*">Visit us at <a href="https://www.civo.com/" style="color:white;line-height:15px;text-decoration:none" target="_blank">www.civo.com</a></td>
<td width="2"><br>
</td>
<td style="text-align:center;padding:4px;background-color:rgb(240,240,240)" width="200"><a href="https://www.civo.com/signup" style="color:rgb(35,157,255);line-height:15px;margin-right:6px;text-decoration:none" target="_blank">Signup for an account now
></a></td>
</tr>
</tbody>
</table>
</div>
</div>
</div>
</blockquote></div>