<div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div class="gmail_default" style="font-family:verdana,sans-serif;color:rgb(102,102,102)">Grant, </div><div class="gmail_default" style="font-family:verdana,sans-serif;color:rgb(102,102,102)"><br></div><div class="gmail_default" style="font-family:verdana,sans-serif;color:rgb(102,102,102)">As a guess I am suspecting your "fail_if_no_path" might be the issue but I am not sure on the inner workings or mechanism at play during the reboot or why it's getting stuck here for you. Your storage vendor may have documentation to state what the multipath (and iscsid) config should be from your host. Before changing config though I recommend getting the root cause realised. /var/log/messages log could help. </div><div class="gmail_default" style="font-family:verdana,sans-serif;color:rgb(102,102,102)"><br></div><div class="gmail_default" style="font-family:verdana,sans-serif;color:rgb(102,102,102)">Also if you go into the multipath CLI "multipathd -k" and issue "show config" you may see a "NETAPP" config there already. Depending on the IDs your storage may be matching that rather than the default config within multipath.conf FYI. </div><div class="gmail_default" style="font-family:verdana,sans-serif;color:rgb(102,102,102)"><br></div><div class="gmail_default" style="font-family:verdana,sans-serif;color:rgb(102,102,102)">Regards,</div><div class="gmail_default" style="font-family:verdana,sans-serif;color:rgb(102,102,102)"><br></div><div><div dir="ltr"><div dir="ltr">Tony Pearce<br><br></div></div></div><br></div></div></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, 28 Oct 2020 at 15:56, Grant Morley <<a href="mailto:grant@civo.com" target="_blank">grant@civo.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div>
<p>Hi Tony,</p>
<p>We are using NetApp SolidFire for our storage. Instances seem to
be in a normal "working" state before we try and reboot them.</p>
<p>I haven't looked into `/usr/bin/rescan-scsi-bus.sh` but I will
now so thanks for that.</p>
<p>We are using multipath but kept it on the defaults so it looks
like only 1 path is being used.</p>
<p>I had a feeling it was down to heavily loaded compute causing the
issue. </p>
<p>The config for iscsi is also the defaults from which openstack
Ansible deployed.</p>
<p>Thanks for your help.</p>
<p>Grant<br>
</p>
<div>On 28/10/2020 02:25, Tony Pearce wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div class="gmail_default" style="font-family:verdana,sans-serif;color:rgb(102,102,102)">Hi
Grant, what storage are you using here? Is the
instance in an apparently "working" state
before you try and reboot it? </div>
<div class="gmail_default" style="font-family:verdana,sans-serif;color:rgb(102,102,102)"><br>
</div>
<div class="gmail_default" style="font-family:verdana,sans-serif;color:rgb(102,102,102)">Have
you looked into `/usr/bin/rescan-scsi-bus.sh`
? Please see this reference link in the first
instance: [1] "When ‘rescan-scsi-bus.sh -i’ is
run, script execute as well a LIP_RESET
(ISSUE_LIP) which may cause a disruption in
I/O on the server and even cause an outage in
case of a system running on heavy load." </div>
<div class="gmail_default" style="font-family:verdana,sans-serif;color:rgb(102,102,102)"><br>
</div>
<div class="gmail_default"><font face="verdana,
sans-serif" color="#666666">Are you using
multipath? Some helpful commands:</font></div>
<div class="gmail_default"><font face="verdana,
sans-serif" color="#666666"><br>
</font></div>
<div class="gmail_default"><font face="verdana,
sans-serif" color="#666666">`tail -f
/var/log/messages | grep multipath`<br>
</font></div>
<div class="gmail_default"><font face="verdana,
sans-serif" color="#666666"><br>
</font></div>
<div class="gmail_default"><font face="verdana,
sans-serif" color="#666666">`multipathd -k`
= will go into mutipath cli. Then while in
the cli:<br>
<div class="gmail_default">show config</div>
<div class="gmail_default">show paths<br>
</div>
<div class="gmail_default"><br>
</div>
<div class="gmail_default">If the cli is
accessible then you're likely using
multipath even if 1 path. Then the
multipath.conf is taking effect even if
it's a default config. </div>
</font></div>
<div class="gmail_default"><font face="verdana,
sans-serif" color="#666666"><br>
</font></div>
<div class="gmail_default"><font face="verdana,
sans-serif" color="#666666">Config files
relating to iscsi storage: <br>
/etc/iscsi/iscsid.conf<br>
</font></div>
<div class="gmail_default"><font face="verdana,
sans-serif" color="#666666">/etc/multipath/multipath.conf </font></div>
<div class="gmail_default"><font face="verdana,
sans-serif" color="#666666"><br>
</font></div>
<div class="gmail_default"><font face="verdana,
sans-serif" color="#666666">[1] <a href="https://www.thegeekdiary.com/when-to-use-rescan-scsi-bus-sh-i-lip-flag-in-centos-rhel/" target="_blank">https://www.thegeekdiary.com/when-to-use-rescan-scsi-bus-sh-i-lip-flag-in-centos-rhel/</a></font><br>
</div>
<div class="gmail_default" style="font-family:verdana,sans-serif;color:rgb(102,102,102)"><br>
</div>
<div>
<div dir="ltr">
<div dir="ltr">
<div class="gmail_default" style="font-family:verdana,sans-serif;color:rgb(102,102,102)">Regards,</div>
<br>
</div>
<div dir="ltr">Tony Pearce<br>
<br>
</div>
</div>
</div>
<br>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Wed, 28 Oct 2020 at 03:39,
Grant Morley <<a href="mailto:grant@civo.com" target="_blank">grant@civo.com</a>> wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div>
<p>Hi all,</p>
<p>We are seeing some oddness on a couple of our compute
hosts that seems to be related to iSCSI. On a couple of
our hosts I am seeing this error in the nova compute logs:</p>
<p>2020-10-27 18:56:14.814 31490 WARNING
os_brick.initiator.connectors.iscsi
[req-8613ae69-1661-49cf-8bdc-6fec875d01ba - - - - -]
Couldn't find iscsi sessions because iscsiadm err:
iscsiadm: could not read session targetname: 5<br>
iscsiadm: could not find session info for session1707</p>
<p>That seems to also stop any instance on the compute host
from being able to reboot. Reboots seem to get accepted
but the instance never completes and gets stuck in the
reboot state:</p>
<p>2020-10-27 19:11:58.891 48612 INFO nova.compute.manager
[-] [instance: c8079e85-4777-4615-9d5a-3d1151e11984]
During sync_power_state the instance has a pending task
(reboot_started_hard). Skip.<br>
2020-10-27 19:11:58.891 48612 INFO nova.compute.manager
[-] [instance: 31128f26-910d-411f-98e0-c95dd36f4f0f]
During sync_power_state the instance has a pending task
(reboot_started_hard). Skip.<br>
</p>
<p>Does anyone know of a way to resolve this without
rebooting the entire compute host? I can't see any other
issues other than the fact there is this iSCSI error which
in turn seems to stop nova from processing anything for
any instance.</p>
<p>Any advice would be much appreciated.</p>
<p>Regards,<br>
</p>
<div>-- <br>
<div dir="auto">
<table width="100%" cellspacing="0" cellpadding="0">
<tbody>
<tr>
<td>
<div style="line-height:15px;color:rgb(109,190,255);margin-bottom:5px"><span style="font-weight:bold;font-size:110%;color:rgb(35,157,255)">Grant
Morley</span></div>
<div style="line-height:15px;color:rgb(136,136,136);margin-bottom:10px">Cloud
Engineer, <span style="color:rgb(35,157,255)">Civo
Ltd</span></div>
<div style="margin-right:110px;line-height:15px;color:rgb(136,136,136);margin-bottom:10px"><span>Unit
H-K, Gateway 1000, Whittle Way<br>
Stevenage, Herts, SG1 2FP, UK</span></div>
</td>
<td valign="top" align="right"><img src="https://www.civo.com/images/email-logo.jpg" style="width: 100px; color: rgb(168, 216, 255); line-height: 15px; float: none; margin-top: 0px; margin-right: 10px;"></td>
</tr>
</tbody>
</table>
<table width="100%" cellspacing="0" cellpadding="0" border="0">
<tbody>
<tr>
<td style="padding:4px;background-color:rgb(35,157,255);color:white" width="*">Visit us at <a href="https://www.civo.com/" style="color:white;line-height:15px;text-decoration:none" target="_blank">www.civo.com</a></td>
<td width="2"><br>
</td>
<td style="text-align:center;padding:4px;background-color:rgb(240,240,240)" width="200"><a href="https://www.civo.com/signup" style="color:rgb(35,157,255);line-height:15px;margin-right:6px;text-decoration:none" target="_blank">Signup
for an account now ></a></td>
</tr>
</tbody>
</table>
</div>
</div>
</div>
</blockquote>
</div>
</blockquote>
<div>-- <br>
<div dir="auto">
<table width="100%" cellspacing="0" cellpadding="0">
<tbody>
<tr>
<td>
<div style="line-height:15px;color:rgb(109,190,255);margin-bottom:5px"><span style="font-weight:bold;font-size:110%;color:rgb(35,157,255)">Grant
Morley</span></div>
<div style="line-height:15px;color:rgb(136,136,136);margin-bottom:10px">Cloud Engineer, <span style="color:rgb(35,157,255)">Civo Ltd</span></div>
<div style="margin-right:110px;line-height:15px;color:rgb(136,136,136);margin-bottom:10px"><span>Unit
H-K, Gateway 1000, Whittle Way<br>
Stevenage, Herts, SG1 2FP, UK</span></div>
</td>
<td valign="top" align="right"><img src="https://www.civo.com/images/email-logo.jpg" style="width: 100px; color: rgb(168, 216, 255); line-height: 15px; float: none; margin-top: 0px; margin-right: 10px;"></td>
</tr>
</tbody>
</table>
<table width="100%" cellspacing="0" cellpadding="0" border="0">
<tbody>
<tr>
<td style="padding:4px;background-color:rgb(35,157,255);color:white" width="*">Visit us at <a href="https://www.civo.com/" style="color:white;line-height:15px;text-decoration:none" target="_blank">www.civo.com</a></td>
<td width="2"><br>
</td>
<td style="text-align:center;padding:4px;background-color:rgb(240,240,240)" width="200"><a href="https://www.civo.com/signup" style="color:rgb(35,157,255);line-height:15px;margin-right:6px;text-decoration:none" target="_blank">Signup for an account now
></a></td>
</tr>
</tbody>
</table>
</div>
</div>
</div>
</blockquote></div>