<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
</head>
<body>
<p>Hi Tony,</p>
<p>We are using NetApp SolidFire for our storage. Instances seem to
be in a normal "working" state before we try and reboot them.</p>
<p>I haven't looked into `/usr/bin/rescan-scsi-bus.sh` but I will
now so thanks for that.</p>
<p>We are using multipath but kept it on the defaults so it looks
like only 1 path is being used.</p>
<p>I had a feeling it was down to heavily loaded compute causing the
issue. </p>
<p>The config for iscsi is also the defaults from which openstack
Ansible deployed.</p>
<p>Thanks for your help.</p>
<p>Grant<br>
</p>
<div class="moz-cite-prefix">On 28/10/2020 02:25, Tony Pearce wrote:<br>
</div>
<blockquote type="cite"
cite="mid:CAK1+d=5m2LJM12565nPOsafi3SvMmpX+okwFSnWoTUV=qHSVXA@mail.gmail.com">
<meta http-equiv="content-type" content="text/html; charset=UTF-8">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div class="gmail_default"
style="font-family:verdana,sans-serif;color:rgb(102,102,102)">Hi
Grant, what storage are you using here? Is the
instance in an apparently "working" state
before you try and reboot it? </div>
<div class="gmail_default"
style="font-family:verdana,sans-serif;color:rgb(102,102,102)"><br>
</div>
<div class="gmail_default"
style="font-family:verdana,sans-serif;color:rgb(102,102,102)">Have
you looked into `/usr/bin/rescan-scsi-bus.sh`
? Please see this reference link in the first
instance: [1] "When ‘rescan-scsi-bus.sh -i’ is
run, script execute as well a LIP_RESET
(ISSUE_LIP) which may cause a disruption in
I/O on the server and even cause an outage in
case of a system running on heavy load." </div>
<div class="gmail_default"
style="font-family:verdana,sans-serif;color:rgb(102,102,102)"><br>
</div>
<div class="gmail_default"><font face="verdana,
sans-serif" color="#666666">Are you using
multipath? Some helpful commands:</font></div>
<div class="gmail_default"><font face="verdana,
sans-serif" color="#666666"><br>
</font></div>
<div class="gmail_default"><font face="verdana,
sans-serif" color="#666666">`tail -f
/var/log/messages | grep multipath`<br>
</font></div>
<div class="gmail_default"><font face="verdana,
sans-serif" color="#666666"><br>
</font></div>
<div class="gmail_default"><font face="verdana,
sans-serif" color="#666666">`multipathd -k`
= will go into mutipath cli. Then while in
the cli:<br>
<div class="gmail_default">show config</div>
<div class="gmail_default">show paths<br>
</div>
<div class="gmail_default"><br>
</div>
<div class="gmail_default">If the cli is
accessible then you're likely using
multipath even if 1 path. Then the
multipath.conf is taking effect even if
it's a default config. </div>
</font></div>
<div class="gmail_default"><font face="verdana,
sans-serif" color="#666666"><br>
</font></div>
<div class="gmail_default"><font face="verdana,
sans-serif" color="#666666">Config files
relating to iscsi storage: <br>
/etc/iscsi/iscsid.conf<br>
</font></div>
<div class="gmail_default"><font face="verdana,
sans-serif" color="#666666">/etc/multipath/multipath.conf </font></div>
<div class="gmail_default"><font face="verdana,
sans-serif" color="#666666"><br>
</font></div>
<div class="gmail_default"><font face="verdana,
sans-serif" color="#666666">[1] <a
href="https://www.thegeekdiary.com/when-to-use-rescan-scsi-bus-sh-i-lip-flag-in-centos-rhel/"
moz-do-not-send="true">https://www.thegeekdiary.com/when-to-use-rescan-scsi-bus-sh-i-lip-flag-in-centos-rhel/</a></font><br>
</div>
<div class="gmail_default"
style="font-family:verdana,sans-serif;color:rgb(102,102,102)"><br>
</div>
<div>
<div dir="ltr" class="gmail_signature">
<div dir="ltr">
<div class="gmail_default"
style="font-family:verdana,sans-serif;color:rgb(102,102,102)">Regards,</div>
<br>
</div>
<div dir="ltr">Tony Pearce<br>
<br>
</div>
</div>
</div>
<br>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Wed, 28 Oct 2020 at 03:39,
Grant Morley <<a href="mailto:grant@civo.com"
moz-do-not-send="true">grant@civo.com</a>> wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px
0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div>
<p>Hi all,</p>
<p>We are seeing some oddness on a couple of our compute
hosts that seems to be related to iSCSI. On a couple of
our hosts I am seeing this error in the nova compute logs:</p>
<p>2020-10-27 18:56:14.814 31490 WARNING
os_brick.initiator.connectors.iscsi
[req-8613ae69-1661-49cf-8bdc-6fec875d01ba - - - - -]
Couldn't find iscsi sessions because iscsiadm err:
iscsiadm: could not read session targetname: 5<br>
iscsiadm: could not find session info for session1707</p>
<p>That seems to also stop any instance on the compute host
from being able to reboot. Reboots seem to get accepted
but the instance never completes and gets stuck in the
reboot state:</p>
<p>2020-10-27 19:11:58.891 48612 INFO nova.compute.manager
[-] [instance: c8079e85-4777-4615-9d5a-3d1151e11984]
During sync_power_state the instance has a pending task
(reboot_started_hard). Skip.<br>
2020-10-27 19:11:58.891 48612 INFO nova.compute.manager
[-] [instance: 31128f26-910d-411f-98e0-c95dd36f4f0f]
During sync_power_state the instance has a pending task
(reboot_started_hard). Skip.<br>
</p>
<p>Does anyone know of a way to resolve this without
rebooting the entire compute host? I can't see any other
issues other than the fact there is this iSCSI error which
in turn seems to stop nova from processing anything for
any instance.</p>
<p>Any advice would be much appreciated.</p>
<p>Regards,<br>
</p>
<div>-- <br>
<div dir="auto">
<table width="100%" cellspacing="0" cellpadding="0">
<tbody>
<tr>
<td>
<div
style="line-height:15px;color:rgb(109,190,255);margin-bottom:5px"><span
style="font-weight:bold;font-size:110%;color:rgb(35,157,255)">Grant
Morley</span></div>
<div
style="line-height:15px;color:rgb(136,136,136);margin-bottom:10px">Cloud
Engineer, <span style="color:rgb(35,157,255)">Civo
Ltd</span></div>
<div
style="margin-right:110px;line-height:15px;color:rgb(136,136,136);margin-bottom:10px"><span>Unit
H-K, Gateway 1000, Whittle Way<br>
Stevenage, Herts, SG1 2FP, UK</span></div>
</td>
<td valign="top" align="right"><img
src="https://www.civo.com/images/email-logo.jpg"
style="width: 100px; color: rgb(168, 216,
255); line-height: 15px; float: none;
margin-top: 0px; margin-right: 10px;"
moz-do-not-send="true"></td>
</tr>
</tbody>
</table>
<table width="100%" cellspacing="0" cellpadding="0"
border="0">
<tbody>
<tr>
<td
style="padding:4px;background-color:rgb(35,157,255);color:white"
width="*">Visit us at <a
href="https://www.civo.com/"
style="color:white;line-height:15px;text-decoration:none"
target="_blank" moz-do-not-send="true">www.civo.com</a></td>
<td width="2"><br>
</td>
<td
style="text-align:center;padding:4px;background-color:rgb(240,240,240)"
width="200"><a
href="https://www.civo.com/signup"
style="color:rgb(35,157,255);line-height:15px;margin-right:6px;text-decoration:none"
target="_blank" moz-do-not-send="true">Signup
for an account now ></a></td>
</tr>
</tbody>
</table>
</div>
</div>
</div>
</blockquote>
</div>
</blockquote>
<div class="moz-signature">-- <br>
<div dir="auto">
<table width="100%" cellspacing="0" cellpadding="0">
<tbody>
<tr>
<td>
<div style="line-height: 15px; color: rgb(109, 190,
255); margin-bottom: 5px;"><span style="font-weight:
bold; font-size: 110%; color: rgb(35, 157, 255);">Grant
Morley</span></div>
<div style="line-height: 15px; color: #888888;
margin-bottom: 10px;">Cloud Engineer, <span
style="color: rgb(35, 157, 255)">Civo Ltd</span></div>
<div style="margin-right: 110px; line-height: 15px;
color: #888888; margin-bottom: 10px;"><span style="">Unit
H-K, Gateway 1000, Whittle Way<br>
Stevenage, Herts, SG1 2FP, UK</span></div>
</td>
<td valign="top" align="right"><img
src="https://www.civo.com/images/email-logo.jpg"
style="width: 100px; color: rgb(168, 216, 255);
line-height: 15px; float: none; margin-top: 0px;
margin-right: 10px;"></td>
</tr>
</tbody>
</table>
<table width="100%" cellspacing="0" cellpadding="0" border="0">
<tbody>
<tr>
<td style="padding: 4px; background-color: rgb(35, 157,
255); color: white;" width="*">Visit us at <a
href="https://www.civo.com/" style="color: white;
line-height: 15px; text-decoration: none;">www.civo.com</a></td>
<td width="2"><br>
</td>
<td style="text-align: center; padding: 4px;
background-color: rgb(240, 240, 240);
background-position: initial initial; background-repeat:
initial initial;" width="200"><a
href="https://www.civo.com/signup" style="color:
rgb(35, 157, 255); line-height: 15px; margin-right:
6px; text-decoration: none;">Signup for an account now
></a></td>
</tr>
</tbody>
</table>
</div>
</div>
</body>
</html>