<div dir="auto">It might not be the compute nodes. I would check the rabbitmq, neutron and nova logs on the controllers.<div dir="auto"><br></div><div dir="auto"><br></div><div dir="auto">Paras</div></div><div class="gmail_extra"><br><div class="gmail_quote">On Dec 18, 2017 5:30 PM, "Volodymyr Litovka" <<a href="mailto:doka.ua@gmx.com">doka.ua@gmx.com</a>> wrote:<br type="attribution"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div text="#000000" bgcolor="#FFFFFF">
Hi Jim,<br>
<br>
switch debug to true in nova.conf and check <b>also</b> other logs
- nova-scheduler, nova-placement, nova-conductor.<br>
<br>
<div class="m_7988713619055522055moz-cite-prefix">On 12/19/17 12:54 AM, Jim Okken wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">
<div class="gmail_extra">hi list,</div>
<div class="gmail_extra"><br>
</div>
<div class="gmail_extra">hoping someone could shed some light on
this issue I just started seeing today</div>
<div class="gmail_extra"><br>
</div>
<div class="gmail_extra">all my compute nodes started showing as
"Down" in the Horizon -> Hypervisors -> Compute Nodes
tab</div>
<div class="gmail_extra"><br>
</div>
<div class="gmail_extra"><br>
</div>
<div class="gmail_extra">
<div class="gmail_extra">root@node-1:~# nova service-list</div>
<div class="gmail_extra">+-----+------------------+----<wbr>---------------+----------+---<wbr>------+-------+---------------<wbr>-------------+----------------<wbr>-+</div>
<div class="gmail_extra">| Id | Binary | Host
| Zone | Status | State | Updated_at
| Disabled Reason |</div>
<div class="gmail_extra">+-----+------------------+----<wbr>---------------+----------+---<wbr>------+-------+---------------<wbr>-------------+----------------<wbr>-+</div>
<div class="gmail_extra">| 325 | nova-compute | <a href="http://node-9.mydom.com" target="_blank">node-9.mydom.com</a>
| nova | enabled | down | 2017-12-18T21:59:38.000000 |
- |</div>
<div class="gmail_extra">| 448 | nova-compute | <a href="http://node-14.mydom.com" target="_blank">node-14.mydom.com</a>
| nova | enabled | up | 2017-12-18T22:41:42.000000 |
- |</div>
<div class="gmail_extra">| 451 | nova-compute | <a href="http://node-17.mydom.com" target="_blank">node-17.mydom.com</a>
| nova | enabled | up | 2017-12-18T22:42:04.000000 |
- |</div>
<div class="gmail_extra">| 454 | nova-compute | <a href="http://node-11.mydom.com" target="_blank">node-11.mydom.com</a>
| nova | enabled | up | 2017-12-18T22:42:02.000000 |
- |</div>
<div class="gmail_extra">| 457 | nova-compute | <a href="http://node-12.mydom.com" target="_blank">node-12.mydom.com</a>
| nova | enabled | up | 2017-12-18T22:42:12.000000 |
- |</div>
<div class="gmail_extra">| 472 | nova-compute | <a href="http://node-16.mydom.com" target="_blank">node-16.mydom.com</a>
| nova | enabled | down | 2017-12-18T00:16:01.000000 |
- |</div>
<div class="gmail_extra">| 475 | nova-compute | <a href="http://node-10.mydom.com" target="_blank">node-10.mydom.com</a>
| nova | enabled | down | 2017-12-18T00:26:09.000000 |
- |</div>
<div class="gmail_extra">| 478 | nova-compute | <a href="http://node-13.mydom.com" target="_blank">node-13.mydom.com</a>
| nova | enabled | down | 2017-12-17T23:54:06.000000 |
- |</div>
<div class="gmail_extra">| 481 | nova-compute | <a href="http://node-15.mydom.com" target="_blank">node-15.mydom.com</a>
| nova | enabled | up | 2017-12-18T22:41:34.000000 |
- |</div>
<div class="gmail_extra">| 484 | nova-compute | <a href="http://node-8.mydom.com" target="_blank">node-8.mydom.com</a>
| nova | enabled | down | 2017-12-17T23:55:50.000000 |
- |</div>
<div><br>
</div>
</div>
<div class="gmail_extra"><br>
</div>
<div class="gmail_extra">if I stop and the start nova-compute on
the down nodes the stop will take several minutes and then the
start will be quick and fine. but after about 2 hours the
nova-compute service will show down again.</div>
<div class="gmail_extra"><br>
</div>
<div class="gmail_extra">i am not seeing any ERRORS in nova
logs.</div>
<div class="gmail_extra"><br>
</div>
<div class="gmail_extra">I get this for the status of a node
that is showing as "UP"<br>
<br>
<div class="gmail_extra">
<div class="gmail_extra"><br>
</div>
<div class="gmail_extra"><br>
</div>
<div class="gmail_extra">root@node-14:~# systemctl status
nova-compute.service</div>
<div class="gmail_extra">â nova-compute.service - OpenStack
Compute</div>
<div class="gmail_extra"> Loaded: loaded
(/lib/systemd/system/nova-<wbr>compute.service; enabled; vendor
preset: enabled)</div>
<div class="gmail_extra"> Active: active (running) since
Mon 2017-12-18 21:57:10 UTC; 35min ago</div>
<div class="gmail_extra"> Docs: man:nova-compute(1)</div>
<div class="gmail_extra"> Process: 32193
ExecStartPre=/bin/chown nova:adm /var/log/nova
(code=exited, status=0/SUCCESS)</div>
<div class="gmail_extra"> Process: 32190
ExecStartPre=/bin/chown nova:nova /var/lock/nova
/var/lib/nova (code=exited, status=0/SUCCESS)</div>
<div class="gmail_extra"> Process: 32187
ExecStartPre=/bin/mkdir -p /var/lock/nova /var/log/nova
/var/lib/nova (code=exited, status=0/SUCCESS)</div>
<div class="gmail_extra"> Main PID: 32196 (nova-compute)</div>
<div class="gmail_extra"> CGroup:
/system.slice/nova-compute.<wbr>service</div>
<div class="gmail_extra"> ââ32196 /usr/bin/python
/usr/bin/nova-compute
--config-file=/etc/nova/nova-<wbr>compute.conf
--config-file=/etc/nova/nova.<wbr>conf
--log-file=/var/log/nova/nova-<wbr>compute.log</div>
<div class="gmail_extra"><br>
</div>
<div class="gmail_extra">Dec 18 22:31:47 <a href="http://node-14.mydom.com" target="_blank">node-14.mydom.com</a>
nova-compute[32196]: 2017-12-18 22:31:47.570 32196 DEBUG
oslo_messaging._drivers.<wbr>amqpdriver
[req-f30b2331-2097-4981-89c8-<wbr>acea4a81f7f2 - - - - -] CALL
msg_id: 2877b9707da144f3a91e7b80e2705f<wbr>b3 exchange 'nova'
topic 'conductor' _send
/usr/lib/python2.7/dist-<wbr>packages/oslo_messaging/_<wbr>drivers/amqpdriver.py:448</div>
<div class="gmail_extra">Dec 18 22:31:47 <a href="http://node-14.mydom.com" target="_blank">node-14.mydom.com</a>
nova-compute[32196]: 2017-12-18 22:31:47.604 32196 DEBUG
oslo_messaging._drivers.<wbr>amqpdriver [-] received reply
msg_id: 2877b9707da144f3a91e7b80e2705f<wbr>b3 __call__
/usr/lib/python2.7/dist-<wbr>packages/oslo_messaging/_<wbr>drivers/amqpdriver.py:296</div>
<div class="gmail_extra">Dec 18 22:31:47 <a href="http://node-14.mydom.com" target="_blank">node-14.mydom.com</a>
nova-compute[32196]: 2017-12-18 22:31:47.605 32196 INFO
nova.compute.resource_tracker
[req-f30b2331-2097-4981-89c8-<wbr>acea4a81f7f2 - - - - -] Total
usable vcpus: 40, total allocated vcpus: 0</div>
<div class="gmail_extra">Dec 18 22:31:47 <a href="http://node-14.mydom.com" target="_blank">node-14.mydom.com</a>
nova-compute[32196]: 2017-12-18 22:31:47.606 32196 INFO
nova.compute.resource_tracker
[req-f30b2331-2097-4981-89c8-<wbr>acea4a81f7f2 - - - - -] Final
resource view: name=<a href="http://node-14.mydom.com" target="_blank">node-14.mydom.com</a>
phys_ram=128812MB used_ram=512MB phys_disk=6691GB
used_disk=0GB total_vcpus=40 used_vcpus=0 pci_stats=[]</div>
<div class="gmail_extra">Dec 18 22:31:47 <a href="http://node-14.mydom.com" target="_blank">node-14.mydom.com</a>
nova-compute[32196]: 2017-12-18 22:31:47.610 32196 DEBUG
oslo_messaging._drivers.<wbr>amqpdriver
[req-f30b2331-2097-4981-89c8-<wbr>acea4a81f7f2 - - - - -] CALL
msg_id: ad32abe833f4440d86c15b911aa35c<wbr>43 exchange 'nova'
topic 'conductor' _send
/usr/lib/python2.7/dist-<wbr>packages/oslo_messaging/_<wbr>drivers/amqpdriver.py:448</div>
<div class="gmail_extra">Dec 18 22:31:47 <a href="http://node-14.mydom.com" target="_blank">node-14.mydom.com</a>
nova-compute[32196]: 2017-12-18 22:31:47.632 32196 DEBUG
oslo_messaging._drivers.<wbr>amqpdriver [-] received reply
msg_id: ad32abe833f4440d86c15b911aa35c<wbr>43 __call__
/usr/lib/python2.7/dist-<wbr>packages/oslo_messaging/_<wbr>drivers/amqpdriver.py:296</div>
<div class="gmail_extra">Dec 18 22:31:47 <a href="http://node-14.mydom.com" target="_blank">node-14.mydom.com</a>
nova-compute[32196]: 2017-12-18 22:31:47.633 32196 WARNING
nova.scheduler.client.report
[req-f30b2331-2097-4981-89c8-<wbr>acea4a81f7f2 - - - - -]
Unable to refresh my resource provider record</div>
<div class="gmail_extra">Dec 18 22:31:47 <a href="http://node-14.mydom.com" target="_blank">node-14.mydom.com</a>
nova-compute[32196]: 2017-12-18 22:31:47.634 32196 INFO
nova.compute.resource_tracker
[req-f30b2331-2097-4981-89c8-<wbr>acea4a81f7f2 - - - - -]
Compute_service record updated for node-14.mydom.com:n<a href="http://ode-14.mydom.com" target="_blank">ode-14.<wbr>mydom.com</a></div>
<div class="gmail_extra">Dec 18 22:31:52 <a href="http://node-14.mydom.com" target="_blank">node-14.mydom.com</a>
nova-compute[32196]: 2017-12-18 22:31:52.247 32196 DEBUG
oslo_messaging._drivers.<wbr>amqpdriver
[req-f30b2331-2097-4981-89c8-<wbr>acea4a81f7f2 - - - - -] CALL
msg_id: 4cbf019c36ce41cd89d34e59f6acc5<wbr>5f exchange 'nova'
topic 'conductor' _send
/usr/lib/python2.7/dist-<wbr>packages/oslo_messaging/_<wbr>drivers/amqpdriver.py:448</div>
<div class="gmail_extra">Dec 18 22:31:52 <a href="http://node-14.mydom.com" target="_blank">node-14.mydom.com</a>
nova-compute[32196]: 2017-12-18 22:31:52.265 32196 DEBUG
oslo_messaging._drivers.<wbr>amqpdriver [-] received reply
msg_id: 4cbf019c36ce41cd89d34e59f6acc5<wbr>5f __call__
/usr/lib/python2.7/dist-<wbr>packages/oslo_messaging/_<wbr>drivers/amqpdriver.py:296</div>
<div class="gmail_extra">root@node-14:~#</div>
<div class="gmail_extra"><br>
</div>
</div>
<div class="gmail_extra"><br>
</div>
<div class="gmail_extra"><br>
</div>
<div class="gmail_extra"><br>
</div>
<div class="gmail_extra"><br>
</div>
<div class="gmail_extra"><br>
</div>
<div class="gmail_extra"><br>
</div>
<div class="gmail_extra">I get this for the status of a node
that is showing "DOWN"</div>
</div>
<div class="gmail_extra"><br>
</div>
<div class="gmail_extra"><br>
</div>
<div class="gmail_extra">
<div class="gmail_extra">root@node-9:~# systemctl status
nova-compute.service</div>
<div class="gmail_extra">â nova-compute.service - OpenStack
Compute</div>
<div class="gmail_extra"> Loaded: loaded
(/lib/systemd/system/nova-<wbr>compute.service; enabled; vendor
preset: enabled)</div>
<div class="gmail_extra"> Active: active (running) since Mon
2017-12-18 21:20:30 UTC; 1h 11min ago</div>
<div class="gmail_extra"> Docs: man:nova-compute(1)</div>
<div class="gmail_extra"> Process: 9488
ExecStartPre=/bin/chown nova:adm /var/log/nova (code=exited,
status=0/SUCCESS)</div>
<div class="gmail_extra"> Process: 9485
ExecStartPre=/bin/chown nova:nova /var/lock/nova
/var/lib/nova (code=exited, status=0/SUCCESS)</div>
<div class="gmail_extra"> Process: 9482
ExecStartPre=/bin/mkdir -p /var/lock/nova /var/log/nova
/var/lib/nova (code=exited, status=0/SUCCESS)</div>
<div class="gmail_extra"> Main PID: 9491 (nova-compute)</div>
<div class="gmail_extra"> CGroup:
/system.slice/nova-compute.<wbr>service</div>
<div class="gmail_extra"> ââ 9491 /usr/bin/python
/usr/bin/nova-compute
--config-file=/etc/nova/nova-<wbr>compute.conf
--config-file=/etc/nova/nova.<wbr>conf
--log-file=/var/log/nova/nova-<wbr>compute.log</div>
<div class="gmail_extra"> ââ20428 /usr/bin/python
/usr/bin/nova-compute
--config-file=/etc/nova/nova-<wbr>compute.conf
--config-file=/etc/nova/nova.<wbr>conf
--log-file=/var/log/nova/nova-<wbr>compute.log</div>
<div class="gmail_extra"><br>
</div>
<div class="gmail_extra">Dec 18 22:00:32 <a href="http://node-9.mydom.com" target="_blank">node-9.mydom.com</a>
nova-compute[9491]: 2017-12-18 22:00:32.065 9491 INFO
nova.virt.libvirt.imagecache
[req-7623143a-2263-448e-8100-<wbr>b6248501ab42 - - - - -] image
2e42dd45-0b7d-468a-bc1e-<wbr>3ece5bdc1638 at
(/mnt/MSA_FC_Vol1/nodes/_base/<wbr>fe073169880f11099449fef24d86f3<wbr>c1f8bb9763):
checking</div>
<div class="gmail_extra">Dec 18 22:00:32 <a href="http://node-9.mydom.com" target="_blank">node-9.mydom.com</a>
nova-compute[9491]: 2017-12-18 22:00:32.066 9491 INFO
nova.virt.libvirt.imagecache
[req-7623143a-2263-448e-8100-<wbr>b6248501ab42 - - - - -] image
2e42dd45-0b7d-468a-bc1e-<wbr>3ece5bdc1638 at
(/mnt/MSA_FC_Vol1/nodes/_base/<wbr>fe073169880f11099449fef24d86f3<wbr>c1f8bb9763):
in use: on this node 0 local, 4 on other nodes sharing this</div>
<div class="gmail_extra">Dec 18 22:00:32 <a href="http://node-9.mydom.com" target="_blank">node-9.mydom.com</a>
sudo[40588]: nova : TTY=unknown ; PWD=/var/lib/nova ;
USER=root ; COMMAND=/usr/bin/nova-rootwrap
/etc/nova/rootwrap.conf touch -c
/mnt/MSA_FC_Vol1/nodes/_base/<wbr>fe073169880f11099449fef24d86f3<wbr>c1f8bb9763</div>
<div class="gmail_extra">Dec 18 22:00:32 <a href="http://node-9.mydom.com" target="_blank">node-9.mydom.com</a>
sudo[40588]: pam_unix(sudo:session): session opened for user
root by (uid=0)</div>
<div class="gmail_extra">Dec 18 22:00:32 <a href="http://node-9.mydom.com" target="_blank">node-9.mydom.com</a>
sudo[40588]: pam_unix(sudo:session): session closed for user
root</div>
<div class="gmail_extra">Dec 18 22:00:32 <a href="http://node-9.mydom.com" target="_blank">node-9.mydom.com</a>
nova-compute[9491]: 2017-12-18 22:00:32.148 9491 INFO
nova.virt.libvirt.imagecache
[req-7623143a-2263-448e-8100-<wbr>b6248501ab42 - - - - -] image
01db7edf-ee68-4fbc-bf3e-<wbr>1ac4bc990488 at
(/mnt/MSA_FC_Vol1/nodes/_base/<wbr>a49721a231fdd7b45293b29dd13c34<wbr>207c9c891b):
checking</div>
<div class="gmail_extra">Dec 18 22:00:32 <a href="http://node-9.mydom.com" target="_blank">node-9.mydom.com</a>
nova-compute[9491]: 2017-12-18 22:00:32.149 9491 INFO
nova.virt.libvirt.imagecache
[req-7623143a-2263-448e-8100-<wbr>b6248501ab42 - - - - -] image
01db7edf-ee68-4fbc-bf3e-<wbr>1ac4bc990488 at
(/mnt/MSA_FC_Vol1/nodes/_base/<wbr>a49721a231fdd7b45293b29dd13c34<wbr>207c9c891b):
in use: on this node 0 local, 2 on other nodes sharing this</div>
<div class="gmail_extra">Dec 18 22:00:32 <a href="http://node-9.mydom.com" target="_blank">node-9.mydom.com</a>
sudo[40591]: nova : TTY=unknown ; PWD=/var/lib/nova ;
USER=root ; COMMAND=/usr/bin/nova-rootwrap
/etc/nova/rootwrap.conf touch -c
/mnt/MSA_FC_Vol1/nodes/_base/<wbr>a49721a231fdd7b45293b29dd13c34<wbr>207c9c891b</div>
<div class="gmail_extra">Dec 18 22:00:32 <a href="http://node-9.mydom.com" target="_blank">node-9.mydom.com</a>
sudo[40591]: pam_unix(sudo:session): session opened for user
root by (uid=0)</div>
<div class="gmail_extra">Dec 18 22:00:32 <a href="http://node-9.mydom.com" target="_blank">node-9.mydom.com</a>
sudo[40591]: pam_unix(sudo:session): session closed for user
root</div>
<div class="gmail_extra">lines 1-22/22 (END)</div>
<div class="gmail_extra">root@node-9:~#</div>
<div class="gmail_extra"><br>
</div>
<div class="gmail_extra"><br>
</div>
<div class="gmail_extra"><br>
</div>
<div class="gmail_extra"><br>
</div>
<div class="gmail_extra"><br>
</div>
<div class="gmail_extra"><br>
</div>
<div class="gmail_extra">Does anything in the status messages
show what could be wrong? What do the "nova : TTY=unknown"
messages mean?</div>
<div class="gmail_extra"><br>
</div>
<div class="gmail_extra">thanks!!</div>
<div class="gmail_extra"><br>
</div>
<div class="gmail_extra">-- Jim</div>
</div>
<div class="gmail_extra"><br>
</div>
</div>
<br>
<fieldset class="m_7988713619055522055mimeAttachmentHeader"></fieldset>
<br>
<pre>______________________________<wbr>_________________
Mailing list: <a class="m_7988713619055522055moz-txt-link-freetext" href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack" target="_blank">http://lists.openstack.org/<wbr>cgi-bin/mailman/listinfo/<wbr>openstack</a>
Post to : <a class="m_7988713619055522055moz-txt-link-abbreviated" href="mailto:openstack@lists.openstack.org" target="_blank">openstack@lists.openstack.org</a>
Unsubscribe : <a class="m_7988713619055522055moz-txt-link-freetext" href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack" target="_blank">http://lists.openstack.org/<wbr>cgi-bin/mailman/listinfo/<wbr>openstack</a>
</pre>
</blockquote>
<br>
<pre class="m_7988713619055522055moz-signature" cols="72">--
Volodymyr Litovka
"Vision without Execution is Hallucination." -- Thomas Edison</pre>
</div>
<br>______________________________<wbr>_________________<br>
Mailing list: <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack" rel="noreferrer" target="_blank">http://lists.openstack.org/<wbr>cgi-bin/mailman/listinfo/<wbr>openstack</a><br>
Post to : <a href="mailto:openstack@lists.openstack.org">openstack@lists.openstack.org</a><br>
Unsubscribe : <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack" rel="noreferrer" target="_blank">http://lists.openstack.org/<wbr>cgi-bin/mailman/listinfo/<wbr>openstack</a><br>
<br></blockquote></div></div>