Open Stack

I see “no valid host” at 10:48:39 in the conductor log:

2021-06-17T10:48:39.65107847Z stdout F nova.exception.NoValidHost: No valid host was found.

In the scheduler log at 10:37:43 we see the scheduler starting and the RMQ error, followed by the scheduling failure at 10:48:39. It looks like the scheduler can’t connect to RMQ.

2021-06-17T10:37:43.038362995Z stdout F 2021-06-17 10:37:43.038 1 INFO nova.service [-] Starting scheduler node (version 21.2.1)
2021-06-17T10:37:43.08082925Z stdout F 2021-06-17 10:37:43.079 1 ERROR oslo.messaging._drivers.impl_rabbit [req-775b2b25-b9ef-4642-a6fd-7574eab7cc37 - - - - -] Connection failed: failed to resolve broker hostname (retrying in 0 seconds): OSError: failed to resolve broker hostname
2021-06-17T10:48:39.197318418Z stdout F 2021-06-17 10:48:39.196 1 INFO nova.scheduler.manager [req-b30bb3d6-0ab1-4dbb-aed9-1d9cece3eac7 d9f7048c1cd947cfa8ecef128a6cee89 e8813293073545f99658adbec2f80c1d - default default] Got no allocation candidates from the Placement API. This could be due to insufficient resources or a temporary occurrence as compute nodes start up.

From: open infra <openinfradn at gmail.com>
Sent: Thursday, June 17, 2021 12:38 PM
To: Braden, Albert <C-Albert.Braden at charter.com>
Cc: openstack-discuss <openstack-discuss at lists.openstack.org>
Subject: Re: [EXTERNAL] Error creating VMs

CAUTION: The e-mail below is from an external source. Please exercise caution before opening attachments, clicking links, or following guidance.
Another instance created at   2021-06-17T16:10:06Z, and it also failed.
I can't see rabbitmq errors around that timestamp.

scheduler log http://paste.openstack.org/show/806737/<https://www.google.com/url?q=http://paste.openstack.org/show/806737/&sa=D&source=hangouts&ust=1624034044021000&usg=AFQjCNHSJ6dWR_hMfE6e6YUw14demT_mZA>
conducter log http://paste.openstack.org/show/806738/<https://www.google.com/url?q=http://paste.openstack.org/show/806738/&sa=D&source=hangouts&ust=1624034079136000&usg=AFQjCNHuMa5Eerlpx0-wlOM-fBaDXpHptQ>

At  2021-06-17 15:52:38.810, system was able to connect to AMQP
http://paste.openstack.org/show/806739/<https://www.google.com/url?q=http://paste.openstack.org/show/806739/&sa=D&source=hangouts&ust=1624034275886000&usg=AFQjCNFree0y1N8q-PQnd6Unz5AQZG5q2g>



controller-0:/var/log/pods$ sudo rabbitmqctl list_queues
Password:
Listing queues ...
sysinv.ceph_manager.192.168.204.1 0
sysinv.ceph_manager 0
barbican.workers 0
sysinv.fpga_agent_manager.controller-0 0
barbican.workers.barbican.queue 0
sysinv.conductor_manager 0
sysinv.agent_manager_fanout_fed76e414eb04da084ab35a1c27e1bf1 0
sysinv.agent_manager 0
sysinv.ceph_manager_fanout_d585fb522f46431da60741573a7f8575 0
notifications.info<http://notifications.info> 0
sysinv.conductor_manager_fanout_cba8926fa47f4780a9e17f3d9b889500 0
sysinv.fpga_agent_manager 0
sysinv-keystone-listener-workers 0
sysinv.fpga_agent_manager_fanout_f00a01bfd3f54a64860f8d7454e9a78e 0
sysinv.agent_manager.controller-0 0
barbican.workers_fanout_2c97c319faa943e88eaed4c101e530c7 0
sysinv.conductor_manager.controller-0 0

On Thu, Jun 17, 2021 at 7:51 PM Braden, Albert <C-Albert.Braden at charter.com<mailto:C-Albert.Braden at charter.com>> wrote:
It looks like RMQ is up but services can’t connect to it. What do you see in the RMQ web interface?


This might be a clue:


ERROR oslo.messaging._drivers.impl_rabbit [req-b30bb3d6-0ab1-4dbb-aed9-1d9cece3eac7 d9f7048c1cd947cfa8ecef128a6cee89 e8813293073545f99658adbec2f80c1d - default default] Connection failed: failed to resolve broker hostname (retrying in 0 seconds): OSError: failed to resolve broker hostname

Check for a typo in your config that points services to an incorrect RMQ hostname, or a networking issue that prevents them from connecting.

From: open infra <openinfradn at gmail.com<mailto:openinfradn at gmail.com>>
Sent: Thursday, June 17, 2021 10:06 AM
To: Braden, Albert <C-Albert.Braden at charter.com<mailto:C-Albert.Braden at charter.com>>
Cc: openstack-discuss <openstack-discuss at lists.openstack.org<mailto:openstack-discuss at lists.openstack.org>>
Subject: Re: [EXTERNAL] Error creating VMs

CAUTION: The e-mail below is from an external source. Please exercise caution before opening attachments, clicking links, or following guidance.
controller-0:/var/log/pods$ sudo rabbitmqctl cluster_status
Password:
Cluster status of node rabbit at localhost ...
[{nodes,[{disc,[rabbit at localhost]}]},
 {running_nodes,[rabbit at localhost]},
 {cluster_name,<<"rabbit at controller-0">>},
 {partitions,[]},
 {alarms,[{rabbit at localhost,[]}]}]

On Thu, Jun 17, 2021 at 7:32 PM Braden, Albert <C-Albert.Braden at charter.com<mailto:C-Albert.Braden at charter.com>> wrote:
It looks like your RMQ is broken. What do you get from “rabbitmqctl cluster_status”?

From: open infra <openinfradn at gmail.com<mailto:openinfradn at gmail.com>>
Sent: Thursday, June 17, 2021 9:58 AM
To: Braden, Albert <C-Albert.Braden at charter.com<mailto:C-Albert.Braden at charter.com>>
Cc: openstack-discuss <openstack-discuss at lists.openstack.org<mailto:openstack-discuss at lists.openstack.org>>
Subject: Re: [EXTERNAL] Error creating VMs

CAUTION: The e-mail below is from an external source. Please exercise caution before opening attachments, clicking links, or following guidance.

Hi Albert,

Sorry for the inconvenience.

Please note that I have recreated both the data network at starlingx (physical network of openstack) and the network of openstack.

But I still have the same issue. Please find scheduler and conductor logs.

# Scheduler Logs

E-MAIL CONFIDENTIALITY NOTICE: 
The contents of this e-mail message and any attachments are intended solely for the addressee(s) and may contain confidential and/or legally privileged information. If you are not the intended recipient of this message or if this message has been addressed to you in error, please immediately alert the sender by reply e-mail and then delete this message and any attachments. If you are not the intended recipient, you are notified that any use, dissemination, distribution, copying, or storage of this message or any attachment is strictly prohibited.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-discuss/attachments/20210617/6c944339/attachment-0001.html>