VM provisioning is resolved.
But still having an issue with openstack networking.

"Unable to create the network. No tenant network is available for allocation."

http://paste.openstack.org/show/807014/

On Thu, Jun 17, 2021 at 11:14 PM Braden, Albert <C-Albert.Braden@charter.com> wrote:

I see “no valid host” at 10:48:39 in the conductor log:

 

2021-06-17T10:48:39.65107847Z stdout F nova.exception.NoValidHost: No valid host was found.

 

In the scheduler log at 10:37:43 we see the scheduler starting and the RMQ error, followed by the scheduling failure at 10:48:39. It looks like the scheduler can’t connect to RMQ.

 

2021-06-17T10:37:43.038362995Z stdout F 2021-06-17 10:37:43.038 1 INFO nova.service [-] Starting scheduler node (version 21.2.1)

2021-06-17T10:37:43.08082925Z stdout F 2021-06-17 10:37:43.079 1 ERROR oslo.messaging._drivers.impl_rabbit [req-775b2b25-b9ef-4642-a6fd-7574eab7cc37 - - - - -] Connection failed: failed to resolve broker hostname (retrying in 0 seconds): OSError: failed to resolve broker hostname

2021-06-17T10:48:39.197318418Z stdout F 2021-06-17 10:48:39.196 1 INFO nova.scheduler.manager [req-b30bb3d6-0ab1-4dbb-aed9-1d9cece3eac7 d9f7048c1cd947cfa8ecef128a6cee89 e8813293073545f99658adbec2f80c1d - default default] Got no allocation candidates from the Placement API. This could be due to insufficient resources or a temporary occurrence as compute nodes start up.

 

From: open infra <openinfradn@gmail.com>
Sent: Thursday, June 17, 2021 12:38 PM
To: Braden, Albert <C-Albert.Braden@charter.com>
Cc: openstack-discuss <openstack-discuss@lists.openstack.org>
Subject: Re: [EXTERNAL] Error creating VMs

 

CAUTION: The e-mail below is from an external source. Please exercise caution before opening attachments, clicking links, or following guidance.

Another instance created at   2021-06-17T16:10:06Z, and it also failed.

I can't see rabbitmq errors around that timestamp.

 

 

At  2021-06-17 15:52:38.810, system was able to connect to AMQP 

 

 

 

controller-0:/var/log/pods$ sudo rabbitmqctl list_queues
Password: 
Listing queues ...
sysinv.ceph_manager.192.168.204.1 0
sysinv.ceph_manager 0
barbican.workers 0
sysinv.fpga_agent_manager.controller-0 0
barbican.workers.barbican.queue 0
sysinv.conductor_manager 0
sysinv.agent_manager_fanout_fed76e414eb04da084ab35a1c27e1bf1 0
sysinv.agent_manager 0
sysinv.ceph_manager_fanout_d585fb522f46431da60741573a7f8575 0
notifications.info 0
sysinv.conductor_manager_fanout_cba8926fa47f4780a9e17f3d9b889500 0
sysinv.fpga_agent_manager 0
sysinv-keystone-listener-workers 0
sysinv.fpga_agent_manager_fanout_f00a01bfd3f54a64860f8d7454e9a78e 0
sysinv.agent_manager.controller-0 0
barbican.workers_fanout_2c97c319faa943e88eaed4c101e530c7 0
sysinv.conductor_manager.controller-0 0

 

On Thu, Jun 17, 2021 at 7:51 PM Braden, Albert <C-Albert.Braden@charter.com> wrote:

It looks like RMQ is up but services can’t connect to it. What do you see in the RMQ web interface?

 

This might be a clue: 
 

ERROR oslo.messaging._drivers.impl_rabbit [req-b30bb3d6-0ab1-4dbb-aed9-1d9cece3eac7 d9f7048c1cd947cfa8ecef128a6cee89 e8813293073545f99658adbec2f80c1d - default default] Connection failed: failed to resolve broker hostname (retrying in 0 seconds): OSError: failed to resolve broker hostname

 

Check for a typo in your config that points services to an incorrect RMQ hostname, or a networking issue that prevents them from connecting.

 

From: open infra <openinfradn@gmail.com>
Sent: Thursday, June 17, 2021 10:06 AM
To: Braden, Albert <C-Albert.Braden@charter.com>
Cc: openstack-discuss <openstack-discuss@lists.openstack.org>
Subject: Re: [EXTERNAL] Error creating VMs

 

CAUTION: The e-mail below is from an external source. Please exercise caution before opening attachments, clicking links, or following guidance.

controller-0:/var/log/pods$ sudo rabbitmqctl cluster_status
Password: 
Cluster status of node rabbit@localhost ...
[{nodes,[{disc,[rabbit@localhost]}]},
 {running_nodes,[rabbit@localhost]},
 {cluster_name,<<"rabbit@controller-0">>},
 {partitions,[]},
 {alarms,[{rabbit@localhost,[]}]}]

 

On Thu, Jun 17, 2021 at 7:32 PM Braden, Albert <C-Albert.Braden@charter.com> wrote:

It looks like your RMQ is broken. What do you get from “rabbitmqctl cluster_status”?

 

From: open infra <openinfradn@gmail.com>
Sent: Thursday, June 17, 2021 9:58 AM
To: Braden, Albert <C-Albert.Braden@charter.com>
Cc: openstack-discuss <openstack-discuss@lists.openstack.org>
Subject: Re: [EXTERNAL] Error creating VMs

 

CAUTION: The e-mail below is from an external source. Please exercise caution before opening attachments, clicking links, or following guidance. 

Hi Albert,
Sorry for the inconvenience.
Please note that I have recreated both the data network at starlingx (physical network of openstack) and the network of openstack.
But I still have the same issue. Please find scheduler and conductor logs.
# Scheduler Logs

 

The contents of this e-mail message and
any attachments are intended solely for the
addressee(s) and may contain confidential
and/or legally privileged information. If you
are not the intended recipient of this message
or if this message has been addressed to you
in error, please immediately alert the sender
by reply e-mail and then delete this message
and any attachments. If you are not the
intended recipient, you are notified that
any use, dissemination, distribution, copying,
or storage of this message or any attachment
is strictly prohibited.