Open Stack

Another instance created at   2021-06-17T16:10:06Z, and it also failed.
I can't see rabbitmq errors around that timestamp.

scheduler log http://paste.openstack.org/show/806737/
<https://www.google.com/url?q=http://paste.openstack.org/show/806737/&sa=D&source=hangouts&ust=1624034044021000&usg=AFQjCNHSJ6dWR_hMfE6e6YUw14demT_mZA>
conducter log http://paste.openstack.org/show/806738/
<https://www.google.com/url?q=http://paste.openstack.org/show/806738/&sa=D&source=hangouts&ust=1624034079136000&usg=AFQjCNHuMa5Eerlpx0-wlOM-fBaDXpHptQ>

At  2021-06-17 15:52:38.810, system was able to connect to AMQP
http://paste.openstack.org/show/806739/
<https://www.google.com/url?q=http://paste.openstack.org/show/806739/&sa=D&source=hangouts&ust=1624034275886000&usg=AFQjCNFree0y1N8q-PQnd6Unz5AQZG5q2g>



controller-0:/var/log/pods$ sudo rabbitmqctl list_queues
Password:
Listing queues ...
sysinv.ceph_manager.192.168.204.1 0
sysinv.ceph_manager 0
barbican.workers 0
sysinv.fpga_agent_manager.controller-0 0
barbican.workers.barbican.queue 0
sysinv.conductor_manager 0
sysinv.agent_manager_fanout_fed76e414eb04da084ab35a1c27e1bf1 0
sysinv.agent_manager 0
sysinv.ceph_manager_fanout_d585fb522f46431da60741573a7f8575 0
notifications.info 0
sysinv.conductor_manager_fanout_cba8926fa47f4780a9e17f3d9b889500 0
sysinv.fpga_agent_manager 0
sysinv-keystone-listener-workers 0
sysinv.fpga_agent_manager_fanout_f00a01bfd3f54a64860f8d7454e9a78e 0
sysinv.agent_manager.controller-0 0
barbican.workers_fanout_2c97c319faa943e88eaed4c101e530c7 0
sysinv.conductor_manager.controller-0 0

On Thu, Jun 17, 2021 at 7:51 PM Braden, Albert <C-Albert.Braden at charter.com>
wrote:

> It looks like RMQ is up but services can’t connect to it. What do you see
> in the RMQ web interface?
>
>
>
> This might be a clue:
>
>
>
> ERROR oslo.messaging._drivers.impl_rabbit
> [req-b30bb3d6-0ab1-4dbb-aed9-1d9cece3eac7 d9f7048c1cd947cfa8ecef128a6cee89
> e8813293073545f99658adbec2f80c1d - default default] Connection failed:
> failed to resolve broker hostname (retrying in 0 seconds): OSError: failed
> to resolve broker hostname
>
>
>
> Check for a typo in your config that points services to an incorrect RMQ
> hostname, or a networking issue that prevents them from connecting.
>
>
>
> *From:* open infra <openinfradn at gmail.com>
> *Sent:* Thursday, June 17, 2021 10:06 AM
> *To:* Braden, Albert <C-Albert.Braden at charter.com>
> *Cc:* openstack-discuss <openstack-discuss at lists.openstack.org>
> *Subject:* Re: [EXTERNAL] Error creating VMs
>
>
>
> *CAUTION:* The e-mail below is from an external source. Please exercise
> caution before opening attachments, clicking links, or following guidance.
>
> controller-0:/var/log/pods$ sudo rabbitmqctl cluster_status
> Password:
> Cluster status of node rabbit at localhost ...
> [{nodes,[{disc,[rabbit at localhost]}]},
>  {running_nodes,[rabbit at localhost]},
>  {cluster_name,<<"rabbit at controller-0">>},
>  {partitions,[]},
>  {alarms,[{rabbit at localhost,[]}]}]
>
>
>
> On Thu, Jun 17, 2021 at 7:32 PM Braden, Albert <
> C-Albert.Braden at charter.com> wrote:
>
> It looks like your RMQ is broken. What do you get from “rabbitmqctl
> cluster_status”?
>
>
>
> *From:* open infra <openinfradn at gmail.com>
> *Sent:* Thursday, June 17, 2021 9:58 AM
> *To:* Braden, Albert <C-Albert.Braden at charter.com>
> *Cc:* openstack-discuss <openstack-discuss at lists.openstack.org>
> *Subject:* Re: [EXTERNAL] Error creating VMs
>
>
>
> *CAUTION:* The e-mail below is from an external source. Please exercise
> caution before opening attachments, clicking links, or following guidance.
>
> Hi Albert,
>
> Sorry for the inconvenience.
>
> Please note that I have recreated both the data network at starlingx (physical network of openstack) and the network of openstack.
>
> But I still have the same issue. Please find scheduler and conductor logs.
>
> # Scheduler Logs
>
> http://paste.openstack.org/show/806729/
>
>
>
> ## Conductor logs
>
> http://paste.openstack.org/show/806727/
>
>
>
> On Thu, Jun 17, 2021 at 6:55 PM Braden, Albert <
> C-Albert.Braden at charter.com> wrote:
>
> Both links show the conductor log. It tells us that no valid host was
> found. The scheduler log should show why no valid host was found.
>
>
>
> *From:* open infra <openinfradn at gmail.com>
> *Sent:* Thursday, June 17, 2021 2:02 AM
> *To:* Braden, Albert <C-Albert.Braden at charter.com>
> *Cc:* openstack-discuss <openstack-discuss at lists.openstack.org>
> *Subject:* Re: [EXTERNAL] Error creating VMs
>
>
>
> *CAUTION:* The e-mail below is from an external source. Please exercise
> caution before opening attachments, clicking links, or following guidance.
>
> Found nova-conductor and nova-scheduler  logs [1],[2].
>
> But I have no idea about what caused this issue.
>
>
>
> [1]  http://paste.openstack.org/show/806720/
>
> [2] http://paste.openstack.org/show/806721/
>
>
>
>
>
> On Wed, Jun 16, 2021 at 8:08 PM Braden, Albert <
> C-Albert.Braden at charter.com> wrote:
>
> Someone will need to dig through the starlingx documentation and figure
> out where the log files are located. Do you want to do that?
>
>
>
> https://docs.starlingx.io/
>
>
>
> *From:* open infra <openinfradn at gmail.com>
> *Sent:* Wednesday, June 16, 2021 9:12 AM
> *To:* Braden, Albert <C-Albert.Braden at charter.com>
> *Cc:* openstack-discuss <openstack-discuss at lists.openstack.org>
> *Subject:* Re: [EXTERNAL] Error creating VMs
>
>
>
> *CAUTION:* The e-mail below is from an external source. Please exercise
> caution before opening attachments, clicking links, or following guidance.
>
> I can see sm-scheduler.log not sure if its the correct log file you
> mentioned, but I can't see conductor log.
>
> This environment have been deployed using starlingx.
>
>
>
> On Wed, Jun 16, 2021 at 6:16 PM Braden, Albert <
> C-Albert.Braden at charter.com> wrote:
>
> Can you check the scheduler and conductor logs on the controllers? There
> should be entries describing why the instance failed to schedule. You may
> need to set “debug=true” in nova.conf to get more details.
>
>
>
> *From:* open infra <openinfradn at gmail.com>
> *Sent:* Wednesday, June 16, 2021 1:36 AM
> *To:* openstack-discuss <openstack-discuss at lists.openstack.org>
> *Subject:* [EXTERNAL] Error creating VMs
>
>
>
> *CAUTION:* The e-mail below is from an external source. Please exercise
> caution before opening attachments, clicking links, or following guidance.
>
> Hi
>
>
>
> After setting up OpenStack environment using STX R5 (two controllers, two
> storage nodes and one worker), I have deployed a VM. VM ended up with ERROR
> status.
>
>
>
> I highly appreciate if someone can guide to dig further (what logs to
> check ) or to fix this issue.
>
>
>
> http://paste.openstack.org/show/806626/
>
>
>
> Regards
>
> Danishka
>
>
>
> The contents of this e-mail message and
> any attachments are intended solely for the
> addressee(s) and may contain confidential
> and/or legally privileged information. If you
> are not the intended recipient of this message
> or if this message has been addressed to you
> in error, please immediately alert the sender
> by reply e-mail and then delete this message
> and any attachments. If you are not the
> intended recipient, you are notified that
> any use, dissemination, distribution, copying,
> or storage of this message or any attachment
> is strictly prohibited.
>
> The contents of this e-mail message and
> any attachments are intended solely for the
> addressee(s) and may contain confidential
> and/or legally privileged information. If you
> are not the intended recipient of this message
> or if this message has been addressed to you
> in error, please immediately alert the sender
> by reply e-mail and then delete this message
> and any attachments. If you are not the
> intended recipient, you are notified that
> any use, dissemination, distribution, copying,
> or storage of this message or any attachment
> is strictly prohibited.
>
> The contents of this e-mail message and
> any attachments are intended solely for the
> addressee(s) and may contain confidential
> and/or legally privileged information. If you
> are not the intended recipient of this message
> or if this message has been addressed to you
> in error, please immediately alert the sender
> by reply e-mail and then delete this message
> and any attachments. If you are not the
> intended recipient, you are notified that
> any use, dissemination, distribution, copying,
> or storage of this message or any attachment
> is strictly prohibited.
>
> The contents of this e-mail message and
> any attachments are intended solely for the
> addressee(s) and may contain confidential
> and/or legally privileged information. If you
> are not the intended recipient of this message
> or if this message has been addressed to you
> in error, please immediately alert the sender
> by reply e-mail and then delete this message
> and any attachments. If you are not the
> intended recipient, you are notified that
> any use, dissemination, distribution, copying,
> or storage of this message or any attachment
> is strictly prohibited.
>
> The contents of this e-mail message and
> any attachments are intended solely for the
> addressee(s) and may contain confidential
> and/or legally privileged information. If you
> are not the intended recipient of this message
> or if this message has been addressed to you
> in error, please immediately alert the sender
> by reply e-mail and then delete this message
> and any attachments. If you are not the
> intended recipient, you are notified that
> any use, dissemination, distribution, copying,
> or storage of this message or any attachment
> is strictly prohibited.
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-discuss/attachments/20210617/13e3a9f3/attachment-0001.html>