Hi After setting up OpenStack environment using STX R5 (two controllers, two storage nodes and one worker), I have deployed a VM. VM ended up with ERROR status. I highly appreciate if someone can guide to dig further (what logs to check ) or to fix this issue. http://paste.openstack.org/show/806626/ Regards Danishka
Can you check the scheduler and conductor logs on the controllers? There should be entries describing why the instance failed to schedule. You may need to set “debug=true” in nova.conf to get more details. From: open infra <openinfradn@gmail.com> Sent: Wednesday, June 16, 2021 1:36 AM To: openstack-discuss <openstack-discuss@lists.openstack.org> Subject: [EXTERNAL] Error creating VMs CAUTION: The e-mail below is from an external source. Please exercise caution before opening attachments, clicking links, or following guidance. Hi After setting up OpenStack environment using STX R5 (two controllers, two storage nodes and one worker), I have deployed a VM. VM ended up with ERROR status. I highly appreciate if someone can guide to dig further (what logs to check ) or to fix this issue. http://paste.openstack.org/show/806626/ Regards Danishka E-MAIL CONFIDENTIALITY NOTICE: The contents of this e-mail message and any attachments are intended solely for the addressee(s) and may contain confidential and/or legally privileged information. If you are not the intended recipient of this message or if this message has been addressed to you in error, please immediately alert the sender by reply e-mail and then delete this message and any attachments. If you are not the intended recipient, you are notified that any use, dissemination, distribution, copying, or storage of this message or any attachment is strictly prohibited.
I can see sm-scheduler.log not sure if its the correct log file you mentioned, but I can't see conductor log. This environment have been deployed using starlingx. On Wed, Jun 16, 2021 at 6:16 PM Braden, Albert <C-Albert.Braden@charter.com> wrote:
Can you check the scheduler and conductor logs on the controllers? There should be entries describing why the instance failed to schedule. You may need to set “debug=true” in nova.conf to get more details.
*From:* open infra <openinfradn@gmail.com> *Sent:* Wednesday, June 16, 2021 1:36 AM *To:* openstack-discuss <openstack-discuss@lists.openstack.org> *Subject:* [EXTERNAL] Error creating VMs
*CAUTION:* The e-mail below is from an external source. Please exercise caution before opening attachments, clicking links, or following guidance.
Hi
After setting up OpenStack environment using STX R5 (two controllers, two storage nodes and one worker), I have deployed a VM. VM ended up with ERROR status.
I highly appreciate if someone can guide to dig further (what logs to check ) or to fix this issue.
http://paste.openstack.org/show/806626/
Regards
Danishka
The contents of this e-mail message and any attachments are intended solely for the addressee(s) and may contain confidential and/or legally privileged information. If you are not the intended recipient of this message or if this message has been addressed to you in error, please immediately alert the sender by reply e-mail and then delete this message and any attachments. If you are not the intended recipient, you are notified that any use, dissemination, distribution, copying, or storage of this message or any attachment is strictly prohibited.
Someone will need to dig through the starlingx documentation and figure out where the log files are located. Do you want to do that? https://docs.starlingx.io/ From: open infra <openinfradn@gmail.com> Sent: Wednesday, June 16, 2021 9:12 AM To: Braden, Albert <C-Albert.Braden@charter.com> Cc: openstack-discuss <openstack-discuss@lists.openstack.org> Subject: Re: [EXTERNAL] Error creating VMs CAUTION: The e-mail below is from an external source. Please exercise caution before opening attachments, clicking links, or following guidance. I can see sm-scheduler.log not sure if its the correct log file you mentioned, but I can't see conductor log. This environment have been deployed using starlingx. On Wed, Jun 16, 2021 at 6:16 PM Braden, Albert <C-Albert.Braden@charter.com<mailto:C-Albert.Braden@charter.com>> wrote: Can you check the scheduler and conductor logs on the controllers? There should be entries describing why the instance failed to schedule. You may need to set “debug=true” in nova.conf to get more details. From: open infra <openinfradn@gmail.com<mailto:openinfradn@gmail.com>> Sent: Wednesday, June 16, 2021 1:36 AM To: openstack-discuss <openstack-discuss@lists.openstack.org<mailto:openstack-discuss@lists.openstack.org>> Subject: [EXTERNAL] Error creating VMs CAUTION: The e-mail below is from an external source. Please exercise caution before opening attachments, clicking links, or following guidance. Hi After setting up OpenStack environment using STX R5 (two controllers, two storage nodes and one worker), I have deployed a VM. VM ended up with ERROR status. I highly appreciate if someone can guide to dig further (what logs to check ) or to fix this issue. http://paste.openstack.org/show/806626/ Regards Danishka The contents of this e-mail message and any attachments are intended solely for the addressee(s) and may contain confidential and/or legally privileged information. If you are not the intended recipient of this message or if this message has been addressed to you in error, please immediately alert the sender by reply e-mail and then delete this message and any attachments. If you are not the intended recipient, you are notified that any use, dissemination, distribution, copying, or storage of this message or any attachment is strictly prohibited. E-MAIL CONFIDENTIALITY NOTICE: The contents of this e-mail message and any attachments are intended solely for the addressee(s) and may contain confidential and/or legally privileged information. If you are not the intended recipient of this message or if this message has been addressed to you in error, please immediately alert the sender by reply e-mail and then delete this message and any attachments. If you are not the intended recipient, you are notified that any use, dissemination, distribution, copying, or storage of this message or any attachment is strictly prohibited.
worker-0 contin following under /var/log/pods kube-system_calico-node-brhq2_b3e3c8d7-ebeb-405a-af83-a19240d82997 openstack_neutron-metadata-agent-worker-0-13cc482d-6tgrf_88f4f98d-4526-4148-b99c-4e2ee9072995 kube-system_kube-multus-ds-amd64-k4pr4_1b0342da-641c-4248-a3ad-c67a9f8bcce5 openstack_neutron-ovs-agent-worker-0-13cc482d-6m9sw_0bc04a1a-8d36-4a37-99b3-19f6ebba9856 kube-system_kube-proxy-8hv88_dc495497-b346-4b28-ac5d-5aec9848f886 openstack_neutron-sriov-agent-worker-0-13cc482d-22rsz_a09c060e-3c4b-4aae-ad7f-ec2f96c0fcc3 kube-system_kube-sriov-cni-ds-amd64-5rpfv_02eab05d-eb1b-4a17-8559-8e6809d830d7 openstack_nova-compute-worker-0-13cc482d-27k2r_dcdd6855-b3e3-4137-b582-2fb665569490 openstack_libvirt-libvirt-default-4dcqc_c29b390b-7b69-4661-b0ac-cfc7b8c2d175 openstack_openvswitch-db-pnvcp_2855de13-37a7-4045-8444-0bbb54a6d8f3 openstack_neutron-dhcp-agent-worker-0-13cc482d-7pt4l_219e142a-1277-4b29-b4b5-dd365db4efb8 openstack_openvswitch-vswitchd-scrmc_c9eb3f17-9383-4f04-80bf-1cb63550d092 openstack_neutron-l3-agent-worker-0-13cc482d-7sfnp_ee272908-5710-4d2d-9206-8ec20a91ff10 openstack_osh-openstack-garbd-garbd-58dc7995cf-7hrd9_0366de80-b0d7-428e-8e6f-a9206231fd7a On Wed, Jun 16, 2021 at 8:08 PM Braden, Albert <C-Albert.Braden@charter.com> wrote:
Someone will need to dig through the starlingx documentation and figure out where the log files are located. Do you want to do that?
*From:* open infra <openinfradn@gmail.com> *Sent:* Wednesday, June 16, 2021 9:12 AM *To:* Braden, Albert <C-Albert.Braden@charter.com> *Cc:* openstack-discuss <openstack-discuss@lists.openstack.org> *Subject:* Re: [EXTERNAL] Error creating VMs
*CAUTION:* The e-mail below is from an external source. Please exercise caution before opening attachments, clicking links, or following guidance.
I can see sm-scheduler.log not sure if its the correct log file you mentioned, but I can't see conductor log.
This environment have been deployed using starlingx.
On Wed, Jun 16, 2021 at 6:16 PM Braden, Albert < C-Albert.Braden@charter.com> wrote:
Can you check the scheduler and conductor logs on the controllers? There should be entries describing why the instance failed to schedule. You may need to set “debug=true” in nova.conf to get more details.
*From:* open infra <openinfradn@gmail.com> *Sent:* Wednesday, June 16, 2021 1:36 AM *To:* openstack-discuss <openstack-discuss@lists.openstack.org> *Subject:* [EXTERNAL] Error creating VMs
*CAUTION:* The e-mail below is from an external source. Please exercise caution before opening attachments, clicking links, or following guidance.
Hi
After setting up OpenStack environment using STX R5 (two controllers, two storage nodes and one worker), I have deployed a VM. VM ended up with ERROR status.
I highly appreciate if someone can guide to dig further (what logs to check ) or to fix this issue.
http://paste.openstack.org/show/806626/
Regards
Danishka
The contents of this e-mail message and any attachments are intended solely for the addressee(s) and may contain confidential and/or legally privileged information. If you are not the intended recipient of this message or if this message has been addressed to you in error, please immediately alert the sender by reply e-mail and then delete this message and any attachments. If you are not the intended recipient, you are notified that any use, dissemination, distribution, copying, or storage of this message or any attachment is strictly prohibited.
The contents of this e-mail message and any attachments are intended solely for the addressee(s) and may contain confidential and/or legally privileged information. If you are not the intended recipient of this message or if this message has been addressed to you in error, please immediately alert the sender by reply e-mail and then delete this message and any attachments. If you are not the intended recipient, you are notified that any use, dissemination, distribution, copying, or storage of this message or any attachment is strictly prohibited.
Just clarifying: I have created an external network (including a subnet). openstack network create --share --external --provider-network-type vlan provider-external --provider-physical-network physnet0 Then I have created internal network using GUI. Used 'Networks' tab under Projects > Network. But it failed with the following error. "Failed to create network "internal-net": Unable to create the network. No tenant network is available for allocation." But when I try to create internal network under "Admin > Network > Networks", still I need to provide physical network like in external network creation. So, do I need to create both external and internal network using the same physical network? On Wed, Jun 16, 2021 at 8:57 PM open infra <openinfradn@gmail.com> wrote:
worker-0 contin following under /var/log/pods
kube-system_calico-node-brhq2_b3e3c8d7-ebeb-405a-af83-a19240d82997
openstack_neutron-metadata-agent-worker-0-13cc482d-6tgrf_88f4f98d-4526-4148-b99c-4e2ee9072995
kube-system_kube-multus-ds-amd64-k4pr4_1b0342da-641c-4248-a3ad-c67a9f8bcce5
openstack_neutron-ovs-agent-worker-0-13cc482d-6m9sw_0bc04a1a-8d36-4a37-99b3-19f6ebba9856
kube-system_kube-proxy-8hv88_dc495497-b346-4b28-ac5d-5aec9848f886
openstack_neutron-sriov-agent-worker-0-13cc482d-22rsz_a09c060e-3c4b-4aae-ad7f-ec2f96c0fcc3
kube-system_kube-sriov-cni-ds-amd64-5rpfv_02eab05d-eb1b-4a17-8559-8e6809d830d7
openstack_nova-compute-worker-0-13cc482d-27k2r_dcdd6855-b3e3-4137-b582-2fb665569490
openstack_libvirt-libvirt-default-4dcqc_c29b390b-7b69-4661-b0ac-cfc7b8c2d175
openstack_openvswitch-db-pnvcp_2855de13-37a7-4045-8444-0bbb54a6d8f3
openstack_neutron-dhcp-agent-worker-0-13cc482d-7pt4l_219e142a-1277-4b29-b4b5-dd365db4efb8 openstack_openvswitch-vswitchd-scrmc_c9eb3f17-9383-4f04-80bf-1cb63550d092
openstack_neutron-l3-agent-worker-0-13cc482d-7sfnp_ee272908-5710-4d2d-9206-8ec20a91ff10
openstack_osh-openstack-garbd-garbd-58dc7995cf-7hrd9_0366de80-b0d7-428e-8e6f-a9206231fd7a
On Wed, Jun 16, 2021 at 8:08 PM Braden, Albert < C-Albert.Braden@charter.com> wrote:
Someone will need to dig through the starlingx documentation and figure out where the log files are located. Do you want to do that?
*From:* open infra <openinfradn@gmail.com> *Sent:* Wednesday, June 16, 2021 9:12 AM *To:* Braden, Albert <C-Albert.Braden@charter.com> *Cc:* openstack-discuss <openstack-discuss@lists.openstack.org> *Subject:* Re: [EXTERNAL] Error creating VMs
*CAUTION:* The e-mail below is from an external source. Please exercise caution before opening attachments, clicking links, or following guidance.
I can see sm-scheduler.log not sure if its the correct log file you mentioned, but I can't see conductor log.
This environment have been deployed using starlingx.
On Wed, Jun 16, 2021 at 6:16 PM Braden, Albert < C-Albert.Braden@charter.com> wrote:
Can you check the scheduler and conductor logs on the controllers? There should be entries describing why the instance failed to schedule. You may need to set “debug=true” in nova.conf to get more details.
*From:* open infra <openinfradn@gmail.com> *Sent:* Wednesday, June 16, 2021 1:36 AM *To:* openstack-discuss <openstack-discuss@lists.openstack.org> *Subject:* [EXTERNAL] Error creating VMs
*CAUTION:* The e-mail below is from an external source. Please exercise caution before opening attachments, clicking links, or following guidance.
Hi
After setting up OpenStack environment using STX R5 (two controllers, two storage nodes and one worker), I have deployed a VM. VM ended up with ERROR status.
I highly appreciate if someone can guide to dig further (what logs to check ) or to fix this issue.
http://paste.openstack.org/show/806626/
Regards
Danishka
The contents of this e-mail message and any attachments are intended solely for the addressee(s) and may contain confidential and/or legally privileged information. If you are not the intended recipient of this message or if this message has been addressed to you in error, please immediately alert the sender by reply e-mail and then delete this message and any attachments. If you are not the intended recipient, you are notified that any use, dissemination, distribution, copying, or storage of this message or any attachment is strictly prohibited.
The contents of this e-mail message and any attachments are intended solely for the addressee(s) and may contain confidential and/or legally privileged information. If you are not the intended recipient of this message or if this message has been addressed to you in error, please immediately alert the sender by reply e-mail and then delete this message and any attachments. If you are not the intended recipient, you are notified that any use, dissemination, distribution, copying, or storage of this message or any attachment is strictly prohibited.
Found nova-conductor and nova-scheduler logs [1],[2]. But I have no idea about what caused this issue. [1] http://paste.openstack.org/show/806720/ [2] http://paste.openstack.org/show/806721/ On Wed, Jun 16, 2021 at 8:08 PM Braden, Albert <C-Albert.Braden@charter.com> wrote:
Someone will need to dig through the starlingx documentation and figure out where the log files are located. Do you want to do that?
*From:* open infra <openinfradn@gmail.com> *Sent:* Wednesday, June 16, 2021 9:12 AM *To:* Braden, Albert <C-Albert.Braden@charter.com> *Cc:* openstack-discuss <openstack-discuss@lists.openstack.org> *Subject:* Re: [EXTERNAL] Error creating VMs
*CAUTION:* The e-mail below is from an external source. Please exercise caution before opening attachments, clicking links, or following guidance.
I can see sm-scheduler.log not sure if its the correct log file you mentioned, but I can't see conductor log.
This environment have been deployed using starlingx.
On Wed, Jun 16, 2021 at 6:16 PM Braden, Albert < C-Albert.Braden@charter.com> wrote:
Can you check the scheduler and conductor logs on the controllers? There should be entries describing why the instance failed to schedule. You may need to set “debug=true” in nova.conf to get more details.
*From:* open infra <openinfradn@gmail.com> *Sent:* Wednesday, June 16, 2021 1:36 AM *To:* openstack-discuss <openstack-discuss@lists.openstack.org> *Subject:* [EXTERNAL] Error creating VMs
*CAUTION:* The e-mail below is from an external source. Please exercise caution before opening attachments, clicking links, or following guidance.
Hi
After setting up OpenStack environment using STX R5 (two controllers, two storage nodes and one worker), I have deployed a VM. VM ended up with ERROR status.
I highly appreciate if someone can guide to dig further (what logs to check ) or to fix this issue.
http://paste.openstack.org/show/806626/
Regards
Danishka
The contents of this e-mail message and any attachments are intended solely for the addressee(s) and may contain confidential and/or legally privileged information. If you are not the intended recipient of this message or if this message has been addressed to you in error, please immediately alert the sender by reply e-mail and then delete this message and any attachments. If you are not the intended recipient, you are notified that any use, dissemination, distribution, copying, or storage of this message or any attachment is strictly prohibited.
The contents of this e-mail message and any attachments are intended solely for the addressee(s) and may contain confidential and/or legally privileged information. If you are not the intended recipient of this message or if this message has been addressed to you in error, please immediately alert the sender by reply e-mail and then delete this message and any attachments. If you are not the intended recipient, you are notified that any use, dissemination, distribution, copying, or storage of this message or any attachment is strictly prohibited.
Both links show the conductor log. It tells us that no valid host was found. The scheduler log should show why no valid host was found. From: open infra <openinfradn@gmail.com> Sent: Thursday, June 17, 2021 2:02 AM To: Braden, Albert <C-Albert.Braden@charter.com> Cc: openstack-discuss <openstack-discuss@lists.openstack.org> Subject: Re: [EXTERNAL] Error creating VMs CAUTION: The e-mail below is from an external source. Please exercise caution before opening attachments, clicking links, or following guidance. Found nova-conductor and nova-scheduler logs [1],[2]. But I have no idea about what caused this issue. [1] http://paste.openstack.org/show/806720/ [2] http://paste.openstack.org/show/806721/ On Wed, Jun 16, 2021 at 8:08 PM Braden, Albert <C-Albert.Braden@charter.com<mailto:C-Albert.Braden@charter.com>> wrote: Someone will need to dig through the starlingx documentation and figure out where the log files are located. Do you want to do that? https://docs.starlingx.io/ From: open infra <openinfradn@gmail.com<mailto:openinfradn@gmail.com>> Sent: Wednesday, June 16, 2021 9:12 AM To: Braden, Albert <C-Albert.Braden@charter.com<mailto:C-Albert.Braden@charter.com>> Cc: openstack-discuss <openstack-discuss@lists.openstack.org<mailto:openstack-discuss@lists.openstack.org>> Subject: Re: [EXTERNAL] Error creating VMs CAUTION: The e-mail below is from an external source. Please exercise caution before opening attachments, clicking links, or following guidance. I can see sm-scheduler.log not sure if its the correct log file you mentioned, but I can't see conductor log. This environment have been deployed using starlingx. On Wed, Jun 16, 2021 at 6:16 PM Braden, Albert <C-Albert.Braden@charter.com<mailto:C-Albert.Braden@charter.com>> wrote: Can you check the scheduler and conductor logs on the controllers? There should be entries describing why the instance failed to schedule. You may need to set “debug=true” in nova.conf to get more details. From: open infra <openinfradn@gmail.com<mailto:openinfradn@gmail.com>> Sent: Wednesday, June 16, 2021 1:36 AM To: openstack-discuss <openstack-discuss@lists.openstack.org<mailto:openstack-discuss@lists.openstack.org>> Subject: [EXTERNAL] Error creating VMs CAUTION: The e-mail below is from an external source. Please exercise caution before opening attachments, clicking links, or following guidance. Hi After setting up OpenStack environment using STX R5 (two controllers, two storage nodes and one worker), I have deployed a VM. VM ended up with ERROR status. I highly appreciate if someone can guide to dig further (what logs to check ) or to fix this issue. http://paste.openstack.org/show/806626/ Regards Danishka The contents of this e-mail message and any attachments are intended solely for the addressee(s) and may contain confidential and/or legally privileged information. If you are not the intended recipient of this message or if this message has been addressed to you in error, please immediately alert the sender by reply e-mail and then delete this message and any attachments. If you are not the intended recipient, you are notified that any use, dissemination, distribution, copying, or storage of this message or any attachment is strictly prohibited. The contents of this e-mail message and any attachments are intended solely for the addressee(s) and may contain confidential and/or legally privileged information. If you are not the intended recipient of this message or if this message has been addressed to you in error, please immediately alert the sender by reply e-mail and then delete this message and any attachments. If you are not the intended recipient, you are notified that any use, dissemination, distribution, copying, or storage of this message or any attachment is strictly prohibited. E-MAIL CONFIDENTIALITY NOTICE: The contents of this e-mail message and any attachments are intended solely for the addressee(s) and may contain confidential and/or legally privileged information. If you are not the intended recipient of this message or if this message has been addressed to you in error, please immediately alert the sender by reply e-mail and then delete this message and any attachments. If you are not the intended recipient, you are notified that any use, dissemination, distribution, copying, or storage of this message or any attachment is strictly prohibited.
Hi Albert, Sorry for the inconvenience. Please note that I have recreated both the data network at starlingx (physical network of openstack) and the network of openstack. But I still have the same issue. Please find scheduler and conductor logs. # Scheduler Logs http://paste.openstack.org/show/806729/ ## Conductor logs http://paste.openstack.org/show/806727/ On Thu, Jun 17, 2021 at 6:55 PM Braden, Albert <C-Albert.Braden@charter.com> wrote:
Both links show the conductor log. It tells us that no valid host was found. The scheduler log should show why no valid host was found.
*From:* open infra <openinfradn@gmail.com> *Sent:* Thursday, June 17, 2021 2:02 AM *To:* Braden, Albert <C-Albert.Braden@charter.com> *Cc:* openstack-discuss <openstack-discuss@lists.openstack.org> *Subject:* Re: [EXTERNAL] Error creating VMs
*CAUTION:* The e-mail below is from an external source. Please exercise caution before opening attachments, clicking links, or following guidance.
Found nova-conductor and nova-scheduler logs [1],[2].
But I have no idea about what caused this issue.
[1] http://paste.openstack.org/show/806720/
[2] http://paste.openstack.org/show/806721/
On Wed, Jun 16, 2021 at 8:08 PM Braden, Albert < C-Albert.Braden@charter.com> wrote:
Someone will need to dig through the starlingx documentation and figure out where the log files are located. Do you want to do that?
*From:* open infra <openinfradn@gmail.com> *Sent:* Wednesday, June 16, 2021 9:12 AM *To:* Braden, Albert <C-Albert.Braden@charter.com> *Cc:* openstack-discuss <openstack-discuss@lists.openstack.org> *Subject:* Re: [EXTERNAL] Error creating VMs
*CAUTION:* The e-mail below is from an external source. Please exercise caution before opening attachments, clicking links, or following guidance.
I can see sm-scheduler.log not sure if its the correct log file you mentioned, but I can't see conductor log.
This environment have been deployed using starlingx.
On Wed, Jun 16, 2021 at 6:16 PM Braden, Albert < C-Albert.Braden@charter.com> wrote:
Can you check the scheduler and conductor logs on the controllers? There should be entries describing why the instance failed to schedule. You may need to set “debug=true” in nova.conf to get more details.
*From:* open infra <openinfradn@gmail.com> *Sent:* Wednesday, June 16, 2021 1:36 AM *To:* openstack-discuss <openstack-discuss@lists.openstack.org> *Subject:* [EXTERNAL] Error creating VMs
*CAUTION:* The e-mail below is from an external source. Please exercise caution before opening attachments, clicking links, or following guidance.
Hi
After setting up OpenStack environment using STX R5 (two controllers, two storage nodes and one worker), I have deployed a VM. VM ended up with ERROR status.
I highly appreciate if someone can guide to dig further (what logs to check ) or to fix this issue.
http://paste.openstack.org/show/806626/
Regards
Danishka
The contents of this e-mail message and any attachments are intended solely for the addressee(s) and may contain confidential and/or legally privileged information. If you are not the intended recipient of this message or if this message has been addressed to you in error, please immediately alert the sender by reply e-mail and then delete this message and any attachments. If you are not the intended recipient, you are notified that any use, dissemination, distribution, copying, or storage of this message or any attachment is strictly prohibited.
The contents of this e-mail message and any attachments are intended solely for the addressee(s) and may contain confidential and/or legally privileged information. If you are not the intended recipient of this message or if this message has been addressed to you in error, please immediately alert the sender by reply e-mail and then delete this message and any attachments. If you are not the intended recipient, you are notified that any use, dissemination, distribution, copying, or storage of this message or any attachment is strictly prohibited.
The contents of this e-mail message and any attachments are intended solely for the addressee(s) and may contain confidential and/or legally privileged information. If you are not the intended recipient of this message or if this message has been addressed to you in error, please immediately alert the sender by reply e-mail and then delete this message and any attachments. If you are not the intended recipient, you are notified that any use, dissemination, distribution, copying, or storage of this message or any attachment is strictly prohibited.
It looks like your RMQ is broken. What do you get from “rabbitmqctl cluster_status”? From: open infra <openinfradn@gmail.com> Sent: Thursday, June 17, 2021 9:58 AM To: Braden, Albert <C-Albert.Braden@charter.com> Cc: openstack-discuss <openstack-discuss@lists.openstack.org> Subject: Re: [EXTERNAL] Error creating VMs CAUTION: The e-mail below is from an external source. Please exercise caution before opening attachments, clicking links, or following guidance. Hi Albert, Sorry for the inconvenience. Please note that I have recreated both the data network at starlingx (physical network of openstack) and the network of openstack. But I still have the same issue. Please find scheduler and conductor logs. # Scheduler Logs http://paste.openstack.org/show/806729/ ## Conductor logs http://paste.openstack.org/show/806727/ On Thu, Jun 17, 2021 at 6:55 PM Braden, Albert <C-Albert.Braden@charter.com<mailto:C-Albert.Braden@charter.com>> wrote: Both links show the conductor log. It tells us that no valid host was found. The scheduler log should show why no valid host was found. From: open infra <openinfradn@gmail.com<mailto:openinfradn@gmail.com>> Sent: Thursday, June 17, 2021 2:02 AM To: Braden, Albert <C-Albert.Braden@charter.com<mailto:C-Albert.Braden@charter.com>> Cc: openstack-discuss <openstack-discuss@lists.openstack.org<mailto:openstack-discuss@lists.openstack.org>> Subject: Re: [EXTERNAL] Error creating VMs CAUTION: The e-mail below is from an external source. Please exercise caution before opening attachments, clicking links, or following guidance. Found nova-conductor and nova-scheduler logs [1],[2]. But I have no idea about what caused this issue. [1] http://paste.openstack.org/show/806720/ [2] http://paste.openstack.org/show/806721/ On Wed, Jun 16, 2021 at 8:08 PM Braden, Albert <C-Albert.Braden@charter.com<mailto:C-Albert.Braden@charter.com>> wrote: Someone will need to dig through the starlingx documentation and figure out where the log files are located. Do you want to do that? https://docs.starlingx.io/ From: open infra <openinfradn@gmail.com<mailto:openinfradn@gmail.com>> Sent: Wednesday, June 16, 2021 9:12 AM To: Braden, Albert <C-Albert.Braden@charter.com<mailto:C-Albert.Braden@charter.com>> Cc: openstack-discuss <openstack-discuss@lists.openstack.org<mailto:openstack-discuss@lists.openstack.org>> Subject: Re: [EXTERNAL] Error creating VMs CAUTION: The e-mail below is from an external source. Please exercise caution before opening attachments, clicking links, or following guidance. I can see sm-scheduler.log not sure if its the correct log file you mentioned, but I can't see conductor log. This environment have been deployed using starlingx. On Wed, Jun 16, 2021 at 6:16 PM Braden, Albert <C-Albert.Braden@charter.com<mailto:C-Albert.Braden@charter.com>> wrote: Can you check the scheduler and conductor logs on the controllers? There should be entries describing why the instance failed to schedule. You may need to set “debug=true” in nova.conf to get more details. From: open infra <openinfradn@gmail.com<mailto:openinfradn@gmail.com>> Sent: Wednesday, June 16, 2021 1:36 AM To: openstack-discuss <openstack-discuss@lists.openstack.org<mailto:openstack-discuss@lists.openstack.org>> Subject: [EXTERNAL] Error creating VMs CAUTION: The e-mail below is from an external source. Please exercise caution before opening attachments, clicking links, or following guidance. Hi After setting up OpenStack environment using STX R5 (two controllers, two storage nodes and one worker), I have deployed a VM. VM ended up with ERROR status. I highly appreciate if someone can guide to dig further (what logs to check ) or to fix this issue. http://paste.openstack.org/show/806626/ Regards Danishka The contents of this e-mail message and any attachments are intended solely for the addressee(s) and may contain confidential and/or legally privileged information. If you are not the intended recipient of this message or if this message has been addressed to you in error, please immediately alert the sender by reply e-mail and then delete this message and any attachments. If you are not the intended recipient, you are notified that any use, dissemination, distribution, copying, or storage of this message or any attachment is strictly prohibited. The contents of this e-mail message and any attachments are intended solely for the addressee(s) and may contain confidential and/or legally privileged information. If you are not the intended recipient of this message or if this message has been addressed to you in error, please immediately alert the sender by reply e-mail and then delete this message and any attachments. If you are not the intended recipient, you are notified that any use, dissemination, distribution, copying, or storage of this message or any attachment is strictly prohibited. The contents of this e-mail message and any attachments are intended solely for the addressee(s) and may contain confidential and/or legally privileged information. If you are not the intended recipient of this message or if this message has been addressed to you in error, please immediately alert the sender by reply e-mail and then delete this message and any attachments. If you are not the intended recipient, you are notified that any use, dissemination, distribution, copying, or storage of this message or any attachment is strictly prohibited. E-MAIL CONFIDENTIALITY NOTICE: The contents of this e-mail message and any attachments are intended solely for the addressee(s) and may contain confidential and/or legally privileged information. If you are not the intended recipient of this message or if this message has been addressed to you in error, please immediately alert the sender by reply e-mail and then delete this message and any attachments. If you are not the intended recipient, you are notified that any use, dissemination, distribution, copying, or storage of this message or any attachment is strictly prohibited.
controller-0:/var/log/pods$ sudo rabbitmqctl cluster_status Password: Cluster status of node rabbit@localhost ... [{nodes,[{disc,[rabbit@localhost]}]}, {running_nodes,[rabbit@localhost]}, {cluster_name,<<"rabbit@controller-0">>}, {partitions,[]}, {alarms,[{rabbit@localhost,[]}]}] On Thu, Jun 17, 2021 at 7:32 PM Braden, Albert <C-Albert.Braden@charter.com> wrote:
It looks like your RMQ is broken. What do you get from “rabbitmqctl cluster_status”?
*From:* open infra <openinfradn@gmail.com> *Sent:* Thursday, June 17, 2021 9:58 AM *To:* Braden, Albert <C-Albert.Braden@charter.com> *Cc:* openstack-discuss <openstack-discuss@lists.openstack.org> *Subject:* Re: [EXTERNAL] Error creating VMs
*CAUTION:* The e-mail below is from an external source. Please exercise caution before opening attachments, clicking links, or following guidance.
Hi Albert,
Sorry for the inconvenience.
Please note that I have recreated both the data network at starlingx (physical network of openstack) and the network of openstack.
But I still have the same issue. Please find scheduler and conductor logs.
# Scheduler Logs
http://paste.openstack.org/show/806729/
## Conductor logs
http://paste.openstack.org/show/806727/
On Thu, Jun 17, 2021 at 6:55 PM Braden, Albert < C-Albert.Braden@charter.com> wrote:
Both links show the conductor log. It tells us that no valid host was found. The scheduler log should show why no valid host was found.
*From:* open infra <openinfradn@gmail.com> *Sent:* Thursday, June 17, 2021 2:02 AM *To:* Braden, Albert <C-Albert.Braden@charter.com> *Cc:* openstack-discuss <openstack-discuss@lists.openstack.org> *Subject:* Re: [EXTERNAL] Error creating VMs
*CAUTION:* The e-mail below is from an external source. Please exercise caution before opening attachments, clicking links, or following guidance.
Found nova-conductor and nova-scheduler logs [1],[2].
But I have no idea about what caused this issue.
[1] http://paste.openstack.org/show/806720/
[2] http://paste.openstack.org/show/806721/
On Wed, Jun 16, 2021 at 8:08 PM Braden, Albert < C-Albert.Braden@charter.com> wrote:
Someone will need to dig through the starlingx documentation and figure out where the log files are located. Do you want to do that?
*From:* open infra <openinfradn@gmail.com> *Sent:* Wednesday, June 16, 2021 9:12 AM *To:* Braden, Albert <C-Albert.Braden@charter.com> *Cc:* openstack-discuss <openstack-discuss@lists.openstack.org> *Subject:* Re: [EXTERNAL] Error creating VMs
*CAUTION:* The e-mail below is from an external source. Please exercise caution before opening attachments, clicking links, or following guidance.
I can see sm-scheduler.log not sure if its the correct log file you mentioned, but I can't see conductor log.
This environment have been deployed using starlingx.
On Wed, Jun 16, 2021 at 6:16 PM Braden, Albert < C-Albert.Braden@charter.com> wrote:
Can you check the scheduler and conductor logs on the controllers? There should be entries describing why the instance failed to schedule. You may need to set “debug=true” in nova.conf to get more details.
*From:* open infra <openinfradn@gmail.com> *Sent:* Wednesday, June 16, 2021 1:36 AM *To:* openstack-discuss <openstack-discuss@lists.openstack.org> *Subject:* [EXTERNAL] Error creating VMs
*CAUTION:* The e-mail below is from an external source. Please exercise caution before opening attachments, clicking links, or following guidance.
Hi
After setting up OpenStack environment using STX R5 (two controllers, two storage nodes and one worker), I have deployed a VM. VM ended up with ERROR status.
I highly appreciate if someone can guide to dig further (what logs to check ) or to fix this issue.
http://paste.openstack.org/show/806626/
Regards
Danishka
The contents of this e-mail message and any attachments are intended solely for the addressee(s) and may contain confidential and/or legally privileged information. If you are not the intended recipient of this message or if this message has been addressed to you in error, please immediately alert the sender by reply e-mail and then delete this message and any attachments. If you are not the intended recipient, you are notified that any use, dissemination, distribution, copying, or storage of this message or any attachment is strictly prohibited.
The contents of this e-mail message and any attachments are intended solely for the addressee(s) and may contain confidential and/or legally privileged information. If you are not the intended recipient of this message or if this message has been addressed to you in error, please immediately alert the sender by reply e-mail and then delete this message and any attachments. If you are not the intended recipient, you are notified that any use, dissemination, distribution, copying, or storage of this message or any attachment is strictly prohibited.
The contents of this e-mail message and any attachments are intended solely for the addressee(s) and may contain confidential and/or legally privileged information. If you are not the intended recipient of this message or if this message has been addressed to you in error, please immediately alert the sender by reply e-mail and then delete this message and any attachments. If you are not the intended recipient, you are notified that any use, dissemination, distribution, copying, or storage of this message or any attachment is strictly prohibited.
The contents of this e-mail message and any attachments are intended solely for the addressee(s) and may contain confidential and/or legally privileged information. If you are not the intended recipient of this message or if this message has been addressed to you in error, please immediately alert the sender by reply e-mail and then delete this message and any attachments. If you are not the intended recipient, you are notified that any use, dissemination, distribution, copying, or storage of this message or any attachment is strictly prohibited.
It looks like RMQ is up but services can’t connect to it. What do you see in the RMQ web interface? This might be a clue: ERROR oslo.messaging._drivers.impl_rabbit [req-b30bb3d6-0ab1-4dbb-aed9-1d9cece3eac7 d9f7048c1cd947cfa8ecef128a6cee89 e8813293073545f99658adbec2f80c1d - default default] Connection failed: failed to resolve broker hostname (retrying in 0 seconds): OSError: failed to resolve broker hostname Check for a typo in your config that points services to an incorrect RMQ hostname, or a networking issue that prevents them from connecting. From: open infra <openinfradn@gmail.com> Sent: Thursday, June 17, 2021 10:06 AM To: Braden, Albert <C-Albert.Braden@charter.com> Cc: openstack-discuss <openstack-discuss@lists.openstack.org> Subject: Re: [EXTERNAL] Error creating VMs CAUTION: The e-mail below is from an external source. Please exercise caution before opening attachments, clicking links, or following guidance. controller-0:/var/log/pods$ sudo rabbitmqctl cluster_status Password: Cluster status of node rabbit@localhost ... [{nodes,[{disc,[rabbit@localhost]}]}, {running_nodes,[rabbit@localhost]}, {cluster_name,<<"rabbit@controller-0">>}, {partitions,[]}, {alarms,[{rabbit@localhost,[]}]}] On Thu, Jun 17, 2021 at 7:32 PM Braden, Albert <C-Albert.Braden@charter.com<mailto:C-Albert.Braden@charter.com>> wrote: It looks like your RMQ is broken. What do you get from “rabbitmqctl cluster_status”? From: open infra <openinfradn@gmail.com<mailto:openinfradn@gmail.com>> Sent: Thursday, June 17, 2021 9:58 AM To: Braden, Albert <C-Albert.Braden@charter.com<mailto:C-Albert.Braden@charter.com>> Cc: openstack-discuss <openstack-discuss@lists.openstack.org<mailto:openstack-discuss@lists.openstack.org>> Subject: Re: [EXTERNAL] Error creating VMs CAUTION: The e-mail below is from an external source. Please exercise caution before opening attachments, clicking links, or following guidance. Hi Albert, Sorry for the inconvenience. Please note that I have recreated both the data network at starlingx (physical network of openstack) and the network of openstack. But I still have the same issue. Please find scheduler and conductor logs. # Scheduler Logs http://paste.openstack.org/show/806729/ ## Conductor logs http://paste.openstack.org/show/806727/ On Thu, Jun 17, 2021 at 6:55 PM Braden, Albert <C-Albert.Braden@charter.com<mailto:C-Albert.Braden@charter.com>> wrote: Both links show the conductor log. It tells us that no valid host was found. The scheduler log should show why no valid host was found. From: open infra <openinfradn@gmail.com<mailto:openinfradn@gmail.com>> Sent: Thursday, June 17, 2021 2:02 AM To: Braden, Albert <C-Albert.Braden@charter.com<mailto:C-Albert.Braden@charter.com>> Cc: openstack-discuss <openstack-discuss@lists.openstack.org<mailto:openstack-discuss@lists.openstack.org>> Subject: Re: [EXTERNAL] Error creating VMs CAUTION: The e-mail below is from an external source. Please exercise caution before opening attachments, clicking links, or following guidance. Found nova-conductor and nova-scheduler logs [1],[2]. But I have no idea about what caused this issue. [1] http://paste.openstack.org/show/806720/ [2] http://paste.openstack.org/show/806721/ On Wed, Jun 16, 2021 at 8:08 PM Braden, Albert <C-Albert.Braden@charter.com<mailto:C-Albert.Braden@charter.com>> wrote: Someone will need to dig through the starlingx documentation and figure out where the log files are located. Do you want to do that? https://docs.starlingx.io/ From: open infra <openinfradn@gmail.com<mailto:openinfradn@gmail.com>> Sent: Wednesday, June 16, 2021 9:12 AM To: Braden, Albert <C-Albert.Braden@charter.com<mailto:C-Albert.Braden@charter.com>> Cc: openstack-discuss <openstack-discuss@lists.openstack.org<mailto:openstack-discuss@lists.openstack.org>> Subject: Re: [EXTERNAL] Error creating VMs CAUTION: The e-mail below is from an external source. Please exercise caution before opening attachments, clicking links, or following guidance. I can see sm-scheduler.log not sure if its the correct log file you mentioned, but I can't see conductor log. This environment have been deployed using starlingx. On Wed, Jun 16, 2021 at 6:16 PM Braden, Albert <C-Albert.Braden@charter.com<mailto:C-Albert.Braden@charter.com>> wrote: Can you check the scheduler and conductor logs on the controllers? There should be entries describing why the instance failed to schedule. You may need to set “debug=true” in nova.conf to get more details. From: open infra <openinfradn@gmail.com<mailto:openinfradn@gmail.com>> Sent: Wednesday, June 16, 2021 1:36 AM To: openstack-discuss <openstack-discuss@lists.openstack.org<mailto:openstack-discuss@lists.openstack.org>> Subject: [EXTERNAL] Error creating VMs CAUTION: The e-mail below is from an external source. Please exercise caution before opening attachments, clicking links, or following guidance. Hi After setting up OpenStack environment using STX R5 (two controllers, two storage nodes and one worker), I have deployed a VM. VM ended up with ERROR status. I highly appreciate if someone can guide to dig further (what logs to check ) or to fix this issue. http://paste.openstack.org/show/806626/ Regards Danishka The contents of this e-mail message and any attachments are intended solely for the addressee(s) and may contain confidential and/or legally privileged information. If you are not the intended recipient of this message or if this message has been addressed to you in error, please immediately alert the sender by reply e-mail and then delete this message and any attachments. If you are not the intended recipient, you are notified that any use, dissemination, distribution, copying, or storage of this message or any attachment is strictly prohibited. The contents of this e-mail message and any attachments are intended solely for the addressee(s) and may contain confidential and/or legally privileged information. If you are not the intended recipient of this message or if this message has been addressed to you in error, please immediately alert the sender by reply e-mail and then delete this message and any attachments. If you are not the intended recipient, you are notified that any use, dissemination, distribution, copying, or storage of this message or any attachment is strictly prohibited. The contents of this e-mail message and any attachments are intended solely for the addressee(s) and may contain confidential and/or legally privileged information. If you are not the intended recipient of this message or if this message has been addressed to you in error, please immediately alert the sender by reply e-mail and then delete this message and any attachments. If you are not the intended recipient, you are notified that any use, dissemination, distribution, copying, or storage of this message or any attachment is strictly prohibited. The contents of this e-mail message and any attachments are intended solely for the addressee(s) and may contain confidential and/or legally privileged information. If you are not the intended recipient of this message or if this message has been addressed to you in error, please immediately alert the sender by reply e-mail and then delete this message and any attachments. If you are not the intended recipient, you are notified that any use, dissemination, distribution, copying, or storage of this message or any attachment is strictly prohibited. E-MAIL CONFIDENTIALITY NOTICE: The contents of this e-mail message and any attachments are intended solely for the addressee(s) and may contain confidential and/or legally privileged information. If you are not the intended recipient of this message or if this message has been addressed to you in error, please immediately alert the sender by reply e-mail and then delete this message and any attachments. If you are not the intended recipient, you are notified that any use, dissemination, distribution, copying, or storage of this message or any attachment is strictly prohibited.
Another instance created at 2021-06-17T16:10:06Z, and it also failed. I can't see rabbitmq errors around that timestamp. scheduler log http://paste.openstack.org/show/806737/ <https://www.google.com/url?q=http://paste.openstack.org/show/806737/&sa=D&source=hangouts&ust=1624034044021000&usg=AFQjCNHSJ6dWR_hMfE6e6YUw14demT_mZA> conducter log http://paste.openstack.org/show/806738/ <https://www.google.com/url?q=http://paste.openstack.org/show/806738/&sa=D&source=hangouts&ust=1624034079136000&usg=AFQjCNHuMa5Eerlpx0-wlOM-fBaDXpHptQ> At 2021-06-17 15:52:38.810, system was able to connect to AMQP http://paste.openstack.org/show/806739/ <https://www.google.com/url?q=http://paste.openstack.org/show/806739/&sa=D&source=hangouts&ust=1624034275886000&usg=AFQjCNFree0y1N8q-PQnd6Unz5AQZG5q2g> controller-0:/var/log/pods$ sudo rabbitmqctl list_queues Password: Listing queues ... sysinv.ceph_manager.192.168.204.1 0 sysinv.ceph_manager 0 barbican.workers 0 sysinv.fpga_agent_manager.controller-0 0 barbican.workers.barbican.queue 0 sysinv.conductor_manager 0 sysinv.agent_manager_fanout_fed76e414eb04da084ab35a1c27e1bf1 0 sysinv.agent_manager 0 sysinv.ceph_manager_fanout_d585fb522f46431da60741573a7f8575 0 notifications.info 0 sysinv.conductor_manager_fanout_cba8926fa47f4780a9e17f3d9b889500 0 sysinv.fpga_agent_manager 0 sysinv-keystone-listener-workers 0 sysinv.fpga_agent_manager_fanout_f00a01bfd3f54a64860f8d7454e9a78e 0 sysinv.agent_manager.controller-0 0 barbican.workers_fanout_2c97c319faa943e88eaed4c101e530c7 0 sysinv.conductor_manager.controller-0 0 On Thu, Jun 17, 2021 at 7:51 PM Braden, Albert <C-Albert.Braden@charter.com> wrote:
It looks like RMQ is up but services can’t connect to it. What do you see in the RMQ web interface?
This might be a clue:
ERROR oslo.messaging._drivers.impl_rabbit [req-b30bb3d6-0ab1-4dbb-aed9-1d9cece3eac7 d9f7048c1cd947cfa8ecef128a6cee89 e8813293073545f99658adbec2f80c1d - default default] Connection failed: failed to resolve broker hostname (retrying in 0 seconds): OSError: failed to resolve broker hostname
Check for a typo in your config that points services to an incorrect RMQ hostname, or a networking issue that prevents them from connecting.
*From:* open infra <openinfradn@gmail.com> *Sent:* Thursday, June 17, 2021 10:06 AM *To:* Braden, Albert <C-Albert.Braden@charter.com> *Cc:* openstack-discuss <openstack-discuss@lists.openstack.org> *Subject:* Re: [EXTERNAL] Error creating VMs
*CAUTION:* The e-mail below is from an external source. Please exercise caution before opening attachments, clicking links, or following guidance.
controller-0:/var/log/pods$ sudo rabbitmqctl cluster_status Password: Cluster status of node rabbit@localhost ... [{nodes,[{disc,[rabbit@localhost]}]}, {running_nodes,[rabbit@localhost]}, {cluster_name,<<"rabbit@controller-0">>}, {partitions,[]}, {alarms,[{rabbit@localhost,[]}]}]
On Thu, Jun 17, 2021 at 7:32 PM Braden, Albert < C-Albert.Braden@charter.com> wrote:
It looks like your RMQ is broken. What do you get from “rabbitmqctl cluster_status”?
*From:* open infra <openinfradn@gmail.com> *Sent:* Thursday, June 17, 2021 9:58 AM *To:* Braden, Albert <C-Albert.Braden@charter.com> *Cc:* openstack-discuss <openstack-discuss@lists.openstack.org> *Subject:* Re: [EXTERNAL] Error creating VMs
*CAUTION:* The e-mail below is from an external source. Please exercise caution before opening attachments, clicking links, or following guidance.
Hi Albert,
Sorry for the inconvenience.
Please note that I have recreated both the data network at starlingx (physical network of openstack) and the network of openstack.
But I still have the same issue. Please find scheduler and conductor logs.
# Scheduler Logs
http://paste.openstack.org/show/806729/
## Conductor logs
http://paste.openstack.org/show/806727/
On Thu, Jun 17, 2021 at 6:55 PM Braden, Albert < C-Albert.Braden@charter.com> wrote:
Both links show the conductor log. It tells us that no valid host was found. The scheduler log should show why no valid host was found.
*From:* open infra <openinfradn@gmail.com> *Sent:* Thursday, June 17, 2021 2:02 AM *To:* Braden, Albert <C-Albert.Braden@charter.com> *Cc:* openstack-discuss <openstack-discuss@lists.openstack.org> *Subject:* Re: [EXTERNAL] Error creating VMs
*CAUTION:* The e-mail below is from an external source. Please exercise caution before opening attachments, clicking links, or following guidance.
Found nova-conductor and nova-scheduler logs [1],[2].
But I have no idea about what caused this issue.
[1] http://paste.openstack.org/show/806720/
[2] http://paste.openstack.org/show/806721/
On Wed, Jun 16, 2021 at 8:08 PM Braden, Albert < C-Albert.Braden@charter.com> wrote:
Someone will need to dig through the starlingx documentation and figure out where the log files are located. Do you want to do that?
*From:* open infra <openinfradn@gmail.com> *Sent:* Wednesday, June 16, 2021 9:12 AM *To:* Braden, Albert <C-Albert.Braden@charter.com> *Cc:* openstack-discuss <openstack-discuss@lists.openstack.org> *Subject:* Re: [EXTERNAL] Error creating VMs
*CAUTION:* The e-mail below is from an external source. Please exercise caution before opening attachments, clicking links, or following guidance.
I can see sm-scheduler.log not sure if its the correct log file you mentioned, but I can't see conductor log.
This environment have been deployed using starlingx.
On Wed, Jun 16, 2021 at 6:16 PM Braden, Albert < C-Albert.Braden@charter.com> wrote:
Can you check the scheduler and conductor logs on the controllers? There should be entries describing why the instance failed to schedule. You may need to set “debug=true” in nova.conf to get more details.
*From:* open infra <openinfradn@gmail.com> *Sent:* Wednesday, June 16, 2021 1:36 AM *To:* openstack-discuss <openstack-discuss@lists.openstack.org> *Subject:* [EXTERNAL] Error creating VMs
*CAUTION:* The e-mail below is from an external source. Please exercise caution before opening attachments, clicking links, or following guidance.
Hi
After setting up OpenStack environment using STX R5 (two controllers, two storage nodes and one worker), I have deployed a VM. VM ended up with ERROR status.
I highly appreciate if someone can guide to dig further (what logs to check ) or to fix this issue.
http://paste.openstack.org/show/806626/
Regards
Danishka
The contents of this e-mail message and any attachments are intended solely for the addressee(s) and may contain confidential and/or legally privileged information. If you are not the intended recipient of this message or if this message has been addressed to you in error, please immediately alert the sender by reply e-mail and then delete this message and any attachments. If you are not the intended recipient, you are notified that any use, dissemination, distribution, copying, or storage of this message or any attachment is strictly prohibited.
The contents of this e-mail message and any attachments are intended solely for the addressee(s) and may contain confidential and/or legally privileged information. If you are not the intended recipient of this message or if this message has been addressed to you in error, please immediately alert the sender by reply e-mail and then delete this message and any attachments. If you are not the intended recipient, you are notified that any use, dissemination, distribution, copying, or storage of this message or any attachment is strictly prohibited.
The contents of this e-mail message and any attachments are intended solely for the addressee(s) and may contain confidential and/or legally privileged information. If you are not the intended recipient of this message or if this message has been addressed to you in error, please immediately alert the sender by reply e-mail and then delete this message and any attachments. If you are not the intended recipient, you are notified that any use, dissemination, distribution, copying, or storage of this message or any attachment is strictly prohibited.
The contents of this e-mail message and any attachments are intended solely for the addressee(s) and may contain confidential and/or legally privileged information. If you are not the intended recipient of this message or if this message has been addressed to you in error, please immediately alert the sender by reply e-mail and then delete this message and any attachments. If you are not the intended recipient, you are notified that any use, dissemination, distribution, copying, or storage of this message or any attachment is strictly prohibited.
The contents of this e-mail message and any attachments are intended solely for the addressee(s) and may contain confidential and/or legally privileged information. If you are not the intended recipient of this message or if this message has been addressed to you in error, please immediately alert the sender by reply e-mail and then delete this message and any attachments. If you are not the intended recipient, you are notified that any use, dissemination, distribution, copying, or storage of this message or any attachment is strictly prohibited.
I see “no valid host” at 10:48:39 in the conductor log: 2021-06-17T10:48:39.65107847Z stdout F nova.exception.NoValidHost: No valid host was found. In the scheduler log at 10:37:43 we see the scheduler starting and the RMQ error, followed by the scheduling failure at 10:48:39. It looks like the scheduler can’t connect to RMQ. 2021-06-17T10:37:43.038362995Z stdout F 2021-06-17 10:37:43.038 1 INFO nova.service [-] Starting scheduler node (version 21.2.1) 2021-06-17T10:37:43.08082925Z stdout F 2021-06-17 10:37:43.079 1 ERROR oslo.messaging._drivers.impl_rabbit [req-775b2b25-b9ef-4642-a6fd-7574eab7cc37 - - - - -] Connection failed: failed to resolve broker hostname (retrying in 0 seconds): OSError: failed to resolve broker hostname 2021-06-17T10:48:39.197318418Z stdout F 2021-06-17 10:48:39.196 1 INFO nova.scheduler.manager [req-b30bb3d6-0ab1-4dbb-aed9-1d9cece3eac7 d9f7048c1cd947cfa8ecef128a6cee89 e8813293073545f99658adbec2f80c1d - default default] Got no allocation candidates from the Placement API. This could be due to insufficient resources or a temporary occurrence as compute nodes start up. From: open infra <openinfradn@gmail.com> Sent: Thursday, June 17, 2021 12:38 PM To: Braden, Albert <C-Albert.Braden@charter.com> Cc: openstack-discuss <openstack-discuss@lists.openstack.org> Subject: Re: [EXTERNAL] Error creating VMs CAUTION: The e-mail below is from an external source. Please exercise caution before opening attachments, clicking links, or following guidance. Another instance created at 2021-06-17T16:10:06Z, and it also failed. I can't see rabbitmq errors around that timestamp. scheduler log http://paste.openstack.org/show/806737/<https://www.google.com/url?q=http://paste.openstack.org/show/806737/&sa=D&source=hangouts&ust=1624034044021000&usg=AFQjCNHSJ6dWR_hMfE6e6YUw14demT_mZA> conducter log http://paste.openstack.org/show/806738/<https://www.google.com/url?q=http://paste.openstack.org/show/806738/&sa=D&source=hangouts&ust=1624034079136000&usg=AFQjCNHuMa5Eerlpx0-wlOM-fBaDXpHptQ> At 2021-06-17 15:52:38.810, system was able to connect to AMQP http://paste.openstack.org/show/806739/<https://www.google.com/url?q=http://paste.openstack.org/show/806739/&sa=D&source=hangouts&ust=1624034275886000&usg=AFQjCNFree0y1N8q-PQnd6Unz5AQZG5q2g> controller-0:/var/log/pods$ sudo rabbitmqctl list_queues Password: Listing queues ... sysinv.ceph_manager.192.168.204.1 0 sysinv.ceph_manager 0 barbican.workers 0 sysinv.fpga_agent_manager.controller-0 0 barbican.workers.barbican.queue 0 sysinv.conductor_manager 0 sysinv.agent_manager_fanout_fed76e414eb04da084ab35a1c27e1bf1 0 sysinv.agent_manager 0 sysinv.ceph_manager_fanout_d585fb522f46431da60741573a7f8575 0 notifications.info<http://notifications.info> 0 sysinv.conductor_manager_fanout_cba8926fa47f4780a9e17f3d9b889500 0 sysinv.fpga_agent_manager 0 sysinv-keystone-listener-workers 0 sysinv.fpga_agent_manager_fanout_f00a01bfd3f54a64860f8d7454e9a78e 0 sysinv.agent_manager.controller-0 0 barbican.workers_fanout_2c97c319faa943e88eaed4c101e530c7 0 sysinv.conductor_manager.controller-0 0 On Thu, Jun 17, 2021 at 7:51 PM Braden, Albert <C-Albert.Braden@charter.com<mailto:C-Albert.Braden@charter.com>> wrote: It looks like RMQ is up but services can’t connect to it. What do you see in the RMQ web interface? This might be a clue: ERROR oslo.messaging._drivers.impl_rabbit [req-b30bb3d6-0ab1-4dbb-aed9-1d9cece3eac7 d9f7048c1cd947cfa8ecef128a6cee89 e8813293073545f99658adbec2f80c1d - default default] Connection failed: failed to resolve broker hostname (retrying in 0 seconds): OSError: failed to resolve broker hostname Check for a typo in your config that points services to an incorrect RMQ hostname, or a networking issue that prevents them from connecting. From: open infra <openinfradn@gmail.com<mailto:openinfradn@gmail.com>> Sent: Thursday, June 17, 2021 10:06 AM To: Braden, Albert <C-Albert.Braden@charter.com<mailto:C-Albert.Braden@charter.com>> Cc: openstack-discuss <openstack-discuss@lists.openstack.org<mailto:openstack-discuss@lists.openstack.org>> Subject: Re: [EXTERNAL] Error creating VMs CAUTION: The e-mail below is from an external source. Please exercise caution before opening attachments, clicking links, or following guidance. controller-0:/var/log/pods$ sudo rabbitmqctl cluster_status Password: Cluster status of node rabbit@localhost ... [{nodes,[{disc,[rabbit@localhost]}]}, {running_nodes,[rabbit@localhost]}, {cluster_name,<<"rabbit@controller-0">>}, {partitions,[]}, {alarms,[{rabbit@localhost,[]}]}] On Thu, Jun 17, 2021 at 7:32 PM Braden, Albert <C-Albert.Braden@charter.com<mailto:C-Albert.Braden@charter.com>> wrote: It looks like your RMQ is broken. What do you get from “rabbitmqctl cluster_status”? From: open infra <openinfradn@gmail.com<mailto:openinfradn@gmail.com>> Sent: Thursday, June 17, 2021 9:58 AM To: Braden, Albert <C-Albert.Braden@charter.com<mailto:C-Albert.Braden@charter.com>> Cc: openstack-discuss <openstack-discuss@lists.openstack.org<mailto:openstack-discuss@lists.openstack.org>> Subject: Re: [EXTERNAL] Error creating VMs CAUTION: The e-mail below is from an external source. Please exercise caution before opening attachments, clicking links, or following guidance. Hi Albert, Sorry for the inconvenience. Please note that I have recreated both the data network at starlingx (physical network of openstack) and the network of openstack. But I still have the same issue. Please find scheduler and conductor logs. # Scheduler Logs E-MAIL CONFIDENTIALITY NOTICE: The contents of this e-mail message and any attachments are intended solely for the addressee(s) and may contain confidential and/or legally privileged information. If you are not the intended recipient of this message or if this message has been addressed to you in error, please immediately alert the sender by reply e-mail and then delete this message and any attachments. If you are not the intended recipient, you are notified that any use, dissemination, distribution, copying, or storage of this message or any attachment is strictly prohibited.
VM provisioning is resolved. But still having an issue with openstack networking. "Unable to create the network. No tenant network is available for allocation." http://paste.openstack.org/show/807014/ On Thu, Jun 17, 2021 at 11:14 PM Braden, Albert <C-Albert.Braden@charter.com> wrote:
I see “no valid host” at 10:48:39 in the conductor log:
2021-06-17T10:48:39.65107847Z stdout F nova.exception.NoValidHost: No valid host was found.
In the scheduler log at 10:37:43 we see the scheduler starting and the RMQ error, followed by the scheduling failure at 10:48:39. It looks like the scheduler can’t connect to RMQ.
2021-06-17T10:37:43.038362995Z stdout F 2021-06-17 10:37:43.038 1 INFO nova.service [-] Starting scheduler node (version 21.2.1)
2021-06-17T10:37:43.08082925Z stdout F 2021-06-17 10:37:43.079 1 ERROR oslo.messaging._drivers.impl_rabbit [req-775b2b25-b9ef-4642-a6fd-7574eab7cc37 - - - - -] Connection failed: failed to resolve broker hostname (retrying in 0 seconds): OSError: failed to resolve broker hostname
2021-06-17T10:48:39.197318418Z stdout F 2021-06-17 10:48:39.196 1 INFO nova.scheduler.manager [req-b30bb3d6-0ab1-4dbb-aed9-1d9cece3eac7 d9f7048c1cd947cfa8ecef128a6cee89 e8813293073545f99658adbec2f80c1d - default default] Got no allocation candidates from the Placement API. This could be due to insufficient resources or a temporary occurrence as compute nodes start up.
*From:* open infra <openinfradn@gmail.com> *Sent:* Thursday, June 17, 2021 12:38 PM *To:* Braden, Albert <C-Albert.Braden@charter.com> *Cc:* openstack-discuss <openstack-discuss@lists.openstack.org> *Subject:* Re: [EXTERNAL] Error creating VMs
*CAUTION:* The e-mail below is from an external source. Please exercise caution before opening attachments, clicking links, or following guidance.
Another instance created at 2021-06-17T16:10:06Z, and it also failed.
I can't see rabbitmq errors around that timestamp.
scheduler log http://paste.openstack.org/show/806737/ <https://www.google.com/url?q=http://paste.openstack.org/show/806737/&sa=D&source=hangouts&ust=1624034044021000&usg=AFQjCNHSJ6dWR_hMfE6e6YUw14demT_mZA>
conducter log http://paste.openstack.org/show/806738/ <https://www.google.com/url?q=http://paste.openstack.org/show/806738/&sa=D&source=hangouts&ust=1624034079136000&usg=AFQjCNHuMa5Eerlpx0-wlOM-fBaDXpHptQ>
At 2021-06-17 15:52:38.810, system was able to connect to AMQP
http://paste.openstack.org/show/806739/ <https://www.google.com/url?q=http://paste.openstack.org/show/806739/&sa=D&source=hangouts&ust=1624034275886000&usg=AFQjCNFree0y1N8q-PQnd6Unz5AQZG5q2g>
controller-0:/var/log/pods$ sudo rabbitmqctl list_queues Password: Listing queues ... sysinv.ceph_manager.192.168.204.1 0 sysinv.ceph_manager 0 barbican.workers 0 sysinv.fpga_agent_manager.controller-0 0 barbican.workers.barbican.queue 0 sysinv.conductor_manager 0 sysinv.agent_manager_fanout_fed76e414eb04da084ab35a1c27e1bf1 0 sysinv.agent_manager 0 sysinv.ceph_manager_fanout_d585fb522f46431da60741573a7f8575 0 notifications.info 0 sysinv.conductor_manager_fanout_cba8926fa47f4780a9e17f3d9b889500 0 sysinv.fpga_agent_manager 0 sysinv-keystone-listener-workers 0 sysinv.fpga_agent_manager_fanout_f00a01bfd3f54a64860f8d7454e9a78e 0 sysinv.agent_manager.controller-0 0 barbican.workers_fanout_2c97c319faa943e88eaed4c101e530c7 0 sysinv.conductor_manager.controller-0 0
On Thu, Jun 17, 2021 at 7:51 PM Braden, Albert < C-Albert.Braden@charter.com> wrote:
It looks like RMQ is up but services can’t connect to it. What do you see in the RMQ web interface?
This might be a clue:
ERROR oslo.messaging._drivers.impl_rabbit [req-b30bb3d6-0ab1-4dbb-aed9-1d9cece3eac7 d9f7048c1cd947cfa8ecef128a6cee89 e8813293073545f99658adbec2f80c1d - default default] Connection failed: failed to resolve broker hostname (retrying in 0 seconds): OSError: failed to resolve broker hostname
Check for a typo in your config that points services to an incorrect RMQ hostname, or a networking issue that prevents them from connecting.
*From:* open infra <openinfradn@gmail.com> *Sent:* Thursday, June 17, 2021 10:06 AM *To:* Braden, Albert <C-Albert.Braden@charter.com> *Cc:* openstack-discuss <openstack-discuss@lists.openstack.org> *Subject:* Re: [EXTERNAL] Error creating VMs
*CAUTION:* The e-mail below is from an external source. Please exercise caution before opening attachments, clicking links, or following guidance.
controller-0:/var/log/pods$ sudo rabbitmqctl cluster_status Password: Cluster status of node rabbit@localhost ... [{nodes,[{disc,[rabbit@localhost]}]}, {running_nodes,[rabbit@localhost]}, {cluster_name,<<"rabbit@controller-0">>}, {partitions,[]}, {alarms,[{rabbit@localhost,[]}]}]
On Thu, Jun 17, 2021 at 7:32 PM Braden, Albert < C-Albert.Braden@charter.com> wrote:
It looks like your RMQ is broken. What do you get from “rabbitmqctl cluster_status”?
*From:* open infra <openinfradn@gmail.com> *Sent:* Thursday, June 17, 2021 9:58 AM *To:* Braden, Albert <C-Albert.Braden@charter.com> *Cc:* openstack-discuss <openstack-discuss@lists.openstack.org> *Subject:* Re: [EXTERNAL] Error creating VMs
*CAUTION:* The e-mail below is from an external source. Please exercise caution before opening attachments, clicking links, or following guidance.
Hi Albert,
Sorry for the inconvenience.
Please note that I have recreated both the data network at starlingx (physical network of openstack) and the network of openstack.
But I still have the same issue. Please find scheduler and conductor logs.
# Scheduler Logs
The contents of this e-mail message and any attachments are intended solely for the addressee(s) and may contain confidential and/or legally privileged information. If you are not the intended recipient of this message or if this message has been addressed to you in error, please immediately alert the sender by reply e-mail and then delete this message and any attachments. If you are not the intended recipient, you are notified that any use, dissemination, distribution, copying, or storage of this message or any attachment is strictly prohibited.
participants (2)
-
Braden, Albert
-
open infra