April 2024 - openstack-discuss - lists.openstack.org

[nova][libvirtd] internal error: unable to execute QEMU command 'query-named-block-nodes'
by Satish Patel 22 Oct '24

22 Oct '24

Folks, I have a ceph cluster and recently I configured rbd-mirror to replicate all data to remove ceph cluster for disaster recovery. Yesterday for POC we did a hard cutover on ceph and point openstack to the new cluster. All other vms came back up fine but 2 vms stuck in this error state in libvirt logs 2024-01-31 22:44:37.591+0000: 474597: error : qemuMonitorJSONCheckErrorFull:399 : internal error: unable to execute QEMU command 'query-named-block-nodes': cannot read image start for probe: Permission denied If it's a wider issue than it should impact all the vms but why only two VM stuck and not starting and libvirt giving me this error in logs nova-compute logs also showing same error 2024-01-31 22:44:40.925 7 INFO nova.compute.manager [None req-b33485b5-8740-48ae-8b5b-a440de3f11a4 c48fcfb9347f413f92fcece065644b00 ca5c652478c7429e964257990800e9cb - - default default] [instance: 2de0f880-77c7-4d2c-9e01-898c57ad3693] Successfully reverted task state from powering-on on failure for instance. 2024-01-31 22:44:40.944 7 ERROR oslo_messaging.rpc.server [None req-b33485b5-8740-48ae-8b5b-a440de3f11a4 c48fcfb9347f413f92fcece065644b00 ca5c652478c7429e964257990800e9cb - - default default] Exception during message handling: libvirt.libvirtError: internal error: unable to execute QEMU command 'query-named-block-nodes': cannot read image start for probe: Permission denied

3 8

[kuryr][tc] Final call for Kuryr PTL/DPL liaisons
by Ghanshyam Mann 15 Oct '24

15 Oct '24

Hello Everyone, Kuryr project ends up being a leaderless project for the 2024.2 Dalmatian cycle[1]. There is no volunteer to serve as PTL or DPL liaisons yet. As the next step, I have proposed to mark this project Inactive: - https://review.opendev.org/c/openstack/governance/+/916945 If you would like to serve as a leader and help maintain this project, this is the right time to step up by replying to this email or commenting in gerrit change 916945. [1] https://etherpad.opendev.org/p/2024.2-leaderless#L99 Ref: https://governance.openstack.org/tc/reference/emerging-technology-and-inact… -gmann

4 10

Kolla Ansible on Ubuntu 20.04 - cloud-init & other network issues
by Tobias McNulty 10 Oct '24

10 Oct '24

Hi, I'm attempting to use Kolla Ansible 14.6.0 to deploy OpenStack Yoga on a small 3-node Ubuntu 20.04 cluster. The nodes have 128 GB RAM each, dual Xeon processors, and dual 10G Intel NICs. The NICs are connected to access ports on a 10G switch with separate VLANs for the local and external networks. All the playbooks run cleanly, but cloud-init is failing in the Ubuntu 20.04 and 22.04 VMs I attempt to boot. The VM images are unmodified from https://cloud-images.ubuntu.com/, and cloud-init works fine if I mount a second volume with user-data. The error is a timeout attempting to reach 169.254.169.254. This occurs both when booting a VM in an internal routed network and directly in an external network. I tried various neutron plugin agents (ovn, linuxbridge, and openvswitch both with and without firewall_driver = openvswitch <https://docs.openstack.org/kolla-ansible/latest/reference/networking/neutro…>) first with a clean install of the entire OS each time, all with the same result. Running tcpdump looking for 169.254.169.254 shows nothing. As a possible clue, the virtual NICs are unable to pass any traffic (e.g., to reach an external DHCP server) unless I completely disable port security on the interface (even if the associated security group is wide open). But disabling port security does not fix cloud-init (not to mention I don't really want to disable port security). Are there any additional requirements related to deploying OpenStack with Kolla on Ubuntu 20.04? This is a fairly vanilla configuration using the multinode inventory as a starting point. I tried to follow the Quick Start <https://docs.openstack.org/kolla-ansible/yoga/user/quickstart.html> as closely as possible; the only material difference I see is that I'm using the same 3 nodes for control + compute. I am using MAAS so it's easy to get a clean OS install on all three nodes ahead of each attempt. I plan to try again with the standard (non-HWE) kernel just in case, but otherwise I am running out of ideas. In case of any additional clues, here are my globals.yml and inventory file, along with the playbook I'm using to configure the network, images, VMs, etc., after bootstrapping the cluster: https://gist.github.com/tobiasmcnulty/7dbbdbc67abc08cbb013bf5983852ed6 Thank you in advance for any advice! Cheers, Tobias

5 9

Live migration fails
by Szabo, Istvan (Agoda) 07 Aug '24

07 Aug '24

Hi, I have couple of compute nodes where the live migration fails with existing vms. When I quickly spawn a vm and try live migration it works so I assume shouldn't be a big problem with the compute node. However I have many existing vms where it fails with a servername not found. /var/log/nova/nova-conductor.log:2021-04-21 14:47:12.605 227612 ERROR nova.conductor.tasks.migrate [req-f4067a26-a233-4673-8c07-9a8a290980b0 dce35e6eceea4312bb0baa0510cef363 ca7e35079f4440c78bd9870724b9638b - default default] [instance: 1517a2ac-3b51-4d8d-80b3-89a5614d1ae0] Unable to find record for source node servername on servername: ComputeHostNotFound: Compute host servername could not be found. /var/log/nova/nova-conductor.log:2021-04-21 14:47:12.605 227612 WARNING nova.scheduler.utils [req-f4067a26-a233-4673-8c07-9a8a290980b0 dce35e6eceea4312bb0baa0510cef363 ca7e35079f4440c78bd9870724b9638b - default default] Failed to compute_task_migrate_server: Compute host servername could not be found.: ComputeHostNotFound: Compute host servername could not be found. /var/log/nova/nova-conductor.log:2021-04-21 14:47:12.605 227612 WARNING nova.scheduler.utils [req-f4067a26-a233-4673-8c07-9a8a290980b0 dce35e6eceea4312bb0baa0510cef363 ca7e35079f4440c78bd9870724b9638b - default default] [instance: 1517a2ac-3b51-4d8d-80b3-89a5614d1ae0] Setting instance to ACTIVE state.: ComputeHostNotFound: Compute host servername could not be found. /var/log/nova/nova-conductor.log:2021-04-21 14:47:12.672 227612 ERROR oslo_messaging.rpc.server [req-f4067a26-a233-4673-8c07-9a8a290980b0 dce35e6eceea4312bb0baa0510cef363 ca7e35079f4440c78bd9870724b9638b - default default] Exception during message handling: ComputeHostNotFound: Compute host am-osfecn-4025 Tried with this command: nova live-migration --block-migrate id. Any idea? Thank you. ________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.

5 16

[tc][all][monasca ] Retiring the Monasca project
by Ghanshyam Mann 03 Jul '24

03 Jul '24

Hello everyone, As you might know, the Monasca project is not active and was marked 'inactive' since 2024.1 cycle[1]. In this cycle also, this project is leaderless[2] and no maintainer stepped up to maintain it. As discussed in the Leaderless project etherpad[2], the next step is to retire the project. I'm sending this email to propose the Monasca retirement and based on the response, I will proceed next. [1] https://governance.openstack.org/tc/reference/emerging-technology-and-inact… [2] https://etherpad.opendev.org/p/2024.2-leaderless#L72 -gmann

6 14

[Tempest] [Blazar] Connection Refused Error 111 on Blazar tempest plugin
by Tran, Kenny 28 May '24

28 May '24

First time asking a question on the mailing list, I am trying to set up tempest against a quick start yoga (kolla-ansible) deployment with Keystone, Zun, Heat, Blazar, and Nova (1 controller, 1 compute). Default tempest tests are behaving as expected but when I try to run the blazar plugin tests they fail with Errno 111 Connection Refused. I ran a tcpdump on the tempest server side and saw no issues. Tcpdump on controller showed a reset when trying to connect to keystone. Is this an issue with blazar not being able to authenticate against the openstack deployment? The default tempest tests succeed so I don't think my admin_password admin_username etc. are incorrect. Kenny Tran Iowa State University Student Computer Engineer 515-441-5237 | trantk(a)iastate.edu

2 1

[trove][tc] Next step on leaderless project Trove
by Ghanshyam Mann 23 May '24

23 May '24

Hi Wu, This is regarding the Trove project leader assignment. We saw your late candidacy for Trove PTL[1]. The TC has listed Trove as a late candidacy project[2] and can consider a PTL appointment. If you are still interested in serving as Trove PTL, please propose the PTL appointment request to TC. Example: https://review.opendev.org/c/openstack/governance/+/915108 [1] https://review.opendev.org/c/openstack/election/+/910488 [2] https://etherpad.opendev.org/p/2024.2-leaderless#L99 -gmann

1 1

[tc][all][sahara] Retiring the Sahara project
by Ghanshyam Mann 22 May '24

22 May '24

Hello everyone, As you might know, the Sahara project is not active and was marked 'inactive' since 2024.1 cycle[1]. In this cycle also, this project is leaderless[2] and no maintainer stepped up to maintain it. As discussed in the Leaderless project etherpad[2], the next step is to retire the project. I'm sending this email to propose the Sahara retirement and based on the response, I will proceed next. [1] https://governance.openstack.org/tc/reference/emerging-technology-and-inact… [2] https://etherpad.opendev.org/p/2024.2-leaderless#L72 -gmann

3 9

[ptl][tc] OpenStack packages PyPi additional external maintainers audit & cleanup
by Ghanshyam Mann 17 May '24

17 May '24

Hi PTLs, As you might know or have seen for your project package on PyPi, OpenStack deliverables on PyPi have additional maintainers, For example, https://pypi.org/project/murano/, https://pypi.org/project/glance/ We should keep only 'openstackci' as a maintainer in PyPi so that releases of OpenStack deliverables can be managed in a single place. Otherwise, we might face the two sets of maintainers' places and packages might get released in PyPi by additional maintainers without the OpenStack project team knowing about it. One such case is in Horizon repo 'xstatic-font-awesome' where a new maintainer is added by an existing additional maintainer and this package was released without the Horizon team knowing about the changes and release. - https://github.com/openstack/xstatic-font-awesome/pull/2 To avoid the 'xstatic-font-awesome' case for other packages, TC discussed it in their weekly meetings[1] and agreed to audit all the OpenStack packages and then clean up the additional maintainers in PyPi (keep only 'openstackci' as maintainers). To help in this task, TC requests project PTL to perform the audit for their project's repo and add comments in the below etherpad. - https://etherpad.opendev.org/p/openstack-pypi-maintainers-cleanup Thanks to knikolla to automate the listing of the OpenStack packages with additional maintainers in PyPi which you can find the result in output.txt at the bottom of this link. I have added the project list of who needs to check their repo in etherpad. - https://gist.github.com/knikolla/7303a65a5ddaa2be553fc6e54619a7a1 Please complete the audit for your project before March 15 so that TC can discuss the next step in vPTG. [1] https://meetings.opendev.org/meetings/tc/2023/tc.2023-01-11-16.00.log.html#… -gmann

18 27

Logscraper host and OpenSearch service upgrade
by Daniel Pawlik 15 May '24

15 May '24

Hello, We would like to notify you that the OpenSearch service [1] would be updated to a newer version on 2024-05-15 at 01:00 PM UTC. This procedure might take a while, depending of the cluster size. During that time, the OpenSearch service would not be available. If anyone has any doubts, please reply to the email. Dan [1] - https://opensearch.logs.openstack.org/_dashboards/app/home

1 1