August 2024 - openstack-discuss - lists.openstack.org

Deployment Caracal with OVN not working (same test setup with Bobcat runs perfectly)
by Maurer Remo (murm) 07 Aug '24

07 Aug '24

Dear all I'm running multiple Openstack production clusters for years. Currently, our clusters hold approx 600 instances mainly for educational purposes. We're deploying the environment using Kolla-Ansible. Host OS Ubuntu Jammy, using the Jammy-Kolla containers. At the moment we're running Bobcat and are testing the Upgrade to Caracal (2024.1). To make the problem reproduceable I set up a one node testenvironment. Host OS: Ubuntu Jammy running on KVM. Kolla-Ansible Version: 18.1.1.dev1 Ansible-Version: 9.8.0 (ansible-core 2.16.9) Kolla config: * passwords.yml from caracal template, values filled in with kolla-genpwd * multinode from caracal template. Control, network and compute node set to the hostname of the testserver. Everything else left as is. * globals.yml from caracal template. Changes (I'm not listing the network config entries): * kolla_base_distro: "ubuntu" * neutron_plugin_agent: "ovn" Deployment runs smoothly. Post deploy tests: * "openstack compute service list", shows all services up (nova-scheduler, nova-conductor, nova-compute) * "openstack network agent list", shows all network agents up (OVN Controller Gateway agent, OVN Metadata agent) To setup network, flavors, etc, I'm running an adapted init_runonce script, creating an external (flat) and an internal (geneve) network and a router. Spinning up an instance leads to a fatal error on nova: --- Dashboard --- Error: Failed to perform requested operation on instance "test", the instance has an error status: Please try again later [Error: Build of instance 38488604-3d83-4676-b0c2-a4218696eba3 aborted: Failed to allocate the network(s), not rescheduling.]. --- --- /var/log/kolla/nova/nova-compute.log --- ... 2024-08-07 15:33:02.742 7 ERROR nova.compute.manager [None req-14f0d15f-5892-406b-a750-698ec54d9b70 ec93e00940164f30b5076a93c0a24722 499b452a194d4cbeb401302eaef1304a - - default default] [instance: 38488604-3d83-4676-b0c2-a4218696eba3] Instance failed to spawn: nova.exception.VirtualInterfaceCreateException: Virtual Interface creation failed 2024-08-07 15:33:02.742 7 ERROR nova.compute.manager [instance: 38488604-3d83-4676-b0c2-a4218696eba3] Traceback (most recent call last): 2024-08-07 15:33:02.742 7 ERROR nova.compute.manager [instance: 38488604-3d83-4676-b0c2-a4218696eba3] File "/var/lib/kolla/venv/lib/python3.10/site-packages/nova/virt/libvirt/driver.py", line 8002, in _create_guest_with_network 2024-08-07 15:33:02.742 7 ERROR nova.compute.manager [instance: 38488604-3d83-4676-b0c2-a4218696eba3] with self.virtapi.wait_for_instance_event( 2024-08-07 15:33:02.742 7 ERROR nova.compute.manager [instance: 38488604-3d83-4676-b0c2-a4218696eba3] File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__ 2024-08-07 15:33:02.742 7 ERROR nova.compute.manager [instance: 38488604-3d83-4676-b0c2-a4218696eba3] next(self.gen) 2024-08-07 15:33:02.742 7 ERROR nova.compute.manager [instance: 38488604-3d83-4676-b0c2-a4218696eba3] File "/var/lib/kolla/venv/lib/python3.10/site-packages/nova/compute/manager.py", line 559, in wait_for_instance_event 2024-08-07 15:33:02.742 7 ERROR nova.compute.manager [instance: 38488604-3d83-4676-b0c2-a4218696eba3] self._wait_for_instance_events( 2024-08-07 15:33:02.742 7 ERROR nova.compute.manager [instance: 38488604-3d83-4676-b0c2-a4218696eba3] File "/var/lib/kolla/venv/lib/python3.10/site-packages/nova/compute/manager.py", line 471, in _wait_for_instance_events 2024-08-07 15:33:02.742 7 ERROR nova.compute.manager [instance: 38488604-3d83-4676-b0c2-a4218696eba3] actual_event = event.wait() 2024-08-07 15:33:02.742 7 ERROR nova.compute.manager [instance: 38488604-3d83-4676-b0c2-a4218696eba3] File "/var/lib/kolla/venv/lib/python3.10/site-packages/nova/compute/manager.py", line 436, in wait 2024-08-07 15:33:02.742 7 ERROR nova.compute.manager [instance: 38488604-3d83-4676-b0c2-a4218696eba3] instance_event = self.event.wait() 2024-08-07 15:33:02.742 7 ERROR nova.compute.manager [instance: 38488604-3d83-4676-b0c2-a4218696eba3] File "/var/lib/kolla/venv/lib/python3.10/site-packages/eventlet/event.py", line 124, in wait 2024-08-07 15:33:02.742 7 ERROR nova.compute.manager [instance: 38488604-3d83-4676-b0c2-a4218696eba3] result = hub.switch() 2024-08-07 15:33:02.742 7 ERROR nova.compute.manager [instance: 38488604-3d83-4676-b0c2-a4218696eba3] File "/var/lib/kolla/venv/lib/python3.10/site-packages/eventlet/hubs/hub.py", line 310, in switch 2024-08-07 15:33:02.742 7 ERROR nova.compute.manager [instance: 38488604-3d83-4676-b0c2-a4218696eba3] return self.greenlet.switch() 2024-08-07 15:33:02.742 7 ERROR nova.compute.manager [instance: 38488604-3d83-4676-b0c2-a4218696eba3] eventlet.timeout.Timeout: 300 seconds 2024-08-07 15:33:02.742 7 ERROR nova.compute.manager [instance: 38488604-3d83-4676-b0c2-a4218696eba3] 2024-08-07 15:33:02.742 7 ERROR nova.compute.manager [instance: 38488604-3d83-4676-b0c2-a4218696eba3] During handling of the above exception, another exception occurred: 2024-08-07 15:33:02.742 7 ERROR nova.compute.manager [instance: 38488604-3d83-4676-b0c2-a4218696eba3] 2024-08-07 15:33:02.742 7 ERROR nova.compute.manager [instance: 38488604-3d83-4676-b0c2-a4218696eba3] Traceback (most recent call last): 2024-08-07 15:33:02.742 7 ERROR nova.compute.manager [instance: 38488604-3d83-4676-b0c2-a4218696eba3] File "/var/lib/kolla/venv/lib/python3.10/site-packages/nova/compute/manager.py", line 2885, in _build_resources 2024-08-07 15:33:02.742 7 ERROR nova.compute.manager [instance: 38488604-3d83-4676-b0c2-a4218696eba3] yield resources 2024-08-07 15:33:02.742 7 ERROR nova.compute.manager [instance: 38488604-3d83-4676-b0c2-a4218696eba3] File "/var/lib/kolla/venv/lib/python3.10/site-packages/nova/compute/manager.py", line 2632, in _build_and_run_instance 2024-08-07 15:33:02.742 7 ERROR nova.compute.manager [instance: 38488604-3d83-4676-b0c2-a4218696eba3] self.driver.spawn(context, instance, image_meta, 2024-08-07 15:33:02.742 7 ERROR nova.compute.manager [instance: 38488604-3d83-4676-b0c2-a4218696eba3] File "/var/lib/kolla/venv/lib/python3.10/site-packages/nova/virt/libvirt/driver.py", line 4642, in spawn 2024-08-07 15:33:02.742 7 ERROR nova.compute.manager [instance: 38488604-3d83-4676-b0c2-a4218696eba3] self._create_guest_with_network( 2024-08-07 15:33:02.742 7 ERROR nova.compute.manager [instance: 38488604-3d83-4676-b0c2-a4218696eba3] File "/var/lib/kolla/venv/lib/python3.10/site-packages/nova/virt/libvirt/driver.py", line 8028, in _create_guest_with_network 2024-08-07 15:33:02.742 7 ERROR nova.compute.manager [instance: 38488604-3d83-4676-b0c2-a4218696eba3] raise exception.VirtualInterfaceCreateException() 2024-08-07 15:33:02.742 7 ERROR nova.compute.manager [instance: 38488604-3d83-4676-b0c2-a4218696eba3] nova.exception.VirtualInterfaceCreateException: Virtual Interface creation failed 2024-08-07 15:33:02.742 7 ERROR nova.compute.manager [instance: 38488604-3d83-4676-b0c2-a4218696eba3] 2024-08-07 15:33:02.753 7 INFO nova.compute.manager [None req-14f0d15f-5892-406b-a750-698ec54d9b70 ec93e00940164f30b5076a93c0a24722 499b452a194d4cbeb401302eaef1304a - - default default] [instance: 38488604-3d83-4676-b0c2-a4218696eba3] Terminating instance ... --- The same configuration works without error under Bobcat (kolla-ansible: 17.4.0, ansible: 8.4.0) When I upgrade a running Bobcat configuration to Caracal, I run into the same issue. When running the upgrade with "--skip-tags openvswitch" the upgraded environment runs perfectly (with the bobcat openvswitch containers). As soon as I upgrade those, the deployment is broken again. I clouldn't find a bug on launchpad which points to this issue. I coudn't find any info in the release notes of caracal which solves my problem. Does anyone struggling on the same issue as well? Do you have any hint? I'm appreciating any comment, help, ... Thanks Remo Maurer

2 1

[oslo][freezer][watcher][requirement][release] Resetting the DPL model liaisons
by Ghanshyam Mann 07 Aug '24

07 Aug '24

Hi, DPL projects, To keep the DPL model/liaisons up to date, TC defined the new policy to reset the DPL state/liaisons every cycle[1]. If the project wants to continue the DPL model, they can opt-in the same; otherwise, they fall into the PTL model/election process. I am resetting the DPL model for all the existing 5 DPL projects[2]. If needed, feel free to opt-in the DPL model for the next cycle. [1] https://governance.openstack.org/tc/reference/distributed-project-leadershi… [2] this series - https://review.opendev.org/c/openstack/governance/+/925834/1 -gmann

1 0

Firewall as a Service - kolla ansible 2023.1
by Thamanna Farhath 07 Aug '24

07 Aug '24

Hello OpenStack Community, I hope this message finds you well. My name is Thamanna , and I am currently working with the OpenStack Kolla-Ansible deployment version 2023.1. I am interested in deploying Firewall as a Service (FWaaS) within our OpenStack environment. However, I have noticed that FWaaS is not available in the Kolla-Ansible 2023.1 version. Could anyone please provide guidance on the following points? 1. Is there an official or recommended way to integrate FWaaS with Kolla-Ansible 2023.1? 2. Are there any specific configurations or additional components required to enable FWaaS? 3. Has anyone in the community successfully deployed FWaaS with this version, and if so, could you share your experience and any potential challenges faced? 4. Are there alternative approaches or tools that you would recommend for deploying FWaaS in an OpenStack environment? Any help or pointers to relevant documentation would be greatly appreciated. Thank you in advance for your assistance! Best regards, Thamanna mailto:thamanna.f@zybisys.com Disclaimer : The content of this email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to which they are addressed. If you have received this email in error, please notify the sender and remove the messages from your system. If you are not the named addressee, it is strictly forbidden for you to share, circulate, distribute or copy any part of this e-mail to any third party without the written consent of the sender. E-mail transmission cannot be guaranteed to be secured or error free as information could be intercepted, corrupted, lost, destroyed, arrive late, incomplete, or may contain viruses. Therefore, we do not accept liability for any errors or omissions in the contents of this message, which arise as a result of e-mail transmission. The recipient should check this e-mail and any attachments for the presence of viruses. The company accepts no liability for any damage caused by any virus transmitted by this email."

3 5

Live migration fails
by Szabo, Istvan (Agoda) 07 Aug '24

07 Aug '24

Hi, I have couple of compute nodes where the live migration fails with existing vms. When I quickly spawn a vm and try live migration it works so I assume shouldn't be a big problem with the compute node. However I have many existing vms where it fails with a servername not found. /var/log/nova/nova-conductor.log:2021-04-21 14:47:12.605 227612 ERROR nova.conductor.tasks.migrate [req-f4067a26-a233-4673-8c07-9a8a290980b0 dce35e6eceea4312bb0baa0510cef363 ca7e35079f4440c78bd9870724b9638b - default default] [instance: 1517a2ac-3b51-4d8d-80b3-89a5614d1ae0] Unable to find record for source node servername on servername: ComputeHostNotFound: Compute host servername could not be found. /var/log/nova/nova-conductor.log:2021-04-21 14:47:12.605 227612 WARNING nova.scheduler.utils [req-f4067a26-a233-4673-8c07-9a8a290980b0 dce35e6eceea4312bb0baa0510cef363 ca7e35079f4440c78bd9870724b9638b - default default] Failed to compute_task_migrate_server: Compute host servername could not be found.: ComputeHostNotFound: Compute host servername could not be found. /var/log/nova/nova-conductor.log:2021-04-21 14:47:12.605 227612 WARNING nova.scheduler.utils [req-f4067a26-a233-4673-8c07-9a8a290980b0 dce35e6eceea4312bb0baa0510cef363 ca7e35079f4440c78bd9870724b9638b - default default] [instance: 1517a2ac-3b51-4d8d-80b3-89a5614d1ae0] Setting instance to ACTIVE state.: ComputeHostNotFound: Compute host servername could not be found. /var/log/nova/nova-conductor.log:2021-04-21 14:47:12.672 227612 ERROR oslo_messaging.rpc.server [req-f4067a26-a233-4673-8c07-9a8a290980b0 dce35e6eceea4312bb0baa0510cef363 ca7e35079f4440c78bd9870724b9638b - default default] Exception during message handling: ComputeHostNotFound: Compute host am-osfecn-4025 Tried with this command: nova live-migration --block-migrate id. Any idea? Thank you. ________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.

5 16

nova db queries
by engineer2024 06 Aug '24

06 Aug '24

Hi, I am trying to execute a query on nova db in a custom code file in nova. During the execution of a function like 'instance_get_all_by_filters_sort' from nova/db/main/api.py , I see this --- try: query_prefix = sqlalchemyutils.paginate_query( query_prefix, models.Instance, limit, sort_keys, marker=marker, sort_dirs=sort_dirs, ) except db_exc.InvalidSortKey: raise exception.InvalidSortKey() instances = query_prefix.all() --- So, my question is how does query_prefix.all() know the mysql db connection url to connect to for executing the query ? Thanks elinux

2 1

[ironic] HTTPS Provisioning
by Michael Koch 06 Aug '24

06 Aug '24

Good day, I hope this is the right way to ask. Is it possible to use IPXE HTTPS or UEFI HTTPS Network boot with Openstack Ironic, because most of the communication or completely should be encrypted. Or how do you have to configure this. Unfortunately, I have not yet found any instructions in the documentation. Of course, it is also necessary to import the certificates or install IPXE on the client side. Perhaps I have overlooked something and would like some clarification. Best regards Michael

2 1

[all][elections][ptl][tc] Combined PTL/TC 2025.1 cycle Election Season
by Sławek Kapłoński 06 Aug '24

06 Aug '24

Hello everyone, Election details: https://governance.openstack.org/election/ The nomination period officially begins Aug 14, 2024 23:45 UTC. Please read the stipulations and timelines for candidates and electorate contained in these governance documents[1][2][3]. Due to circumstances of timing, PTL and TC elections for the coming cycle will run concurrently; deadlines for their nomination and voting activities are synchronized but will still use separate ballots. Please note, if only one candidate is nominated as PTL for a project team during the PTL nomination period, that candidate will win by acclaim, and there will be no poll. There will only be a poll if there is more than one candidate stepping forward for a project team's PTL position. Similarly, there will only be a TC poll if there are more than 4 candidates stepping forward the available TC positions. All members of the community are encouraged to confirm their email address as described at: https://governance.openstack.org/election/#electorate There will be further announcements posted to the mailing list as action is required from the electorate or candidates. This email is for information purposes only. If you have any questions which you feel affect others please reply to this email thread. If you have any questions that you wish to discuss in private please email any of the election officials[4] so that we may address your concerns. [1] https://governance.openstack.org/tc/reference/charter.html [2] https://governance.openstack.org/tc/reference/role-of-the-tc.html [3] https://docs.openstack.org/project-team-guide/ptl.html [4] https://governance.openstack.org/election/#election-officials Thank you, Slaweq

1 0

[tc][all] OpenStack Technical Committee Weekly Summary and Meeting Agenda (2024.2/R-8)
by Goutham Pacha Ravi 06 Aug '24

06 Aug '24

Hello Stackers, It's been a quiet-yet-busy week for the OpenStack Technical Committee as we approach the tail end of the 2024.2 (Dalmatian) release cycle [1]. We're 8 weeks away from the release date (2024-10-02) and three weeks away from the feature code freeze across many OpenStack code repositories (2024-08-29). If you are a maintainer of an OpenStack repository with any Zuul jobs, I'd encourage you to check if your repository is flagged with any configuration errors [2]. It's possible you're referencing a nodeset or a code repository that isn't there anymore (for example, CentOS-8-Stream was recently deleted [3]). Or, it's possible that you have deprecated syntax. There's a lot of prior art if you would like to fix your Zuul configuration. If you're unsure what to do, please send an email to this list or chat with a Zuul maintainer on OFTC's #opendev channel. The nomination period for PTL positions and the OpenStack TC begins next week (2024-08-14). Election officials will be sharing more context over the next week. If you are a PTL or a member of the TC, motivate individuals to take on these leadership roles and support their enthusiasm. === Weekly Meeting === The TC had a productive meeting over IRC last week [4]. We discussed phased updates to the eventlet removal proposal [5] and the health of the CI system over the past week. Several contributors helped land CVE fixes in the past couple of weeks, which required bumping up package constraints within older stable branches. The removal of CentOS-8-Stream nodesets and the Linaro arm64 region from Zuul caused some gate instability; the Infrastructure team will be glad to assist anyone who's triaging failures caused by these actions. The TC noted the need to delay the retirement of "kuryr-kubernetes" until the Tacker project team has stopped relying on it. The next meeting is on Tuesday (2024-08-06), and it will be simultaneously hosted on Zoom and IRC. Information to join this meeting and the meeting's agenda are on our Wiki page [6]. I hope you'll be able to join us there. === Governance Proposals === ==== Merged ==== Follow up on the affiliation diversity handling | https://review.opendev.org/c/openstack/governance/+/923876 Add os-test-images under glance | https://review.opendev.org/c/openstack/governance/+/925044 [follow up] Fix typos in the eventlet removal proposal | https://review.opendev.org/c/openstack/governance/+/924944 ==== Open for Review/Discussion ==== Update criteria for inactive projects to become active again | https://review.opendev.org/c/openstack/governance/+/921500 Inactive state extensions: Monasca | https://review.opendev.org/c/openstack/governance/+/923919 === Upcoming Events === 2024-08-06: OpenInfra Board Meeting: https://board.openinfra.dev/meetings/2024-08-06 2024-08-12: Call for Proposals due for the OpenInfra Days NA [8] 2024-08-14: Nominations open for 2025.1 OpenStack PTL+TC elections 2024-08-24: Open Infra Days, Vietnam: https://2024.vietopeninfra.org/ 2024-08-29: Dalmatian-3 milestone 2024-09-03: OpenInfra Summit Asia, Suwon Thank you very much for reading! On behalf of the OpenStack TC, Goutham Pacha Ravi (gouthamr) OpenStack TC Chair [1] Release Schedule: https://releases.openstack.org/dalmatian/schedule.html [2] Zuul Configuration Errors: https://zuul.opendev.org/t/openstack/config-errors [3] CentOS-8-Stream node pool removal: https://lists.opendev.org/archives/list/service-announce@lists.opendev.org/… [4] TC Meeting IRC Logs, 2024-07-30: https://meetings.opendev.org/meetings/tc/2024/tc.2024-07-30-18.00.log.html [5] Remove Eventlet From Openstack: https://review.opendev.org/c/openstack/governance/+/902585 [6] TC Meeting, 2024-08-06: https://wiki.openstack.org/wiki/Meetings/TechnicalCommittee#Next_Meeting

1 0

Assistance needed - Application Credential
by Naveen Anbarasu 05 Aug '24

05 Aug '24

Hi team, I have created an application credential for the compute service and successfully obtained a token. Access rule: "service": "compute", "method": "GET", "path": "/v2.1/servers" However, when using the token to retrieve the server list, I receive a Error 403 Forbidden error with the message: 'Policy doesn't allow os_compute_api:servers to be performed. But i have a necessary permission within the respective project (member role) How can I solve this problem? Thanks in advance

7 12

designate-sink and find_records()
by Andrew Bogott 05 Aug '24

05 Aug '24

Hello! I use a custom designate sink driver to detect when VMs have been deleted in order to clean up associated resources specific to our cloud: certs, proxies, etc. When deleting proxies I relied on the find_records() call to get a list of IPs associated with the deleted VM; for example: records = central_api.find_records(context, crit) if records: ip = records[0].data LOG.debug("Cleaning up proxy records for IP %s" % ip) self._delete_proxies_for_ip(data["project_name"], ip) Now that the find_records() rpc call has been removed I'm at a bit of a loss. I don't think I can call back into nova for the IPs because by the time sink knows about a VM deletion it has most likely already been deleted from nova. Can anyone suggest a workaround for answering the question "what IPs were associated with that VM that just got deleted?" Or, better yet, is there someplace else in my infra where I should have my special deletion handler outside of designate-sink? Thank you for entertaining this very obscure question :) -Andrew

1 0