November 2024 - openstack-discuss

[kolla] DHCP not accessible on compute nodes and VM IP Assignment Issue in Compute Nodes - Kolla Ansible 2024.1
by Jaganathan, Keerthana (SCC) 20 Nov '24

20 Nov '24

Hello Kolla Ansible community, We are experiencing difficulties with VM IP address assignment in our OpenStack deployment using Kolla Ansible 2024.1 on Rocky Linux 9.2. Here are the details of the issue: 1. Problem description: IP addresses are successfully assigned to VMs created on control nodes, but VMs created on compute-only nodes fail to receive IP addresses. This suggests that DHCP requests originating from the compute nodes are not reaching the DHCP agents on the control nodes. 2. Error messages/logs: Console log of a Cirros VM on a compute node shows: checking http://169.254.169.254/2009-04-04/instance-id failed 1/20: up 0.88. request failed ... failed 20/20: up 54.89. request failed failed to read iid from metadata. tried 20 failed to get instance-id of datasource 3. Configuration: - 7 nodes in total: 5 control/compute/network nodes, 2 compute-only nodes - Single bonded interface (bond0) with 3 VLANs (bond0.10, bond0.20, bond0.30) - network_interface: bond0.10 - api_interface: bond0.20 - tunnel_interface: same as network_interface - neutron_external_interface: bond0 - Provider network: VLAN 41 - ml2_conf.ini [ml2_type_vlan] network_vlan_ranges = physnet1:41:41 - 5 neutron DHCP agents, one on each control node 4. Troubleshooting steps taken: - Verified br-tun OpenFlow rules are dropping packets - Confirmed 5 DHCP agents are running, one on each control node - Openvswitch containers: agent and Here are the openvswitch open flow rules (dump) from the compute node (node16) for br-ex, br-in and br-tun (openvswitch-vswitchd)# ovs-ofctl dump-flows br-ex cookie=0xa9774500bf4a6912, duration=175366.856s, table=0, n_packets=3093, n_bytes=945104, priority=4,in_port="phy-br-ex",dl_vlan=1 actions=mod_vlan_vid:41,NORMAL cookie=0xa9774500bf4a6912, duration=177050.619s, table=0, n_packets=2, n_bytes=176, priority=2,in_port="phy-br-ex" actions=drop cookie=0xa9774500bf4a6912, duration=177050.621s, table=0, n_packets=8178, n_bytes=2363856, priority=0 actions=NORMAL (openvswitch-vswitchd)# ovs-ofctl dump-flows br-int cookie=0x98223299c10a59d3, duration=177059.641s, table=0, n_packets=0, n_bytes=0, priority=65535,dl_vlan=4095 actions=drop cookie=0x98223299c10a59d3, duration=175373.945s, table=0, n_packets=123, n_bytes=9594, priority=10,icmp6,in_port="qvoab4fc3cb-a2",icmp_type=136 actions=resubmit(,24) cookie=0x98223299c10a59d3, duration=175373.944s, table=0, n_packets=0, n_bytes=0, priority=10,arp,in_port="qvoab4fc3cb-a2" actions=resubmit(,24) cookie=0x98223299c10a59d3, duration=177059.625s, table=0, n_packets=20, n_bytes=1668, priority=2,in_port="int-br-ex" actions=drop cookie=0x98223299c10a59d3, duration=175373.946s, table=0, n_packets=3028, n_bytes=939610, priority=9,in_port="qvoab4fc3cb-a2" actions=resubmit(,25) cookie=0x98223299c10a59d3, duration=175375.861s, table=0, n_packets=2256, n_bytes=219762, priority=3,in_port="int-br-ex",dl_vlan=41 actions=mod_vlan_vid:1,resubmit(,58) cookie=0x98223299c10a59d3, duration=177059.643s, table=0, n_packets=2, n_bytes=176, priority=0 actions=resubmit(,58) cookie=0x98223299c10a59d3, duration=177059.644s, table=23, n_packets=0, n_bytes=0, priority=0 actions=drop cookie=0x98223299c10a59d3, duration=175373.946s, table=24, n_packets=123, n_bytes=9594, priority=2,icmp6,in_port="qvoab4fc3cb-a2",icmp_type=136,nd_target=fe80::f816:3eff:fe4f:8020 actions=resubmit(,58) cookie=0x98223299c10a59d3, duration=175373.945s, table=24, n_packets=0, n_bytes=0, priority=2,arp,in_port="qvoab4fc3cb-a2",arp_spa=141.52.56.110 actions=resubmit(,25) cookie=0x98223299c10a59d3, duration=177059.641s, table=24, n_packets=0, n_bytes=0, priority=0 actions=drop cookie=0x98223299c10a59d3, duration=175373.947s, table=25, n_packets=2970, n_bytes=935510, priority=2,in_port="qvoab4fc3cb-a2",dl_src=fa:16:3e:4f:80:20 actions=resubmit(,30) cookie=0x98223299c10a59d3, duration=177059.640s, table=30, n_packets=2970, n_bytes=935510, priority=0 actions=resubmit(,58) cookie=0x98223299c10a59d3, duration=177059.639s, table=31, n_packets=0, n_bytes=0, priority=0 actions=resubmit(,58) cookie=0x98223299c10a59d3, duration=177059.642s, table=58, n_packets=5351, n_bytes=1165042, priority=0 actions=resubmit(,60) cookie=0x98223299c10a59d3, duration=177059.642s, table=60, n_packets=5351, n_bytes=1165042, priority=1 actions=NORMAL cookie=0x98223299c10a59d3, duration=177059.640s, table=62, n_packets=0, n_bytes=0, priority=3 actions=NORMAL (openvswitch-vswitchd)[root@os16 /]# ovs-ofctl dump-flows br-tun cookie=0xad1cb3762184465d, duration=178072.704s, table=0, n_packets=4815, n_bytes=1114796, priority=1,in_port="patch-int" actions=resubmit(,2) cookie=0xad1cb3762184465d, duration=178072.703s, table=0, n_packets=0, n_bytes=0, priority=0 actions=drop cookie=0xad1cb3762184465d, duration=178072.703s, table=2, n_packets=1164, n_bytes=69852, priority=1,arp,dl_dst=ff:ff:ff:ff:ff:ff actions=resubmit(,21) cookie=0xad1cb3762184465d, duration=178072.702s, table=2, n_packets=7, n_bytes=528, priority=0,dl_dst=00:00:00:00:00:00/01:00:00:00:00:00 actions=resubmit(,20) cookie=0xad1cb3762184465d, duration=178072.702s, table=2, n_packets=3644, n_bytes=1044416, priority=0,dl_dst=01:00:00:00:00:00/01:00:00:00:00:00 actions=resubmit(,22) cookie=0xad1cb3762184465d, duration=178072.701s, table=3, n_packets=0, n_bytes=0, priority=0 actions=drop cookie=0xad1cb3762184465d, duration=178072.701s, table=4, n_packets=0, n_bytes=0, priority=0 actions=drop cookie=0xad1cb3762184465d, duration=178072.701s, table=6, n_packets=0, n_bytes=0, priority=0 actions=drop cookie=0xad1cb3762184465d, duration=178072.700s, table=10, n_packets=0, n_bytes=0, priority=1 actions=learn(table=20,hard_timeout=300,priority=1,cookie=0xad1cb3762184465d,NXM_OF_VLAN_TCI[0..11],NXM_OF_ETH_DST[]=NXM_OF_ETH_SRC[],load:0->NXM_OF_VLAN_TCI[],load:NXM_NX_TUN_ID[]->NXM_NX_TUN_ID[],output:OXM_OF_IN_PORT[]),output:"patch-int" cookie=0xad1cb3762184465d, duration=178072.699s, table=20, n_packets=7, n_bytes=528, priority=0 actions=resubmit(,22) cookie=0xad1cb3762184465d, duration=178072.699s, table=21, n_packets=1164, n_bytes=69852, priority=0 actions=resubmit(,22) cookie=0xad1cb3762184465d, duration=178072.699s, table=22, n_packets=4815, n_bytes=1114796, priority=0 actions=drop I would greatly appreciate any guidance on resolving this. Best regards, le3188

1 0

[kolla] IRC meeting cancelled - 20.11.2024
by Bartosz Bezak 20 Nov '24

20 Nov '24

Hello, Unfortunately, this week’s Kolla IRC meeting is cancelled. There are multiple conferences happening this week. Apologies for the late notice. Best regards, Bartosz Bezak

1 0

[openstack-ansible][horizon]The token used to make the request was domain scoped but the policy requires ['system', 'project'] scope
by Danish Khan 20 Nov '24

20 Nov '24

Dear All, I have upgraded one of my test openstack cluster from victoria to Wallaby. After upgrading the hypervisor is not showing correct numbers of VMs on compute nodes: [image: image.png] But openstack CLI is giving correct number of VMs. I am getting below error in horizon container when I click on "hypervisor" from browser: Nov 20 11:17:29 horizon-container-2b244cbd apache2[43317]: [wsgi:error] [pid 43317:tid 139897181898496] [remote x.x.x.151:37180] /openstack/venvs/horizon-23.3.1.dev4/lib/python3.8/site-packages/oslo_policy/policy.py:1065: UserWarning: Policy os_compute_api:os-migrate-server:migrate failed scope check. The token used to make the request was domain scoped but the policy requires ['system', 'project'] scope. This behavior may change in the future where using the intended scope is required I did try few thing but that did not work. Regards, Danish Khan

1 0

[Cinder] Restore volume snaptshots while volume is in use
by Jan 19 Nov '24

19 Nov '24

Hi, for providing a more enterprise like (VMware, HyperV, Proxmox) snapshot feature, it would be great to be able to revert snapshots in use. With the "revert-to-snapshot" functions everything is there to support this feature. IMO if the storage backend (driver) does not support the operation, it should raise an error (ex. resetting a ceph rbd volume while a vm is running is not possible). But if it supports the operation, we should not stop the user doing this, even if the volume is "in-use". One might introduce a force switch, to prevent triggering it unintended. A frontend component like skyline can also show additional information like the status of an attached instance. Since the "Detach and attach boot volumes" spec [1] was never fully implemented, there is no way to revert an instance to a snapshot state without completely rebuilding it. Like described in the spec draft, this is not always suitable and at least always cumbersome. -- Jan [1] https://specs.openstack.org/openstack/nova-specs/specs/stein/approved/detac… --- suggested patch ----- https://github.com/jmussmann/cinder/commit/4f971d77a94806e9ebdccab45392058c… From 4f971d77a94806e9ebdccab45392058c25005761 Mon Sep 17 00:00:00 2001 From: jmussmann <jmussmann+github(a)posteo.de> Date: Thu, 31 Oct 2024 09:13:45 +0100 Subject: [PATCH] add: allow reverting snapshots of volumes in use --- cinder/volume/api.py | 5 +++-- cinder/volume/manager.py | 4 ++-- cinder/volume/rpcapi.py | 4 ++-- 3 files changed, 7 insertions(+), 6 deletions(-) diff --git a/cinder/volume/api.py b/cinder/volume/api.py index 0e20baff0..9aa160740 100644 --- a/cinder/volume/api.py +++ b/cinder/volume/api.py @@ -390,8 +390,9 @@ class API(base.Base): """revert a volume to a snapshot""" context.authorize(vol_action_policy.REVERT_POLICY, target_obj=volume) + volume_state = volume.status v_res = volume.update_single_status_where( - 'reverting', 'available') + 'reverting', ['available', 'in-use']) if not v_res: msg = (_("Can't revert volume %(vol_id)s to its latest snapshot " "%(snap_id)s. Volume's status must be 'available'.") @@ -408,7 +409,7 @@ class API(base.Base): "snap_id": snapshot.id}) raise exception.InvalidSnapshot(reason=msg) - self.volume_rpcapi.revert_to_snapshot(context, volume, snapshot) + self.volume_rpcapi.revert_to_snapshot(context, volume, snapshot, volume_state) def delete(self, context: context.RequestContext, diff --git a/cinder/volume/manager.py b/cinder/volume/manager.py index 30de5d8c7..18914c073 100644 --- a/cinder/volume/manager.py +++ b/cinder/volume/manager.py @@ -1150,7 +1150,7 @@ class VolumeManager(manager.CleanableManager, self.create_snapshot(context, snapshot) return snapshot - def revert_to_snapshot(self, context, volume, snapshot) -> None: + def revert_to_snapshot(self, context, volume, snapshot, original_volume_state) -> None: """Revert a volume to a snapshot. The process of reverting to snapshot consists of several steps: @@ -1207,7 +1207,7 @@ class VolumeManager(manager.CleanableManager, LOG.exception(msg, msg_args) v_res = volume.update_single_status_where( - 'available', 'reverting') + original_volume_state, 'reverting') if not v_res: msg_args = {"id": volume.id, "status": 'available'} diff --git a/cinder/volume/rpcapi.py b/cinder/volume/rpcapi.py index 58de69c1c..0b36cebdd 100644 --- a/cinder/volume/rpcapi.py +++ b/cinder/volume/rpcapi.py @@ -180,11 +180,11 @@ class VolumeAPI(rpc.RPCAPI): volume=volume) @rpc.assert_min_rpc_version('3.15') - def revert_to_snapshot(self, ctxt, volume, snapshot): + def revert_to_snapshot(self, ctxt, volume, snapshot, volume_state): version = self._compat_ver('3.15') cctxt = self._get_cctxt(volume.service_topic_queue, version) cctxt.cast(ctxt, 'revert_to_snapshot', volume=volume, - snapshot=snapshot) + snapshot=snapshot, original_volume_state=volume_state) def delete_volume(self, ctxt: context.RequestContext, -- 2.47.0

3 2

#eventlet-removal - HTTP SGI working group call for discussion
by Herve Beraud 19 Nov '24

19 Nov '24

tl;dr: We should have a cross discussion concerning the replacement of Eventlet.wsgi and we want to invite you to join the discussion and hence help us to define a standardized approach in a collaborative way. The Eventlet WSGI module can be used for 2 things: 1. as a WSGI server to serve any kind of web framework (flask, routes, etc...) [1] 2. and in some cases as a framework for web applications [2] Some OpenStack deliverables are using it in one way or another. Some deliverables like Glance already have credible migrations opportunities. But some of you may still be wondering what is your option to replace the Eventlet WSGI features? That's a good question. Now that the specs defining the removal of Eventlet from oslo.service are merged we have to define the way we want to proceed for the WSGI features. Indeed, we decided to drop the WSGI support from oslo.service. That means that we have to define, at least, a "default" way to replace it. If we want to avoid having multiple ways to manage things, we have to provide some guidance and standardizations. For this reason, after the PTG, we encouraged the creation of the HTTP SGI Working Group [4]. This working group aim to answer which (W|A)SGI servers are needed or credible alternatives. Which framework we have/want/need to use in parallel with the SGI server. The outcomes created by this group will serve as "a default" to guide the community through the replacement of the WSGI features of Eventlet. We prepared comparisons of the available alternatives for each layer: - https://wiki.openstack.org/wiki/SGI-servers-comparison - https://wiki.openstack.org/wiki/web-frameworks-comparison - https://wiki.openstack.org/wiki/Http-libraries-comparison Feel free to comment and to put your pros and cons on the given related etherpads (see the links in the comparison). This topic will be discussed at each irc meeting on the #openstack-eventlet-removal channel [5], until we find a default official option. Next meeting date is: 2024 December 3rd [6]. It would be good to reach an official consensus before the Milestone 3 of Epoxy. The goal is simply to ensure that our community looks in the same direction and that everyone is on the same page. We want to ensure that nobody is left behind. Feel free to reply to this thread or to put your comments directly on the various etherpads [6][7][8][9]. Thanks for reading. [1] https://eventlet.readthedocs.io/en/latest/reference/api/eventlet.html#modul… [2] https://eventlet.readthedocs.io/en/latest/modules/wsgi.html [3] https://review.opendev.org/c/openstack/oslo-specs/+/927503 [4] https://wiki.openstack.org/wiki/Eventlet-removal#The_HTTP_SGI_Working_Group [5] https://meetings.opendev.org/meetings/openstack_eventlet_removal/ [6] https://etherpad.opendev.org/p/epoxy-eventlet-tracking [7] https://etherpad.opendev.org/p/eventlet-removal-SGI-servers-comparison [8] https://etherpad.opendev.org/p/eventlet-removal-web-frameworks-comparison [9] https://etherpad.opendev.org/p/eventlet-removal-http-libraries-comparison -- Hervé Beraud Senior Software Engineer at Red Hat irc: hberaud https://github.com/4383/

1 0

Kolla TLS
by devispain＠mail.com 19 Nov '24

19 Nov '24

Reading the Kolla docs it says: The back-end TLS cert/key can be the same certificate that is used for the VIP, as long as those certificates are configured to allow requests from both the VIP and internal networks. But there is no advice on how to do this, can someone explain how to use letsencrypt vip certs for backend tls, i have found nowhere that explains it at all. Trying to use a self signed cert for backend tls seems to break everything when using proper certs for vips

1 0

python3-dns 2.6.1 breaks Nova
by Francesco Di Nucci 19 Nov '24

19 Nov '24

Hi all, maybe this is already known (https://bugs.launchpad.net/glance/+bug/2054732) but also python3-dns-2.6.1 seems to break Nova and other services, with the error reported in the issue, that the DNS "answered udp() got an unexpected keyword argument 'ignore_errors';" It seems that Almalinux has the new version in the repos, while Rocky Linux still has 2.3.0 Best regards Francesco Di Nucci

2 2

Question about Package Inclusion and User Creation in Kolla Base Image
by Amir Hossein Ahmadi 19 Nov '24

19 Nov '24

Hello Stackers, I'm currently reading through the OpenStack Kolla source code, and I noticed a few design choices in the kolla/base image that I’d like to understand better: - The kolla/base The image includes several packages (such as Apache) that don’t appear necessary for most containers. - Additionally, user and group accounts for all services are created in the base image, even though all containers don’t use them. Could someone explain the reasoning behind including these components in the base image? Is there any documentation available that provides insight into the design approach for Kolla images? Thank you in advance for your help and guidance. Best regards, Amir Hossein, Burna innovation Bridge

1 0

Oslo Spec Review, Eventlet Removal Proposal.
by Daniel Bengtsson 19 Nov '24

19 Nov '24

Hi everyone, I hope you’re all doing well. During yesterday’s Oslo IRC meeting, the main topic was the removal of eventlet in oslo.service[1]. It seems that everyone is in agreement with this proposal, and no objections were raised. I would appreciate it if you could take another look and review the code. If we don’t receive any further objections by Monday, I propose we proceed with merging the spec. Thanks in advance for your help and feedback! [1] https://review.opendev.org/c/openstack/oslo-specs/+/927503?usp=dashboard -- Daniel Bengtsson Software engineer

2 1

[publiccloud-sig] Reminder: next meeting November 20th (tomorrow) - 0800 UTC
by Felix Kronlage-Dammers 19 Nov '24

19 Nov '24

Hi everyone, This Wednesday, tomorrow, (November 20th) the next meeting of the publiccloud-sig is going to happen. We’ll be meeting at 0800 UTC here: https://meetpad.opendev.org/publiccloud-sig <https://meetpad.opendev.org/publiccloud-sig> See also here for all other details: https://wiki.openstack.org/wiki/PublicCloudSIG <https://wiki.openstack.org/wiki/PublicCloudSIG> As I’ll be traveling, I’ll not be able to participate myself. felix -- Felix Kronlage-Dammers Sovereign Cloud Stack — standardized, built and operated by many Ein Projekt der Open Source Business Alliance - Bundesverband für digitale Souveränität e.V. Tel.: +49-30-206539-205 | Matrix: @fkronlage:matrix.org | fkr(a)osb-alliance.com

1 0

[Devstack] Trying to setup development environment on VM
by Ishan Shanware (ishanwar) 19 Nov '24

19 Nov '24

Hi Everyone, I am currently trying to setup in all-in-one development environment for OpenStack on a Debian VM. The VM has 16 GB RAM and 4 CPUs. However, I face an error while trying to run FORCE=yes ./stack.sh. I get an error that keystone does not start. This causes the entire process to fail. Requesting assistance in solving this issue. I am following the this guide: https://docs.openstack.org/devstack/latest/guides/single-machine.html PFA the error log and error message on the terminal. [cid:image001.png@01DB39FF.4A36CA30] Thanks in advance, Ishan

4 3

[octavia] [lbaas] CPU pinning & multiqueue
by yardalgedal＠gmail.com 19 Nov '24

19 Nov '24

Hi, openstack team, is someone tested CPU pinning together with enabled multiqueue for VM? I'm seeing huge performance degradation (750k RPS => 200k RPS) on flavor with 16 vCPUs and 16 gb ram when using image built with CPU pinning vs image without it, we're using zed (I've backported commits to check, but I think newer versions can also be affected), haproxy 2.4, ubuntu 22.04.

4 4

[Glance] Status of image clone feature
by Masahito Muroi 19 Nov '24

19 Nov '24

Hello Glance folks, I'm wondering the development status of the image clone feature[1]. We provide multi region OpenStack cluster and Glance API is deployed in each region individually. We as a cloud operator want to clone same glance image attributes and images across all the regions using the cloning feature. I have checked the image interoperable feature[2] is available, but the feature looks for cloud users, not for cloud operators. If the development is suspended by some reason, we're happy to start talking the cloning feature. 1. https://wiki.openstack.org/wiki/Glance-image-cloning 2. https://docs.openstack.org/glance/latest/admin/interoperable-image-import.h… thank you, Masahito

2 2

[tc][all] OpenStack Technical Committee Weekly Summary and Meeting Agenda (2025.1/R-19)
by Goutham Pacha Ravi 19 Nov '24

19 Nov '24

Hello Stackers, We're now past milestone-1 of the 2025.1 ("Epoxy") release cycle [1]. As a contributor/maintainer, you have probably made plans to adjust your goals and project deliverables in light of the likely drop in capacity for the year-end holiday season that's ahead. In the past week, the OpenStack Technical Committee didn't commit any new governance changes; however, we're actively working on our action items from the PTG [2]. === Weekly Meeting === The TC's weekly meeting last week was well attended [3]. Clark Boylan (clarkb) noted that contributors were reaching out to OpenDev Infrastructure Maintainers via IRC and the Zuul email list seeking help to set up and configure "third-party" CI systems to meet the requirements set by several projects (Nova, Neutron, Cinder, Manila, ...). Contributors often lack clear documentation and support for setting up these systems, leading to misplaced queries on Zuul or OpenDev channels. We agreed that there is a lack of centralized, up-to-date documentation regarding CI systems put together to test things outside of the purview of the community's Zuul CI. We have lost mindshare regarding third-party CI setup and maintenance, and this hurts the participation of vendor driver maintainers. The TC would love to see interested parties come together to re-form the "Third-Party CI SIG." If we're unable to meaningfully help contributors with this, we must evaluate whether third-party CI should remain a mandatory requirement for service projects. It was highlighted that Nova maintains a dedicated weekly topic to discuss specific third-party CI issues, a model other projects might emulate. We also discussed kicking off the election planning for the 2025.2 cycle. There will be 5 seats available on the OpenStack TC, and every OpenStack project team will elect a PTL or liaison [4]. Election officials will convene over the next few weeks to propose election dates and other details. There's still time to volunteer to be an election official. Please contact me off-list if you're interested! The next meeting of the OpenStack Technical Committee is today (11/18/2024) at 1800 UTC. This meeting will be held via IRC on OFTC's #openstack-tc channel. Please find the meeting agenda on the meeting wiki [5]. I hope you're able to attend. Please remember, you can always propose new topics for this meeting, and it's a great way to synchronously engage with the TC every week. === Governance Proposals === ==== Open for Review ==== rework the initial goal proposal as suggested by people | https://review.opendev.org/c/openstack/governance/+/931254 Propose to select the eventlet-removal community goal | https://review.opendev.org/c/openstack/governance/+/931254 Add Cinder Huawei charm | https://review.opendev.org/c/openstack/governance/+/867588 Resolve to adhere to non-biased language | https://review.opendev.org/c/openstack/governance/+/934907 ==== Merged ==== Add ubuntu noble migration goal tracking etherpad | https://review.opendev.org/c/openstack/governance/+/935461 Thank you very much for reading! On behalf of the OpenStack TC, Goutham Pacha Ravi (gouthamr) OpenStack TC Chair [1] 2025.1 "Epoxy" Release Schedule: https://releases.openstack.org/epoxy/schedule.html [2] OpenStack TC's Epoxy PTG Summary: https://lists.openstack.org/archives/list/openstack-discuss@lists.openstack… [3] TC Meeting IRC Log 2024-11-12: https://meetings.opendev.org/meetings/tc/2024/tc.2024-11-12-18.00.log.html [4] OpenStack Election website: https://governance.openstack.org/election/ [5] TC Meeting Agenda, 2024-11-19: https://wiki.openstack.org/wiki/Meetings/TechnicalCommittee

1 0

Re: [Devstack] Trying to setup development environment on VM
by Ishan Shanware (ishanwar) 19 Nov '24

19 Nov '24

Hey Martin, Artemio, Thanks for responding. The VM is sufficiently large for development with about 60 GB of root storage. I was able to fix the keystone failure, by changing the OVS_RUNDIR like this OVS_RUNDIR=$OVS_PREFIX/var/run/ovn. However the script hangs here, and doesn’t move ahead. +functions:wait_for_compute:505 echo 'Didn'\''t find service registered by hostname after 60 seconds' Didn't find service registered by hostname after 60 seconds +functions:wait_for_compute:506 openstack --os-cloud devstack-admin --os-region RegionOne compute service list I am trying to setup a development environment for the nova service. Thanks Ishan From: Martin Kopec <mkopec(a)redhat.com> Date: Tuesday, 19 November 2024 at 4:05 AM To: Artemio Mendoza <artemiom681(a)gmail.com> Cc: Ishan Shanware (ishanwar) <ishanwar(a)cisco.com>, openstack-discuss(a)lists.openstack.org <openstack-discuss(a)lists.openstack.org> Subject: Re: [Devstack] Trying to setup development environment on VM Hi there, the VM should be big enough, the snip from the logs doesn't suggest any reason why keystone didn't start. There is one job that runs devstack on debian every day - https://zuul.openstack.org/builds?job_name=devstack-platform-debian-bookwor… You can try to compare your local.conf with the one used in the jobs. On Mon, 18 Nov 2024 at 21:08, Artemio Mendoza <artemiom681(a)gmail.com<mailto:artemiom681@gmail.com>> wrote: Hi Ishan, In my experience, I was able to set up devstack in the Ubuntu 22.04 (Jammy) VM. I never tried Debian. Hopefully, it can help you. Thank you, Artemio On Mon, Nov 18, 2024 at 12:46 PM Ishan Shanware (ishanwar) <ishanwar(a)cisco.com<mailto:ishanwar@cisco.com>> wrote: Hi Everyone, I am currently trying to setup in all-in-one development environment for OpenStack on a Debian VM. The VM has 16 GB RAM and 4 CPUs. However, I face an error while trying to run FORCE=yes ./stack.sh. I get an error that keystone does not start. This causes the entire process to fail. Requesting assistance in solving this issue. I am following the this guide: https://docs.openstack.org/devstack/latest/guides/single-machine.html PFA the error log and error message on the terminal. [cid:ii_19340db3c084cff311] Thanks in advance, Ishan -- Martin Kopec IRC: kopecmartin

1 0

[tc][ptg] 2025.1 "Epoxy" Technical Committee PTG Summary
by Goutham Pacha Ravi 18 Nov '24

18 Nov '24

Hello Stackers, The following is a summary of the Technical Committee's virtual PTG discussions during the last week. For longer notes / recordings, please consult the Etherpad [1] where we kept minutes, or recordings within a YouTube playlist [2]. === Reviving the Inclusive language discussion === The TC brainstormed with the Diversity and Inclusion Working Group [3] about the renaming of the "master" development branch in repositories. We evaluated whether a TC resolution is necessary to provide a clear, standardized approach. Much of OpenDev tooling allows projects the ability to customize their default branch names, with an understanding that this could sometimes be confusing for tools and end-users. Some open-source communities have attempted similar renaming efforts but faced pushback or incomplete adoption, as highlighted by Kubernetes, Ceph, and GNOME. Additionally, there remain concerns over the feasibility, workload, and support available in the OpenStack community to undertake a coordinated renaming effort. The discussion did not yield a consensus and we will not be pursuing an OpenStack wide effort to renaming all the development branches in OpenStack. This doesn't prevent a future change of course, however. If we do undertake this in the future, we will begin by assessing the community's bandwidth for the effort and we'll identify contributors who will lead and assist before proceeding. The conversation also considered the broader scope of inclusive language. The D&I WG does not strongly recommend renaming the "master" branch of git repositories as it is not linked to problematic contexts like slavery in this case. More concerning terminology, like "slave," should be prioritized: https://wiki.openstack.org/wiki/Diversity/Inclusivity Action Items: - Evaluate and Draft a TC Resolution (gouthamr): TC resolution will propose formalizing OpenStack's stance on inclusive language, which would include specific recommendations about branch naming. - Encourage a timeline to deal with non-Inclusive language in code and documentation (tc) === Community Leaders meeting === Subtopic 1: Translating OpenStack documentation and i18n SIG's challenges The OpenStack i18n team discussed challenges and potential improvements for translating OpenStack documentation, particularly due to a slow shift from Zanata to Weblate. The translation team currently has limited activity and faces infrastructure issues, such as Weblate integration problems after a recent cloud upgrade. Key goals are to engage Asian communities (e.g., Korea, Vietnam, Indonesia) to support documentation translations, focusing initially on a subset of projects like Nova, Cinder, Neutron, and Glance. To streamline the process, the i18n team proposed starting with machine translations, which would be refined by regional language moderators. They also discussed AI-generated translations and compliance with OpenInfra's AI policy, requiring clear indication of AI involvement in commits. The i18n team needs infrastructure support, including Zuul automation for translation updates, and guidance for project core teams on reviewing translations. The discussion continued in the i18n SIG meetings during the rest of the week: https://etherpad.opendev.org/p/oct2024-ptg-i18n Action Items - I18n SIG will work on weblate migration with priority, and that will need resolving infrastructure issues as well as figuring out Zuul integration. - Sylvain Bauza (bauzas) takes on the role of the i18n TC liaison, and will highlight the SIGs issues to the TC - Begin translation efforts for a subset of projects (starting with Nova) by translating content in the “docs” folders. - Apply machine translation as a first step, with periodic updates pushed to each project until Zuul automation is set up. - Clearly mark AI-assisted translations in commit messages and add a notice on documentation pages to indicate machine translation usage, in line with OpenInfra’s AI policy. - Define and document processes for core teams to review translated content, considering creating specialized review teams for translation directories. Subtopic 2: Vulnerability Management and your project We discussed the processes within the OpenStack Vulnerability Management Team (VMT) and the need for consistency/coherence in the face of evolving regulation across the world, besides our interest in maintaining good security hygiene. A key concern expressed was the need to keep core security contacts up to date, and the potential for automatic inclusion of all OpenStack projects under the VMT's purview. Adding all project teams and deliverables under the VMT will provide consistency in our security bug management process. It will also allow the VMT to formally help smaller project teams to maintain a good security hygiene. It will allow for cross-project bugs to be resolved quickly while still maintaining the tenets of embargoed disclosures. We also discussed the possibility of a Common Vulnerabilities and Exposures (CVE) authority within OpenStack. Jeremy Stanley (fungi) emphasised that as a community we must be prioritizing alignment and avoiding premature optimizations while regulations are still being drafted. There were suggestions that the VMT wanted the TC to act as a path of escalation to projects where the core security contacts were unresponsive. If we include all TC governed projects under the VMT, there were also concerns about the VMT's workload increasing. fungi clarified factors that drain the VMT's time (mainly chasing project security contacts to triage issues in a timely manner), and didn't think that adding all OpenStack projects would be a burden on the VMT. Having consistent processes may in fact reduce the VMT's workload. There is a strong desire however to grow the VMT with dedicated volunteers. So if you're reading this and are interested, please hop into OFTC's #openstack-security channel and join us. Action Items: - Project PTLs (or DPLs) must update their core-sec teams on Launchpad/Storyboard, removing inactive members where necessary (e.g., Nova to review its `nova-coresec` members) - PTLs of each project must confirm or designate a security liaison for each project, or confirm that the PTL will take on this role. - gouthamr will propose a TC resolution to include all governed projects automatically included in VMT oversight - the security-sig will continue engaging with the Open Regulatory Compliance Working Group to monitor and influence relevant regulatory developments (without immediate process changes on our end at this time) - We will also create a path of escalation for projects not actively following security processes, which may include notifying TC without detailed specifics or, as a last resort, considering project removal from OpenStack. - The VMT and OpenStack Security Team will evaluate the need and feasibility of a common CVE authority for OpenStack and assess community feedback on its potential value. Subtopic 3: Remove Postgres CI jobs from the projects We discussed whether to discontinue Postgres CI jobs across projects, with Ironic and Neutron teams already planning to do so. This stems from a lack of testing coverage and recurring errors in some projects, particularly Neutron. The group raised concerns about the need for a consistent, community-wide approach to database support, especially given the challenges of switching database backends. Current user survey data indicate that about 5% of deployments use Postgres, but maintaining its support is not feasible without dedicated resources. Action Items - Update OpenStack documentation to clearly outline the level of support and testing for each database backend, helping users understand support status before deploying. - Projects (like Ironic, Neutron) must drop their postgres CI jobs, earlier in this cycle so we can prevent unexpected CI failures in interconnected projects - Reviving postgres support will need volunteers. The TC last published a resolution in 2017 [4] explaining the state of support for postgresql. There must be a follow up to state that non-MySQL backends are not tested within the community. Subtopic 4: Migrate CI testing from Ubuntu 22.04 to Ubuntu 24.04 (this implies python3.12 support/usage) Ghanshyam Mann (gmann) is championing the goal to ensure that the advertised runtimes for 2025.1 are enforced in the community CI jobs. Base CI job changes are already underway Action Items: - gmann will share the goal progress with the community as this progresses - project teams must triage integration job failures with Python3.12/Ubuntu Noble and prioritize fixes. They may temporarily pin selective test jobs to python3.9/Ubuntu Jammy however, we expect these issues to be resolved sooner within this release cycle. === Eventlet Removal === This was a "kick-off" cross project discussion on the proposed goal to drop the usage of the "eventlet" library across OpenStack repositories [5]. The migration plan includes identifying suitable alternatives (e.g., native threads, asyncio, awaitlet) to meet different services' needs while addressing challenges around async compatibility with existing infrastructure (e.g., WSGI). Hervé Beraud (hberaud) led this discussion, and we discussed the pop-up team that has been formed to guide this transition. I'd refer you to the summary that hberaud shared on the openstack-discuss ML earlier for a more in-depth summary of further discussions [6] Action Items: - The TC and the community must review the proposal [7] for outcomes, timeline and select the "Eventlet Removal" goal as a cross-community goal - hberaud will set up working groups focused on specific migration needs: background tasks, networking/HTTP, and database interactions. Details are outlined on the Eventlet Removal Wiki [4] - Mike Bayer (zzzeek) and hberaud will work with project teams to assess where asyncio may be beneficial or problematic, and provide guidelines on selecting alternative solutions - We need volunteers from project teams to join the pop-up team and evaluate eventlet usages in the projects they maintain. Please head to OFTC's #openstack-eventlet-removal channel to participate in the discussion === Bridging the gap between community and contributing organizations === Ildiko Vancsa (ildikov) and Jeremy Stanley (fungi) brought the fourth of a series of community discussions to the TC and the community. We brainstormed ways to increase contributions from OpenInfra member organizations, improve contributor experience, and address blockers to community growth and engagement. The goals outlined are to bridge gaps between casual contributors and experienced community members, optimize review processes, and enhance visibility of contribution pathways and expectations. The challenges discussed included visibility of documentation, contributor engagement with maintainers, review delays, and difficulty navigating IRC. The brainstorming that followed proposed some ideas to try, such as contributor spotlights for maintainers (e.g., Superuser articles or video interviews). Please review the "bridging the gap" PTG etherpad for further discussions around the topic [8] Action Items: - Project PTLs must ensure we have clear documentation on patch review and escalation paths. This includes clearly highlighting the core maintainers, liaisons and PTL and ways to contact them. Teams must also list any code review rules that projects have evolved in the documentation. - Project maintainers must consider “review days” to prioritize and expedite small patches. - Develop a Matrix/Element connection guide for OpenStack rooms as an alternative to IRC promoting the guide across "so you want to contribute" pages. - Formalize guidelines around code review pitfalls === OceanBase Database === Members of the OceanBase project community had a cross-project discussion with OpenStack maintainers through this session. OceanBase was stated to be a compatible alternative to MySQL within OpenStack, without necessitating code changes in OpenStack services. The OceanBase team aims to support broader community involvement by contributing integration code for devstack and deployment tools like Kolla, OpenStack-Helm, and OpenStack-Ansible (OSA). The team learned about resource constraints in the community CI jobs, and we brainstormed approaches to support integration testing. Action Items: - OceanBase contributors will implement OceanBase support in devstack, potentially as a plugin if setup is complex. - The team will explore an OpenStack’s CI job by working with the OpenStack QA team, targeting tempest testing - The team will initiate feature requests within deployment projects (e.g., Kolla, OpenStack-Helm, OSA), and potentially collaborate with OpenStack's Large Scale SIG === Updating the OpenStack tenant policy on CI === As a community, we have been discouraging indiscriminate "recheck"s on failed CI jobs; however, our recent experiences landing interdependent CVE fixes within several repositories necessitated fighting repeated unrelated failures. The discussion was around some desire to avoid this in the future and several approaches were brainstormed. Clark Boylan (clarkb) helped the TC understand OpenStack's policy regarding "clean check" practices in the Zuul CI [9]. This policy is intended to reduce repeated gate failures and encourage debugging. It would be an anti-pattern to explore a way to bypass this. The idea of dropping this policy was discussed and the attendees overwhelmingly agreed that there were significant downsides to doing this, including the infeasibility of re-orienting each project team towards how to responsibly craft their CI to avoid pitfalls. There was a suggestion to allocate a limited “infra budget” per project, allowing them to assess re-check frequency and test requirements based on specific project stability and job complexity. There was a tangential discussion on why Zuul's philosophy of testing does not allow re-running single failed jobs, and several contributors shared their experience with CI elsewhere where this convenience came at the expense of introducing serious bugs. Currently, the community can reach out in OFTC's #opendev channel to get help merging a change despite Zuul's disagreements, de-queing a change, or re-enqueuing it on Zuul. These bypass mechanisms were deemed sufficient for the problem at hand. Action Items: - The need to recheck stems from unstable test jobs. Teams must try to move flaky tests to experimental or periodic test queues and attempt to isolate and fix issues with them. - If you spot common failure patterns (e.g., timeouts, mirror issues, OOM errors), please raise awareness to the project teams or to the Infra administrators via the openstack-discuss mailing list or the #opendev channel on OFTC - Zuul CI could use documentation that clarifies why re-running individual jobs in a failed buildset is not a supported action to manage contributor expectations. === Testing and shipping non-OSI compatible software within OpenStack binary artifacts === This discussion was an opportunity to clarify the community's stance on adding, testing, maintaining, supporting, documenting and shipping OpenStack software that pertains to components that are not licensed with an OSI-approved license [10]. A specific recent example was Masakari's Consul integration. Consul ships with a non-OSI-compliant BSL 1.1 license, and is as such incompatible with OpenStack's license policy. However, the general rule has been that the community would strive to support and test only open source solutions. While there may be integrations to proprietary or non-OSI licensed components, they will not be tested with Community Infrastructure, and in each case, a free and open source alternative must be available. In addition, if an OpenStack service supports a feature, there must be a OSI compatible implementation that the community will support and test with CI, and there can be any number of implementations for the feature concerning non-OSI components, but these must be tested by vendors or users of such integrations. Some projects (like Neutron and Cinder) have documented guidelines for this, other projects (like Nova and Manila) enforce this as an unwritten rule. This creates ambiguity around licensing and integration expectations. At the tail end, there was a suggestion to explore the feasibility of hosting an OpenStack container registry (e.g., registry.openstack.org) This could help us better manage binary artifacts and maintain container consistency across services, though permanent registry storage remains a concern. The discussion around this item was tabled. Action Items: - The TC will formally define and document a policy requiring open-source alternatives for non-OSI software integrations. We must ensure that it is applied consistently across all OpenStack projects. - In continuation of this topic, we will review and clarify how to handle existing integrations with dependencies that have changed licenses (e.g., Redis). Determine whether to seek alternatives, discontinue first-party CI testing, or require third-party CI for these integrations. That's a wrap! It was great fun seeing you all at the PTG. I look forward to working on these AIs with you! Thanks, On behalf of the OpenStack TC, Goutham Pacha Ravi (gouthamr) OpenStack TC Chair [1] PTG Etherpad for TC discussions: https://etherpad.opendev.org/p/r.0f25532f564fb786a89cfe1de39b004b [2] Recordings of the TC PTG: https://www.youtube.com/playlist?list=PLhwOhbQKWT7XGjIwT0mtPpixpuY-tKYoh [3] PTG Etherpad of the D&I working group: https://etherpad.opendev.org/p/r.4f257833bbd7e0284c26f34e2bbc87c6 [4] TC Resolution on the state of testing of database systems: https://governance.openstack.org/tc/resolutions/20170613-postgresql-status.… [5] Eventlet removal wiki: https://wiki.openstack.org/wiki/Eventlet-removal [6] Eventlet removal PTG Summary: https://lists.openstack.org/archives/list/openstack-discuss@lists.openstack… [7] Eventlet removal Goal Proposal: https://review.opendev.org/c/openstack/governance/+/931254 [8] "Bridging the Gap" PTG Etherpad: https://etherpad.opendev.org/p/r.522712847f6a5a1d7bd2031566cde4e9 [9] "Clean Check" CI policy: https://docs.openstack.org/contributors/common/zuul-status.html#why-do-chan… [10] OSI Approved Licenses: https://opensource.org/licenses

2 1

[Devstack] Trying to setup development environment on VM
by Ishan Shanware (ishanwar) 18 Nov '24

18 Nov '24

Hello Everyone, I am currently trying to setup in all-in-one development environment for OpenStack on a Debian VM. The VM has 16 GB RAM and 4 CPUs. However, I face an error while trying to run FORCE=yes ./stack.sh. I get an error that keystone does not start. This causes the entire process to fail. Requesting assistance in solving this issue. I am following the this guide: https://docs.openstack.org/devstack/latest/guides/single-machine.html PFA the error log and error message on the terminal. [Call Trace] ./stack.sh:1126:start_keystone /opt/stack/devstack/lib/keystone:549:die [ERROR] /opt/stack/devstack/lib/keystone:549 keystone did not start Error on exit World dumping... see /opt/stack/logs/worlddump-2024-11-18-150808.txt for details ovs-vsctl: unix:/var/run/openvswitch/db.sock: database connection failed (No such file or directory) ovs-vsctl: unix:/var/run/openvswitch/db.sock: database connection failed (No such file or directory) Thanks in advance, Ishan

2 3

[watcher] re-starting a weekly irc meeting
by Marios Andreou 18 Nov '24

18 Nov '24

Hello OpenStack o/, As discussed at PTG [1] and since there are now a few of us looking to contribute to watcher, it seems like a good time to re-start a regular irc meeting in #openstack-watcher for plans, reviews or any other relevant topics. It seems that Tuesday or Thursday work well for most of the current/new contributors. I started a doodle poll with these two options there https://doodle.com/meeting/participate/id/aMoRKp5d to see if there is any clear preference between the two. I provisionally propose 1200 UTC to (try) and accommodate different time zones (obviously there is no perfect solution for this). If you are interested but the time and/or days do not work for you please reply here so we can try and work it out (If there is enough interest we may consider something like alternate times but first lets see if there is a preference or problem with the current proposal), thanks! marios [1] https://etherpad.opendev.org/p/r.79644f8f3cd04686c4f45c10aa35d8ac#L1

2 1

[Glance] Survey: Property protection file
by Cyril Roelandt 18 Nov '24

18 Nov '24

Hello, Glance allows admins to define custom rules regarding CRUD (create/read/update/delete) operations on custom image properties[1]. This is done by setting the property_protection_file option in glance-api.conf to the path to an INI file defining such permissions. We believe this feature has not seen a lot of use, and are thinking about removing it. We would like to know what you think about this. We set up an online form[2] that you can fill to help us understand real-world use cases for this feature. 1) Are you using a property protection file? 2) Are you using roles, polices, or both for property protection? (Or "None" if you do not use this feature) 3) Can you give examples of config files you use? 4) Would it be an issue for you if Glance were to remove the feature? Only the role-base protection? Only the policy-based protection? On behalf of the Glance team, Cyril [1] https://docs.openstack.org/glance/latest/admin/property-protections.html [2] https://framaforms.org/survey-glance-property-protection-file-1731944193

1 0

[neutron][requirements] Lock failures within Neutron with Eventlet 0.37.0
by Herve Beraud 18 Nov '24

18 Nov '24

Hey Neutrinos, Mid-September we released Eventlet 0.37.0 [1]. Then Jens diligently proposed a patch [2] to bump our global requirements to this newest version of Eventlet. Unfortunately this requirement upgrade is stalled by the Neutron cross job [3]. There are plenty of tests failing with lock errors in threading context (`RuntimeError: cannot notify on un-acquired lock`, please see the logs [3]). A new patch [4] changed the way we manage RLocks and threading at runtime. This patch Upgrade RLocks as the last thing we do. Green existing locks after patching modules, since patching modules might involve imports that create new locks. Neutron tests might potentially need some adjustment to properly work with this new Eventlet version. The support of Python 3.13 is coming and it includes this behavioural change, so if we want the Python 3.13 support of Eventlet in OpenStack, we need first to adjust our OpenStack deliverables to prepare the ground. Can you please check this issue with your Neutron expert eyes. Feel free to also take a look at the changelog of the version 0.37.0 of Eventlet [5], but I think the other patches are irrelevant to this context and can be safely ignored as they are unrelated to locks or threading. To finish Masakari and Mistral also failed but the errors seem a bit different than the Neutron ones. In all the cases Masakri and Mistral cross jobs are non-voting jobs. Thanks for reading. [1] https://github.com/eventlet/eventlet/issues/979 [2] https://review.opendev.org/c/openstack/requirements/+/933257 [3] https://zuul.opendev.org/t/openstack/build/e399c76765f34bfda62e2125b80caf84… [4] https://github.com/eventlet/eventlet/commit/06ec82896ebb9a26edaf6e1ad4d6339… [5] https://github.com/eventlet/eventlet/issues/979 -- Hervé Beraud Senior Software Engineer at Red Hat irc: hberaud https://github.com/4383/

2 2

[neutron] Bug deputy - week of Nov 11th
by Sławek Kapłoński 18 Nov '24

18 Nov '24

Hi, I was Neutron bug deputy last week. Here's summary of what new bugs were opened during that period: ## Critical - https://bugs.launchpad.net/neutron/+bug/2088423 - [functional test] test_ovn_nb_sync* tests fails randomly with Internal Server Error - needs assignment, - https://bugs.launchpad.net/neutron/+bug/2088285 - [neutron-tempest-plugin] neutron-dynamic-routing using os-ken Quagga are failing - assignent to Lajos, fix already proposed, we have https://review.opendev.org/c/openstack/neutron-tempest-plugin/+/935366 and https://review.opendev.org/c/openstack/neutron-tempest-plugin/+/935368 ## High - https://bugs.launchpad.net/neutron/+bug/2088166 - os-ken ``DockerImage`` class failing during the image creation - assigned to Rodolfo and patch is proposed already, - https://bugs.launchpad.net/neutron/+bug/2088286 - external port created by neutron-vpnaas with ovn driver is not fully configured - needs assignment, this seems to be bug related to the `ovn-bgp-agent` and `vpnaas` ## Medium - https://bugs.launchpad.net/neutron/+bug/2088154 - ``ProcessManager.disable`` failing if the process doesn't exist - assigned to Rodolfo and patch is proposed already, ## Low - https://bugs.launchpad.net/neutron/+bug/2087909 - When network log exists, neutron-ovn-db-sync-util has a wrong log message - needs assignment, ## Whishlist - https://bugs.launchpad.net/neutron/+bug/2087863 - fullstack job doesn't collect keepalived and other backend logs - needs assignment - https://bugs.launchpad.net/neutron/+bug/2087939 - [eventlet-deprecation] Remove the usage of eventlet in the OVS agent - needs assignment - https://bugs.launchpad.net/neutron/+bug/2087941 - [eventlet-deprecation] Remove the usage of eventlet in the OVN metadata agent - needs assignment - https://bugs.launchpad.net/neutron/+bug/2087942 - [eventlet-deprecation] Remove the usage of eventlet in the OVN agent - needs assignment - https://bugs.launchpad.net/neutron/+bug/2087943 - [eventlet-deprecation] Remove the usage of eventlet in the L3 agent - needs assignment - https://bugs.launchpad.net/neutron/+bug/2087944 - [eventlet-deprecation] Remove the usage of eventlet in the DHCP agent - needs assignment - https://bugs.launchpad.net/neutron/+bug/2087946 - [eventlet-deprecation] Remove the usage of eventlet in the SR-IOV agent - needs assignment - https://bugs.launchpad.net/neutron/+bug/2087947 - [eventlet-deprecation] Remove the usage of eventlet in the Linux Bridge agent - needs assignment, - https://bugs.launchpad.net/neutron/+bug/2087953 - [eventlet-deprecation] Remove the usage of eventlet in the Neutron API - needs assignment There is bunch of the "eventlet-deprecation" bugs reported this week. I marked all of them with the tag "eventlet-deprecation" so that all can be easily found: https://bugs.launchpad.net/neutron/+bugs?field.tag=eventlet-deprecation -- Slawek Kaplonski Principal Software Engineer Red Hat

1 0

[neutron] external port created by neutron-vpnaas with ovn driver is not fully configured
by przemyslaw.kuczynski＠ochk.pl 18 Nov '24

18 Nov '24

Guys can You help with https://bugs.launchpad.net/neutron/+bug/2088286 Hello We are deploying OpenStack/Neturon with OVN, BGP (ovn-bgp-agent) and VPNaaS and we encountered a problem - because of incomplete configuration of VPN services external port bgp agent isn't announcing it's address. Summary: external port created by neutron-vpnaas with ovn plugin/driver is not fully/properly configured and because of this ovn-bgp-agent won't announce/publish/configure routing path for this address. $ /var/lib/kolla/venv/bin/neutron-ovn-vpn-agent --version neutron-ovn-vpn-agent 24.0.2.dev42 $ openstack network show public1 +---------------------------+--------------------------------------+ | Field | Value | +---------------------------+--------------------------------------+ | admin_state_up | UP | | availability_zone_hints | | | availability_zones | | | created_at | 2024-09-06T15:48:18Z | | description | | | dns_domain | None | | id | ad8c81c1-08fd-4503-833f-912675d1c6d8 | | ipv4_address_scope | None | | ipv6_address_scope | None | | is_default | False | | is_vlan_transparent | None | | mtu | 1500 | | name | public1 | | port_security_enabled | True | | project_id | 0cfa2dc8d9024b7fa0462a9be5d8b832 | | provider:network_type | vlan | | provider:physical_network | physnet1 | | provider:segmentation_id | 2 | | qos_policy_id | None | | revision_number | 2 | | router:external | External | | segments | None | | shared | True | | status | ACTIVE | | subnets | 306ea02d-a5ec-4c1b-bd2f-bff1a88750d6 | | tags | | | tenant_id | 0cfa2dc8d9024b7fa0462a9be5d8b832 | | updated_at | 2024-09-06T15:48:19Z | +---------------------------+--------------------------------------+ $ openstack subnet show 306ea02d-a5ec-4c1b-bd2f-bff1a88750d6 +----------------------+--------------------------------------+ | Field | Value | +----------------------+--------------------------------------+ | allocation_pools | 203.0.113.226-203.0.113.254 | | cidr | 203.0.113.224/27 | | created_at | 2024-09-06T15:48:19Z | | description | | | dns_nameservers | | | dns_publish_fixed_ip | None | | enable_dhcp | False | | gateway_ip | 203.0.113.225 | | host_routes | | | id | 306ea02d-a5ec-4c1b-bd2f-bff1a88750d6 | | ip_version | 4 | | ipv6_address_mode | None | | ipv6_ra_mode | None | | name | public1 | | network_id | ad8c81c1-08fd-4503-833f-912675d1c6d8 | | project_id | 0cfa2dc8d9024b7fa0462a9be5d8b832 | | revision_number | 0 | | segment_id | None | | service_types | | | subnetpool_id | None | | tags | | | updated_at | 2024-09-06T15:48:19Z | +----------------------+--------------------------------------+ $ openstack port list --network ad8c81c1-08fd-4503-833f-912675d1c6d8 --long +--------------------------------------+---------------------------------------------+-------------------+------------------------------------------------------------------------------+--------+-----------------+----------------------------+------+ | ID | Name | MAC Address | Fixed IP Addresses | Status | Security Groups | Device Owner | Tags | +--------------------------------------+---------------------------------------------+-------------------+------------------------------------------------------------------------------+--------+-----------------+----------------------------+------+ | 1dc2b248-1d21-44ff-923f-2405f9f28f4e | | fa:16:3e:3e:97:10 | ip_address='203.0.113.228', subnet_id='306ea02d-a5ec-4c1b-bd2f-bff1a88750d6' | N/A | None | network:floatingip | | | 71a1d9eb-11bb-4278-979b-6a1d83e87ecc | | fa:16:3e:15:5e:b5 | ip_address='203.0.113.229', subnet_id='306ea02d-a5ec-4c1b-bd2f-bff1a88750d6' | ACTIVE | None | network:router_gateway | | | 7398b975-552a-408f-8289-25e52d5cb8fc | | fa:16:3e:e7:dd:5d | ip_address='203.0.113.252', subnet_id='306ea02d-a5ec-4c1b-bd2f-bff1a88750d6' | ACTIVE | None | network:router_gateway | | | ccbb42e0-3258-4ade-a5e6-c873ca0530b7 | | fa:16:3e:97:9d:d7 | | DOWN | None | network:distributed | | | e6a43606-2ea3-4967-9aa2-967c800cbdbe | | fa:16:3e:4e:e9:d8 | ip_address='203.0.113.243', subnet_id='306ea02d-a5ec-4c1b-bd2f-bff1a88750d6' | ACTIVE | None | network:router_gateway | | +--------------------------------------+---------------------------------------------+-------------------+------------------------------------------------------------------------------+--------+-----------------+----------------------------+------+ $ openstack vpn service create --router router_vpn test_vpn_service +----------------+--------------------------------------+ | Field | Value | +----------------+--------------------------------------+ | Description | | | Flavor | None | | ID | 0d8436f0-6cc5-4af5-84f7-10b54ac99784 | | Name | test_vpn_service | | Project | 0cfa2dc8d9024b7fa0462a9be5d8b832 | | Router | 30e448d5-078e-40fe-9418-901f8195b6cb | | State | True | | Status | PENDING_CREATE | | Subnet | None | | external_v4_ip | 203.0.113.246 | | external_v6_ip | None | | project_id | 0cfa2dc8d9024b7fa0462a9be5d8b832 | +----------------+--------------------------------------+ $ openstack vpn ipsec site connection create --peer-id 1.2.3.4 --peer-address 1.2.3.4 --psk 1234 --vpnservice test_vpn_service --ikepolicy ikepolicy --ipsecpolicy ipsecpolicy test_vpn_tunnel --peer-endpoint-group west-peer-epg --local-endpoint-group local_network +--------------------------+----------------------------------------------------+ | Field | Value | +--------------------------+----------------------------------------------------+ | Authentication Algorithm | psk | | Description | | | ID | ca7f63dc-685f-4f9d-bc23-47c7b5b1c577 | | IKE Policy | 1457dc12-c1ec-4574-8985-9e93dcf06f56 | | IPSec Policy | c68d172f-220f-455c-b46c-b9ff8b9e46e4 | | Initiator | bi-directional | | Local Endpoint Group ID | 0f7f645c-8ae9-4dc8-a286-40afabbc2dd7 | | Local ID | | | MTU | 1500 | | Name | test_vpn_tunnel | | Peer Address | 1.2.3.4 | | Peer CIDRs | | | Peer Endpoint Group ID | 842cc6e8-d124-4258-b9d8-8b901d06cd97 | | Peer ID | 1.2.3.4 | | Pre-shared Key | 1234 | | Project | 0cfa2dc8d9024b7fa0462a9be5d8b832 | | Route Mode | static | | State | True | | Status | PENDING_CREATE | | VPN Service | 0d8436f0-6cc5-4af5-84f7-10b54ac99784 | | dpd | {'action': 'hold', 'interval': 30, 'timeout': 120} | | project_id | 0cfa2dc8d9024b7fa0462a9be5d8b832 | +--------------------------+----------------------------------------------------+ $ openstack port list --network ad8c81c1-08fd-4503-833f-912675d1c6d8 --long +--------------------------------------+---------------------------------------------+-------------------+------------------------------------------------------------------------------+--------+-----------------+----------------------------+------+ | ID | Name | MAC Address | Fixed IP Addresses | Status | Security Groups | Device Owner | Tags | +--------------------------------------+---------------------------------------------+-------------------+------------------------------------------------------------------------------+--------+-----------------+----------------------------+------+ | 1dc2b248-1d21-44ff-923f-2405f9f28f4e | | fa:16:3e:3e:97:10 | ip_address='203.0.113.228', subnet_id='306ea02d-a5ec-4c1b-bd2f-bff1a88750d6' | N/A | None | network:floatingip | | | 3d2dbb2a-b004-4955-a4ca-4d4a88d2f702 | vpn-gw-30e448d5-078e-40fe-9418-901f8195b6cb | fa:16:3e:e5:e0:69 | ip_address='203.0.113.246', subnet_id='306ea02d-a5ec-4c1b-bd2f-bff1a88750d6' | ACTIVE | None | network:vpn_router_gateway | | | 71a1d9eb-11bb-4278-979b-6a1d83e87ecc | | fa:16:3e:15:5e:b5 | ip_address='203.0.113.229', subnet_id='306ea02d-a5ec-4c1b-bd2f-bff1a88750d6' | ACTIVE | None | network:router_gateway | | | 7398b975-552a-408f-8289-25e52d5cb8fc | | fa:16:3e:e7:dd:5d | ip_address='203.0.113.252', subnet_id='306ea02d-a5ec-4c1b-bd2f-bff1a88750d6' | ACTIVE | None | network:router_gateway | | | ccbb42e0-3258-4ade-a5e6-c873ca0530b7 | | fa:16:3e:97:9d:d7 | | DOWN | None | network:distributed | | | e6a43606-2ea3-4967-9aa2-967c800cbdbe | | fa:16:3e:4e:e9:d8 | ip_address='203.0.113.243', subnet_id='306ea02d-a5ec-4c1b-bd2f-bff1a88750d6' | ACTIVE | None | network:router_gateway | | +--------------------------------------+---------------------------------------------+-------------------+------------------------------------------------------------------------------+--------+-----------------+----------------------------+------+ routing table on gateway/network node: 203.0.113.228 dev vrf1d63891f-e7 scope link 203.0.113.243 dev vrf1d63891f-e7 scope link 203.0.113.252 dev vrf1d63891f-e7 scope link 203.0.113.229 dev vrf1d63891f-e7 scope link $ ovn-nbctl lsp-list `ovn-nbctl ls-list | grep ad8c81c1-08fd-4503-833f-912675d1c6d8 | cut -f1 -d\ ` | while read a b; do echo -n "$b "; ovn-nbctl lsp-get-addresses $a; done (3d2dbb2a-b004-4955-a4ca-4d4a88d2f702) unknown (71a1d9eb-11bb-4278-979b-6a1d83e87ecc) fa:16:3e:15:5e:b5 203.0.113.229/27 (7398b975-552a-408f-8289-25e52d5cb8fc) fa:16:3e:e7:dd:5d 203.0.113.252/27 (ccbb42e0-3258-4ade-a5e6-c873ca0530b7) fa:16:3e:97:9d:d7 (e6a43606-2ea3-4967-9aa2-967c800cbdbe) fa:16:3e:4e:e9:d8 203.0.113.243/27 (provnet-5aa931a9-ac56-4144-ab7d-c61819a46c2a) unknown 3d2dbb2a-b004-4955-a4ca-4d4a88d2f702 (vpn-gw-30e448d5-078e-40fe-9418-901f8195b6cb) stands out from other ports, has no addresses $ ovn-nbctl lsp-set-addresses 3d2dbb2a-b004-4955-a4ca-4d4a88d2f702 "fa:16:3e:e5:e0:69 203.0.113.246/27" $ ovn-nbctl lsp-list `ovn-nbctl ls-list | grep ad8c81c1-08fd-4503-833f-912675d1c6d8 | cut -f1 -d\ ` | while read a b; do echo -n "$b "; ovn-nbctl lsp-get-addresses $a; done (3d2dbb2a-b004-4955-a4ca-4d4a88d2f702) fa:16:3e:e5:e0:69 203.0.113.246/27 (71a1d9eb-11bb-4278-979b-6a1d83e87ecc) fa:16:3e:15:5e:b5 203.0.113.229/27 (7398b975-552a-408f-8289-25e52d5cb8fc) fa:16:3e:e7:dd:5d 203.0.113.252/27 (ccbb42e0-3258-4ade-a5e6-c873ca0530b7) fa:16:3e:97:9d:d7 (e6a43606-2ea3-4967-9aa2-967c800cbdbe) fa:16:3e:4e:e9:d8 203.0.113.243/27 (provnet-5aa931a9-ac56-4144-ab7d-c61819a46c2a) unknown routing table after: 203.0.113.228 dev vrf1d63891f-e7 scope link 203.0.113.243 dev vrf1d63891f-e7 scope link 203.0.113.246 dev vrf1d63891f-e7 scope link 203.0.113.252 dev vrf1d63891f-e7 scope link 203.0.113.229 dev vrf1d63891f-e7 scope link And now it's possible for tunnel to connect. I tried to identify code of ovn driver/plugin in neutron-vpnaas responsible for external port allocation, but I wasn't able to find it. --

1 0

[ironic] Having issue with deploying ISO image to node
by James Leong 18 Nov '24

18 Nov '24

Hi all, I am trying to play around with ironic, and I am having an issue with deploying an Ubuntu 20.04 ISO image onto my bare metal node. Since I am quite new to bare metal, I am not entirely sure what went wrong as I couldn't find any error messages shown on my log file (ironic-conductor, ironic-api, ironic_neutron_agent, and nova-compute-ironic). This might be my guess; the ironic inspection is hitting timeout, which is causing the deployment to timeout. However, I have no idea why the inspection is failing at timeout. Any help would be appreciated. Best, James

6 7

[openstack-helm] [Cinder] Cinder-Scheduler oslo_privsep.daemon.FailedToDropPrivileges: Privsep daemon failed to start
by daniel890723＠gmail.com 18 Nov '24

18 Nov '24

My Enviroment: | OS | Ubuntu 22.04 | | Kernel | 5.15.0-87-generic | | kubernetes | v1.30.3 | (tested with both v1.28.14 and v1.30.3) Hello I'm testing with Openstack-helm the installation works fine but when i trying to create volume from image and when creating Cinder-Scheduler shows error and the log output: Cinder-Scheduler log: 2024-11-18 01:13:21.651 1 ERROR cinder.scheduler.filter_scheduler [None req-d4455f4e-a374-4251-ab04-7c3523809b6f d7a3b766f6274ea6b83d6f4b28639b6e 5fc62074ca69400f8192bf00941616d5 - - - -] Error scheduling e838be3b-80a2-4c6c-98cc-53cc84527236 from last vol-service: cinder-volume-worker@rbd1#rbd1 : ['Traceback (most recent call last):\n', ' File "/var/lib/openstack/lib/python3.10/site-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task\n result = task.execute(**arguments)\n', ' File "/var/lib/openstack/lib/python3.10/site-packages/cinder/volume/flows/manager/create_volume.py", line 1241, in execute\n model_update = self._create_from_image(context,\n', ' File "/var/lib/openstack/lib/python3.10/site-packages/cinder/utils.py", line 669, in _wrapper\n return r(f, *args, **kwargs)\n', ' File "/var/lib/openstack/lib/python3.10/site-packages/tenacity/__init__.py", line 379, in __call__\n do = self.iter(retry_state=retry_state)\n', ' File "/var/lib/openstack/lib/python3.10/site-packages/tenacity/__init__.py", line 314, in iter\n return fut.result()\n', ' File "/usr/lib/python3.10/concurrent/futures/_base.py", line 451, in result\n return self.__get_result()\n', ' File "/usr/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result\n raise self._exception\n', ' File "/var/lib/openstack/lib/python3.10/site-packages/tenacity/__init__.py", line 382, in __call__\n result = fn(*args, **kwargs)\n', ' File "/var/lib/openstack/lib/python3.10/site-packages/cinder/volume/flows/manager/create_volume.py", line 1115, in _create_from_image\n model_update, cloned = self._prepare_image_cache_entry(\n', ' File "/var/lib/openstack/lib/python3.10/site-packages/decorator.py", line 232, in fun\n return caller(func, *(extras + args), **kw)\n', ' File "/var/lib/openstack/lib/python3.10/site-packages/cinder/coordination.py", line 239, in _synchronized\n return f(*a, **k)\n', ' File "/var/lib/openstack/lib/python3.10/site-packages/cinder/volume/flows/manager/create_volume.py", line 894, in _prepare_image_cache_entry\n model_update = self._create_from_image_cache_or_download(\n', ' File "/var/lib/openstack/lib/python3.10/site-packages/cinder/volume/flows/manager/create_volume.py", line 978, in _create_from_image_cache_or_download\n with image_utils.TemporaryImages.fetch(\n', ' File "/usr/lib/python3.10/contextlib.py", line 135, in __enter__\n return next(self.gen)\n', ' File "/var/lib/openstack/lib/python3.10/site-packages/cinder/image/image_utils.py", line 1375, in fetch\n fetch_verify_image(context, image_service, image_id, tmp)\n', ' File "/var/lib/openstack/lib/python3.10/site-packages/cinder/image/image_utils.py", line 861, in fetch_verify_image\n data = get_qemu_data(image_id, has_meta, format_raw,\n', ' File "/var/lib/openstack/lib/python3.10/site-packages/cinder/image/image_utils.py", line 700, in get_qemu_data\n data = qemu_img_info(dest,\n', ' File "/var/lib/openstack/lib/python3.10/site-packages/cinder/image/image_utils.py", line 169, in qemu_img_info\n format_name = cinder.privsep.format_inspector.get_format_if_safe(\n', ' File "/var/lib/openstack/lib/python3.10/site-packages/oslo_privsep/priv_context.py", line 269, in _wrap\n self.start()\n', ' File "/var/lib/openstack/lib/python3.10/site-packages/oslo_privsep/priv_context.py", line 283, in start\n channel = daemon.RootwrapClientChannel(context=self)\n', ' File "/var/lib/openstack/lib/python3.10/site-packages/oslo_privsep/daemon.py", line 374, in __init__\n super(RootwrapClientChannel, self).__init__(sock, context)\n', ' File "/var/lib/openstack/lib/python3.10/site-packages/oslo_privsep/daemon.py", line 187, in __init__\n self.exchange_ping()\n', ' File "/var/lib/openstack/lib/python3.10/site-packages/oslo_privsep/daemon.py", line 201, in exchange_ping\n raise FailedToDropPrivileges(msg)\n', 'oslo_privsep.daemon.FailedToDropPrivileges: Privsep daemon failed to start\n'] 2024-11-18 01:13:21.652 1 INFO cinder.message.api [None req-d4455f4e-a374-4251-ab04-7c3523809b6f d7a3b766f6274ea6b83d6f4b28639b6e 5fc62074ca69400f8192bf00941616d5 - - - -] Creating message record for request_id = req-d4455f4e-a374-4251-ab04-7c3523809b6f 2024-11-18 01:13:21.667 1 ERROR cinder.scheduler.flows.create_volume [None req-d4455f4e-a374-4251-ab04-7c3523809b6f d7a3b766f6274ea6b83d6f4b28639b6e 5fc62074ca69400f8192bf00941616d5 - - - -] Failed to run task cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create: No valid backend was found. Exceeded max scheduling attempts 3 for resource e838be3b-80a2-4c6c-98cc-53cc84527236: cinder.exception.NoValidBackend: No valid backend was found. Exceeded max scheduling attempts 3 for resource e838be3b-80a2-4c6c-98cc-53cc84527236 I also tried several way to test by set SYS_ADMIN on Cinder_scheduler pod change the version to 2023.2 but non of the above method works Is this because the pod permission or there is something i need to set when deploy using openstack-helm ? P.S : i can create Empty Volume and mount to instance

1 0

[FOSDEM] Reminder about call for participation - Virtualization and Cloud infrastructure Room 2025
by Piotr Kliczewski 17 Nov '24

17 Nov '24

All, This yeah submission deadline is enforced for all devrooms so this year our due date is on 1st of December not as all previous years on 8th. This is our gentle reminder to submit [0] your proposals. See you soon at FOSDEM! [0] https://fosdem.org/submit

1 0