November 2024 - openstack-discuss

[kolla] DHCP not accessible on compute nodes and VM IP Assignment Issue in Compute Nodes - Kolla Ansible 2024.1
by Jaganathan, Keerthana (SCC) 20 Nov '24

20 Nov '24

Hello Kolla Ansible community, We are experiencing difficulties with VM IP address assignment in our OpenStack deployment using Kolla Ansible 2024.1 on Rocky Linux 9.2. Here are the details of the issue: 1. Problem description: IP addresses are successfully assigned to VMs created on control nodes, but VMs created on compute-only nodes fail to receive IP addresses. This suggests that DHCP requests originating from the compute nodes are not reaching the DHCP agents on the control nodes. 2. Error messages/logs: Console log of a Cirros VM on a compute node shows: checking http://169.254.169.254/2009-04-04/instance-id failed 1/20: up 0.88. request failed ... failed 20/20: up 54.89. request failed failed to read iid from metadata. tried 20 failed to get instance-id of datasource 3. Configuration: - 7 nodes in total: 5 control/compute/network nodes, 2 compute-only nodes - Single bonded interface (bond0) with 3 VLANs (bond0.10, bond0.20, bond0.30) - network_interface: bond0.10 - api_interface: bond0.20 - tunnel_interface: same as network_interface - neutron_external_interface: bond0 - Provider network: VLAN 41 - ml2_conf.ini [ml2_type_vlan] network_vlan_ranges = physnet1:41:41 - 5 neutron DHCP agents, one on each control node 4. Troubleshooting steps taken: - Verified br-tun OpenFlow rules are dropping packets - Confirmed 5 DHCP agents are running, one on each control node - Openvswitch containers: agent and Here are the openvswitch open flow rules (dump) from the compute node (node16) for br-ex, br-in and br-tun (openvswitch-vswitchd)# ovs-ofctl dump-flows br-ex cookie=0xa9774500bf4a6912, duration=175366.856s, table=0, n_packets=3093, n_bytes=945104, priority=4,in_port="phy-br-ex",dl_vlan=1 actions=mod_vlan_vid:41,NORMAL cookie=0xa9774500bf4a6912, duration=177050.619s, table=0, n_packets=2, n_bytes=176, priority=2,in_port="phy-br-ex" actions=drop cookie=0xa9774500bf4a6912, duration=177050.621s, table=0, n_packets=8178, n_bytes=2363856, priority=0 actions=NORMAL (openvswitch-vswitchd)# ovs-ofctl dump-flows br-int cookie=0x98223299c10a59d3, duration=177059.641s, table=0, n_packets=0, n_bytes=0, priority=65535,dl_vlan=4095 actions=drop cookie=0x98223299c10a59d3, duration=175373.945s, table=0, n_packets=123, n_bytes=9594, priority=10,icmp6,in_port="qvoab4fc3cb-a2",icmp_type=136 actions=resubmit(,24) cookie=0x98223299c10a59d3, duration=175373.944s, table=0, n_packets=0, n_bytes=0, priority=10,arp,in_port="qvoab4fc3cb-a2" actions=resubmit(,24) cookie=0x98223299c10a59d3, duration=177059.625s, table=0, n_packets=20, n_bytes=1668, priority=2,in_port="int-br-ex" actions=drop cookie=0x98223299c10a59d3, duration=175373.946s, table=0, n_packets=3028, n_bytes=939610, priority=9,in_port="qvoab4fc3cb-a2" actions=resubmit(,25) cookie=0x98223299c10a59d3, duration=175375.861s, table=0, n_packets=2256, n_bytes=219762, priority=3,in_port="int-br-ex",dl_vlan=41 actions=mod_vlan_vid:1,resubmit(,58) cookie=0x98223299c10a59d3, duration=177059.643s, table=0, n_packets=2, n_bytes=176, priority=0 actions=resubmit(,58) cookie=0x98223299c10a59d3, duration=177059.644s, table=23, n_packets=0, n_bytes=0, priority=0 actions=drop cookie=0x98223299c10a59d3, duration=175373.946s, table=24, n_packets=123, n_bytes=9594, priority=2,icmp6,in_port="qvoab4fc3cb-a2",icmp_type=136,nd_target=fe80::f816:3eff:fe4f:8020 actions=resubmit(,58) cookie=0x98223299c10a59d3, duration=175373.945s, table=24, n_packets=0, n_bytes=0, priority=2,arp,in_port="qvoab4fc3cb-a2",arp_spa=141.52.56.110 actions=resubmit(,25) cookie=0x98223299c10a59d3, duration=177059.641s, table=24, n_packets=0, n_bytes=0, priority=0 actions=drop cookie=0x98223299c10a59d3, duration=175373.947s, table=25, n_packets=2970, n_bytes=935510, priority=2,in_port="qvoab4fc3cb-a2",dl_src=fa:16:3e:4f:80:20 actions=resubmit(,30) cookie=0x98223299c10a59d3, duration=177059.640s, table=30, n_packets=2970, n_bytes=935510, priority=0 actions=resubmit(,58) cookie=0x98223299c10a59d3, duration=177059.639s, table=31, n_packets=0, n_bytes=0, priority=0 actions=resubmit(,58) cookie=0x98223299c10a59d3, duration=177059.642s, table=58, n_packets=5351, n_bytes=1165042, priority=0 actions=resubmit(,60) cookie=0x98223299c10a59d3, duration=177059.642s, table=60, n_packets=5351, n_bytes=1165042, priority=1 actions=NORMAL cookie=0x98223299c10a59d3, duration=177059.640s, table=62, n_packets=0, n_bytes=0, priority=3 actions=NORMAL (openvswitch-vswitchd)[root@os16 /]# ovs-ofctl dump-flows br-tun cookie=0xad1cb3762184465d, duration=178072.704s, table=0, n_packets=4815, n_bytes=1114796, priority=1,in_port="patch-int" actions=resubmit(,2) cookie=0xad1cb3762184465d, duration=178072.703s, table=0, n_packets=0, n_bytes=0, priority=0 actions=drop cookie=0xad1cb3762184465d, duration=178072.703s, table=2, n_packets=1164, n_bytes=69852, priority=1,arp,dl_dst=ff:ff:ff:ff:ff:ff actions=resubmit(,21) cookie=0xad1cb3762184465d, duration=178072.702s, table=2, n_packets=7, n_bytes=528, priority=0,dl_dst=00:00:00:00:00:00/01:00:00:00:00:00 actions=resubmit(,20) cookie=0xad1cb3762184465d, duration=178072.702s, table=2, n_packets=3644, n_bytes=1044416, priority=0,dl_dst=01:00:00:00:00:00/01:00:00:00:00:00 actions=resubmit(,22) cookie=0xad1cb3762184465d, duration=178072.701s, table=3, n_packets=0, n_bytes=0, priority=0 actions=drop cookie=0xad1cb3762184465d, duration=178072.701s, table=4, n_packets=0, n_bytes=0, priority=0 actions=drop cookie=0xad1cb3762184465d, duration=178072.701s, table=6, n_packets=0, n_bytes=0, priority=0 actions=drop cookie=0xad1cb3762184465d, duration=178072.700s, table=10, n_packets=0, n_bytes=0, priority=1 actions=learn(table=20,hard_timeout=300,priority=1,cookie=0xad1cb3762184465d,NXM_OF_VLAN_TCI[0..11],NXM_OF_ETH_DST[]=NXM_OF_ETH_SRC[],load:0->NXM_OF_VLAN_TCI[],load:NXM_NX_TUN_ID[]->NXM_NX_TUN_ID[],output:OXM_OF_IN_PORT[]),output:"patch-int" cookie=0xad1cb3762184465d, duration=178072.699s, table=20, n_packets=7, n_bytes=528, priority=0 actions=resubmit(,22) cookie=0xad1cb3762184465d, duration=178072.699s, table=21, n_packets=1164, n_bytes=69852, priority=0 actions=resubmit(,22) cookie=0xad1cb3762184465d, duration=178072.699s, table=22, n_packets=4815, n_bytes=1114796, priority=0 actions=drop I would greatly appreciate any guidance on resolving this. Best regards, le3188

1 0

[kolla] IRC meeting cancelled - 20.11.2024
by Bartosz Bezak 20 Nov '24

20 Nov '24

Hello, Unfortunately, this week’s Kolla IRC meeting is cancelled. There are multiple conferences happening this week. Apologies for the late notice. Best regards, Bartosz Bezak

1 0

[openstack-ansible][horizon]The token used to make the request was domain scoped but the policy requires ['system', 'project'] scope
by Danish Khan 20 Nov '24

20 Nov '24

Dear All, I have upgraded one of my test openstack cluster from victoria to Wallaby. After upgrading the hypervisor is not showing correct numbers of VMs on compute nodes: [image: image.png] But openstack CLI is giving correct number of VMs. I am getting below error in horizon container when I click on "hypervisor" from browser: Nov 20 11:17:29 horizon-container-2b244cbd apache2[43317]: [wsgi:error] [pid 43317:tid 139897181898496] [remote x.x.x.151:37180] /openstack/venvs/horizon-23.3.1.dev4/lib/python3.8/site-packages/oslo_policy/policy.py:1065: UserWarning: Policy os_compute_api:os-migrate-server:migrate failed scope check. The token used to make the request was domain scoped but the policy requires ['system', 'project'] scope. This behavior may change in the future where using the intended scope is required I did try few thing but that did not work. Regards, Danish Khan

1 0

[Cinder] Restore volume snaptshots while volume is in use
by Jan 19 Nov '24

19 Nov '24

Hi, for providing a more enterprise like (VMware, HyperV, Proxmox) snapshot feature, it would be great to be able to revert snapshots in use. With the "revert-to-snapshot" functions everything is there to support this feature. IMO if the storage backend (driver) does not support the operation, it should raise an error (ex. resetting a ceph rbd volume while a vm is running is not possible). But if it supports the operation, we should not stop the user doing this, even if the volume is "in-use". One might introduce a force switch, to prevent triggering it unintended. A frontend component like skyline can also show additional information like the status of an attached instance. Since the "Detach and attach boot volumes" spec [1] was never fully implemented, there is no way to revert an instance to a snapshot state without completely rebuilding it. Like described in the spec draft, this is not always suitable and at least always cumbersome. -- Jan [1] https://specs.openstack.org/openstack/nova-specs/specs/stein/approved/detac… --- suggested patch ----- https://github.com/jmussmann/cinder/commit/4f971d77a94806e9ebdccab45392058c… From 4f971d77a94806e9ebdccab45392058c25005761 Mon Sep 17 00:00:00 2001 From: jmussmann <jmussmann+github(a)posteo.de> Date: Thu, 31 Oct 2024 09:13:45 +0100 Subject: [PATCH] add: allow reverting snapshots of volumes in use --- cinder/volume/api.py | 5 +++-- cinder/volume/manager.py | 4 ++-- cinder/volume/rpcapi.py | 4 ++-- 3 files changed, 7 insertions(+), 6 deletions(-) diff --git a/cinder/volume/api.py b/cinder/volume/api.py index 0e20baff0..9aa160740 100644 --- a/cinder/volume/api.py +++ b/cinder/volume/api.py @@ -390,8 +390,9 @@ class API(base.Base): """revert a volume to a snapshot""" context.authorize(vol_action_policy.REVERT_POLICY, target_obj=volume) + volume_state = volume.status v_res = volume.update_single_status_where( - 'reverting', 'available') + 'reverting', ['available', 'in-use']) if not v_res: msg = (_("Can't revert volume %(vol_id)s to its latest snapshot " "%(snap_id)s. Volume's status must be 'available'.") @@ -408,7 +409,7 @@ class API(base.Base): "snap_id": snapshot.id}) raise exception.InvalidSnapshot(reason=msg) - self.volume_rpcapi.revert_to_snapshot(context, volume, snapshot) + self.volume_rpcapi.revert_to_snapshot(context, volume, snapshot, volume_state) def delete(self, context: context.RequestContext, diff --git a/cinder/volume/manager.py b/cinder/volume/manager.py index 30de5d8c7..18914c073 100644 --- a/cinder/volume/manager.py +++ b/cinder/volume/manager.py @@ -1150,7 +1150,7 @@ class VolumeManager(manager.CleanableManager, self.create_snapshot(context, snapshot) return snapshot - def revert_to_snapshot(self, context, volume, snapshot) -> None: + def revert_to_snapshot(self, context, volume, snapshot, original_volume_state) -> None: """Revert a volume to a snapshot. The process of reverting to snapshot consists of several steps: @@ -1207,7 +1207,7 @@ class VolumeManager(manager.CleanableManager, LOG.exception(msg, msg_args) v_res = volume.update_single_status_where( - 'available', 'reverting') + original_volume_state, 'reverting') if not v_res: msg_args = {"id": volume.id, "status": 'available'} diff --git a/cinder/volume/rpcapi.py b/cinder/volume/rpcapi.py index 58de69c1c..0b36cebdd 100644 --- a/cinder/volume/rpcapi.py +++ b/cinder/volume/rpcapi.py @@ -180,11 +180,11 @@ class VolumeAPI(rpc.RPCAPI): volume=volume) @rpc.assert_min_rpc_version('3.15') - def revert_to_snapshot(self, ctxt, volume, snapshot): + def revert_to_snapshot(self, ctxt, volume, snapshot, volume_state): version = self._compat_ver('3.15') cctxt = self._get_cctxt(volume.service_topic_queue, version) cctxt.cast(ctxt, 'revert_to_snapshot', volume=volume, - snapshot=snapshot) + snapshot=snapshot, original_volume_state=volume_state) def delete_volume(self, ctxt: context.RequestContext, -- 2.47.0

3 2

#eventlet-removal - HTTP SGI working group call for discussion
by Herve Beraud 19 Nov '24

19 Nov '24

tl;dr: We should have a cross discussion concerning the replacement of Eventlet.wsgi and we want to invite you to join the discussion and hence help us to define a standardized approach in a collaborative way. The Eventlet WSGI module can be used for 2 things: 1. as a WSGI server to serve any kind of web framework (flask, routes, etc...) [1] 2. and in some cases as a framework for web applications [2] Some OpenStack deliverables are using it in one way or another. Some deliverables like Glance already have credible migrations opportunities. But some of you may still be wondering what is your option to replace the Eventlet WSGI features? That's a good question. Now that the specs defining the removal of Eventlet from oslo.service are merged we have to define the way we want to proceed for the WSGI features. Indeed, we decided to drop the WSGI support from oslo.service. That means that we have to define, at least, a "default" way to replace it. If we want to avoid having multiple ways to manage things, we have to provide some guidance and standardizations. For this reason, after the PTG, we encouraged the creation of the HTTP SGI Working Group [4]. This working group aim to answer which (W|A)SGI servers are needed or credible alternatives. Which framework we have/want/need to use in parallel with the SGI server. The outcomes created by this group will serve as "a default" to guide the community through the replacement of the WSGI features of Eventlet. We prepared comparisons of the available alternatives for each layer: - https://wiki.openstack.org/wiki/SGI-servers-comparison - https://wiki.openstack.org/wiki/web-frameworks-comparison - https://wiki.openstack.org/wiki/Http-libraries-comparison Feel free to comment and to put your pros and cons on the given related etherpads (see the links in the comparison). This topic will be discussed at each irc meeting on the #openstack-eventlet-removal channel [5], until we find a default official option. Next meeting date is: 2024 December 3rd [6]. It would be good to reach an official consensus before the Milestone 3 of Epoxy. The goal is simply to ensure that our community looks in the same direction and that everyone is on the same page. We want to ensure that nobody is left behind. Feel free to reply to this thread or to put your comments directly on the various etherpads [6][7][8][9]. Thanks for reading. [1] https://eventlet.readthedocs.io/en/latest/reference/api/eventlet.html#modul… [2] https://eventlet.readthedocs.io/en/latest/modules/wsgi.html [3] https://review.opendev.org/c/openstack/oslo-specs/+/927503 [4] https://wiki.openstack.org/wiki/Eventlet-removal#The_HTTP_SGI_Working_Group [5] https://meetings.opendev.org/meetings/openstack_eventlet_removal/ [6] https://etherpad.opendev.org/p/epoxy-eventlet-tracking [7] https://etherpad.opendev.org/p/eventlet-removal-SGI-servers-comparison [8] https://etherpad.opendev.org/p/eventlet-removal-web-frameworks-comparison [9] https://etherpad.opendev.org/p/eventlet-removal-http-libraries-comparison -- Hervé Beraud Senior Software Engineer at Red Hat irc: hberaud https://github.com/4383/

1 0

Kolla TLS
by devispain＠mail.com 19 Nov '24

19 Nov '24

Reading the Kolla docs it says: The back-end TLS cert/key can be the same certificate that is used for the VIP, as long as those certificates are configured to allow requests from both the VIP and internal networks. But there is no advice on how to do this, can someone explain how to use letsencrypt vip certs for backend tls, i have found nowhere that explains it at all. Trying to use a self signed cert for backend tls seems to break everything when using proper certs for vips

1 0

python3-dns 2.6.1 breaks Nova
by Francesco Di Nucci 19 Nov '24

19 Nov '24

Hi all, maybe this is already known (https://bugs.launchpad.net/glance/+bug/2054732) but also python3-dns-2.6.1 seems to break Nova and other services, with the error reported in the issue, that the DNS "answered udp() got an unexpected keyword argument 'ignore_errors';" It seems that Almalinux has the new version in the repos, while Rocky Linux still has 2.3.0 Best regards Francesco Di Nucci

2 2

Question about Package Inclusion and User Creation in Kolla Base Image
by Amir Hossein Ahmadi 19 Nov '24

19 Nov '24

Hello Stackers, I'm currently reading through the OpenStack Kolla source code, and I noticed a few design choices in the kolla/base image that I’d like to understand better: - The kolla/base The image includes several packages (such as Apache) that don’t appear necessary for most containers. - Additionally, user and group accounts for all services are created in the base image, even though all containers don’t use them. Could someone explain the reasoning behind including these components in the base image? Is there any documentation available that provides insight into the design approach for Kolla images? Thank you in advance for your help and guidance. Best regards, Amir Hossein, Burna innovation Bridge

1 0

Oslo Spec Review, Eventlet Removal Proposal.
by Daniel Bengtsson 19 Nov '24

19 Nov '24

Hi everyone, I hope you’re all doing well. During yesterday’s Oslo IRC meeting, the main topic was the removal of eventlet in oslo.service[1]. It seems that everyone is in agreement with this proposal, and no objections were raised. I would appreciate it if you could take another look and review the code. If we don’t receive any further objections by Monday, I propose we proceed with merging the spec. Thanks in advance for your help and feedback! [1] https://review.opendev.org/c/openstack/oslo-specs/+/927503?usp=dashboard -- Daniel Bengtsson Software engineer

2 1

[publiccloud-sig] Reminder: next meeting November 20th (tomorrow) - 0800 UTC
by Felix Kronlage-Dammers 19 Nov '24

19 Nov '24

Hi everyone, This Wednesday, tomorrow, (November 20th) the next meeting of the publiccloud-sig is going to happen. We’ll be meeting at 0800 UTC here: https://meetpad.opendev.org/publiccloud-sig <https://meetpad.opendev.org/publiccloud-sig> See also here for all other details: https://wiki.openstack.org/wiki/PublicCloudSIG <https://wiki.openstack.org/wiki/PublicCloudSIG> As I’ll be traveling, I’ll not be able to participate myself. felix -- Felix Kronlage-Dammers Sovereign Cloud Stack — standardized, built and operated by many Ein Projekt der Open Source Business Alliance - Bundesverband für digitale Souveränität e.V. Tel.: +49-30-206539-205 | Matrix: @fkronlage:matrix.org | fkr(a)osb-alliance.com

1 0