May 2025 - openstack-discuss - lists.openstack.org

[Wallaby] Interface operations not supported for ovn_metadata_agent
by davorin.mikulic＠gmail.com 07 May '25

07 May '25

Dear openstack community, I am trying to get metadata service to work on a TripleO Wallaby Openstack. Upon successful creation of the virtual machine in a geneve type internal network, namespace gets created on the compute node which hosts the VM: [root@overcloud-computesriov-0 ~]# ip netns list ovnmeta-fae7b699-3086-4213-bec7-9d91737c934c (id: 0) [root@overcloud-computesriov-0 ~]# ip netns exec ovnmeta-fae7b699-3086-4213-bec7-9d91737c934c ip a s 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 2: tapfae7b699-31@if34: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000 link/ether fa:16:3e:d0:d6:01 brd ff:ff:ff:ff:ff:ff link-netnsid 0 inet6 fe80::ac69:a2ff:feb3:89c9/64 scope link valid_lft forever preferred_lft forever The problem is that the VM cannot reach the metadata service IP 169.254.169.254 even though it has the route to it via 10.0.0.2 (DHCP port in the internal network). I can also see ovn_metadata_agent crashing every 6 seconds with the following: 2025-05-05 13:22:39.044 226779 INFO neutron.common.config [-] Logging enabled! 2025-05-05 13:22:39.044 226779 INFO neutron.common.config [-] /usr/bin/networking-ovn-metadata-agent version 18.6.1.dev209 2025-05-05 13:22:39.050 226779 WARNING oslo_config.cfg [-] Deprecated: Option "heartbeat_in_pthread" from group "oslo_messaging_rabbit" is deprecated for removal. Its value may be silently ignored in the future. 2025-05-05 13:22:39.064 226779 INFO ovsdbapp.backend.ovs_idl.vlog [-] tcp:127.0.0.1:6640: connecting... 2025-05-05 13:22:39.064 226779 INFO ovsdbapp.backend.ovs_idl.vlog [-] tcp:127.0.0.1:6640: connected 2025-05-05 13:22:39.095 226779 INFO neutron.agent.ovn.metadata.agent [-] Loaded chassis name 983d11af-8faf-4f85-99c2-62c00ed8165b (UUID: 983d11af-8faf-4f85-99c2-62c00ed8165b) and ovn bridge br-int. 2025-05-05 13:22:39.108 226779 INFO neutron.agent.ovn.metadata.ovsdb [-] Getting OvsdbSbOvnIdl for MetadataAgent with retry 2025-05-05 13:22:39.108 226779 INFO ovsdbapp.backend.ovs_idl.vlog [-] tcp:10.100.22.91:6642: connecting... 2025-05-05 13:22:39.109 226779 INFO ovsdbapp.backend.ovs_idl.vlog [-] tcp:10.100.22.91:6642: connected 2025-05-05 13:22:39.126 226779 INFO oslo_service.service [-] Starting 2 workers 2025-05-05 13:22:39.136 226779 INFO oslo.privsep.daemon [-] Running privsep helper: ['sudo', 'neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'privsep-helper', '--config-file', '/etc/neutron/neutron.conf', '--config-file', '/etc/neutron/plugins/networking-ovn/networking-ovn-metadata-agent.ini', '--privsep_context', 'neutron.privileged.default', '--privsep_sock_path', '/tmp/tmpnbw1fxbg/privsep.sock'] 2025-05-05 13:22:39.171 226809 INFO neutron.agent.ovn.metadata.ovsdb [-] Getting OvsdbSbOvnIdl for MetadataAgent with retry 2025-05-05 13:22:39.173 226809 INFO ovsdbapp.backend.ovs_idl.vlog [-] tcp:10.100.22.95:6642: connecting... 2025-05-05 13:22:39.174 226810 INFO neutron.agent.ovn.metadata.ovsdb [-] Getting OvsdbSbOvnIdl for MetadataAgent with retry 2025-05-05 13:22:39.175 226809 INFO ovsdbapp.backend.ovs_idl.vlog [-] tcp:10.100.22.95:6642: connected 2025-05-05 13:22:39.176 226810 INFO ovsdbapp.backend.ovs_idl.vlog [-] tcp:10.100.22.67:6642: connecting... 2025-05-05 13:22:39.178 226810 INFO ovsdbapp.backend.ovs_idl.vlog [-] tcp:10.100.22.67:6642: connected 2025-05-05 13:22:39.204 226809 INFO eventlet.wsgi.server [-] (226809) wsgi starting up on http:/var/lib/neutron/metadata_proxy 2025-05-05 13:22:39.209 226810 INFO eventlet.wsgi.server [-] (226810) wsgi starting up on http:/var/lib/neutron/metadata_proxy 2025-05-05 13:22:40.270 226779 INFO oslo.privsep.daemon [-] Spawned new privsep daemon via rootwrap 2025-05-05 13:22:40.167 226859 INFO oslo.privsep.daemon [-] privsep daemon starting 2025-05-05 13:22:40.175 226859 INFO oslo.privsep.daemon [-] privsep process running with uid/gid: 0/0 2025-05-05 13:22:40.180 226859 INFO oslo.privsep.daemon [-] privsep process running with capabilities (eff/prm/inh): CAP_DAC_OVERRIDE|CAP_DAC_READ_SEARCH|CAP_NET_ADMIN|CAP_SYS_ADMIN/CAP_DAC_OVERRIDE|CAP_DAC_READ_SEARCH|CAP_NET_ADMIN|CAP_SYS_ADMIN/none 2025-05-05 13:22:40.180 226859 INFO oslo.privsep.daemon [-] privsep daemon running as pid 226859 2025-05-05 13:22:40.972 226779 INFO neutron.agent.ovn.metadata.agent [-] Provisioning metadata for network fae7b699-3086-4213-bec7-9d91737c934c 2025-05-05 13:22:41.159 226779 CRITICAL neutron [-] Unhandled error: neutron.privileged.agent.linux.ip_lib.InterfaceOperationNotSupported: Operation not supported on interface tapfae7b699-31, namespace ovnmeta-fae7b699-3086-4213-bec7-9d91737c934c. 2025-05-05 13:22:41.159 226779 ERROR neutron Traceback (most recent call last): 2025-05-05 13:22:41.159 226779 ERROR neutron File "/usr/bin/networking-ovn-metadata-agent", line 10, in <module> 2025-05-05 13:22:41.159 226779 ERROR neutron sys.exit(main()) 2025-05-05 13:22:41.159 226779 ERROR neutron File "/usr/lib/python3.6/site-packages/neutron/cmd/eventlet/agents/ovn_metadata.py", line 24, in main 2025-05-05 13:22:41.159 226779 ERROR neutron metadata_agent.main() 2025-05-05 13:22:41.159 226779 ERROR neutron File "/usr/lib/python3.6/site-packages/neutron/agent/ovn/metadata_agent.py", line 41, in main 2025-05-05 13:22:41.159 226779 ERROR neutron agt.start() 2025-05-05 13:22:41.159 226779 ERROR neutron File "/usr/lib/python3.6/site-packages/neutron/agent/ovn/metadata/agent.py", line 278, in start 2025-05-05 13:22:41.159 226779 ERROR neutron self.sync() 2025-05-05 13:22:41.159 226779 ERROR neutron File "/usr/lib/python3.6/site-packages/neutron/agent/ovn/metadata/agent.py", line 61, in wrapped 2025-05-05 13:22:41.159 226779 ERROR neutron return f(*args, **kwargs) 2025-05-05 13:22:41.159 226779 ERROR neutron File "/usr/lib/python3.6/site-packages/neutron/agent/ovn/metadata/agent.py", line 357, in sync 2025-05-05 13:22:41.159 226779 ERROR neutron self.provision_datapath(datapath) 2025-05-05 13:22:41.159 226779 ERROR neutron File "/usr/lib/python3.6/site-packages/neutron/agent/ovn/metadata/agent.py", line 585, in provision_datapath 2025-05-05 13:22:41.159 226779 ERROR neutron ip2.addr.delete(cidr) 2025-05-05 13:22:41.159 226779 ERROR neutron File "/usr/lib/python3.6/site-packages/neutron/agent/linux/ip_lib.py", line 544, in delete 2025-05-05 13:22:41.159 226779 ERROR neutron delete_ip_address(cidr, self.name, self._parent.namespace) 2025-05-05 13:22:41.159 226779 ERROR neutron File "/usr/lib/python3.6/site-packages/neutron/agent/linux/ip_lib.py", line 842, in delete_ip_address 2025-05-05 13:22:41.159 226779 ERROR neutron net.version, str(net.ip), net.prefixlen, device, namespace) 2025-05-05 13:22:41.159 226779 ERROR neutron File "/usr/lib/python3.6/site-packages/oslo_privsep/priv_context.py", line 247, in _wrap 2025-05-05 13:22:41.159 226779 ERROR neutron return self.channel.remote_call(name, args, kwargs) 2025-05-05 13:22:41.159 226779 ERROR neutron File "/usr/lib/python3.6/site-packages/oslo_privsep/daemon.py", line 224, in remote_call 2025-05-05 13:22:41.159 226779 ERROR neutron raise exc_type(*result[2]) 2025-05-05 13:22:41.159 226779 ERROR neutron neutron.privileged.agent.linux.ip_lib.InterfaceOperationNotSupported: Operation not supported on interface tapfae7b699-31, namespace ovnmeta-fae7b699-3086-4213-bec7-9d91737c934c. 2025-05-05 13:22:41.159 226779 ERROR neutron 2025-05-05 13:22:41.592 226810 INFO oslo_service.service [-] Parent process has died unexpectedly, exiting 2025-05-05 13:22:41.592 226809 INFO oslo_service.service [-] Parent process has died unexpectedly, exiting 2025-05-05 13:22:41.593 226810 INFO eventlet.wsgi.server [-] (226810) wsgi exited, is_accepting=True 2025-05-05 13:22:41.593 226809 INFO eventlet.wsgi.server [-] (226809) wsgi exited, is_accepting=True I can confirm that I tried assigning and deleting the IP to the interface tapfae7b699-31 from the host and from the container (podman exec -it --user root container-id bash) and succeeded. I didn't succeed to assign IP to the interface with a regular neutron user (podman exec -it container-id bash). Also, nova metadata service hosted on the controller is accessible from the compute node: [root@overcloud-computesriov-0 ~]# curl http://10.100.22.127:8775 1.0 2007-01-19 2007-03-01 2007-08-29 2007-10-10 2007-12-15 2008-02-01 2008-09-01 2009-04-04 Did anyone come across anything like this? Best regards, Davorin

3 8

[OVN][Neurton] Help in capturing metrics part of scale testing in OVN
by keerthi vassan 07 May '25

07 May '25

Hello Team Myself new to OVN with Openstack ( provider network based model) , as part of this I am trying to perform an iperf based network test to understand how traffic flow is happening. As part of that, how to validate how much time took for logical flow computation by services in hypervisor. For specific tests, looking for a way to figure out, how much services like ovn-controller, vswitchd & ovsdb server are contributing to it . Do we have any specific tooling to capture these metrics ? Can you please help with this ? Regards, Keerthivasan

2 1

[neutron] OVS Hardware Offloading Performance Issues
by Serhat Rıfat Demircan 06 May '25

06 May '25

Hello everyone, I've been testing OVS hardware offloaded ports for a while. I have ConnectX-5 and ConnectX-6 cards in my lab. Iperf tests are looking promising, but when I try to use hardware offloaded ports for krbd traffic, after some time, traffic falls back to the software path. Another interesting thing for me is without changing anything, it can also switch back to the hardware path. This issue mostly occurs while I am doing this test with 3 virtual machines on the same hypervisor. I'm not sure if this is a configuration issue or I'm hitting to cards limit. If any of you guys are experienced with this all ideas are more than welcome. Openstack version: Caracal Operation System: Ubuntu 22.04 Thank you, Serhat

1 1

[tc][all] OpenStack Technical Committee Weekly Summary and Meeting Agenda (2025.2/R-21)
by Goutham Pacha Ravi 06 May '25

06 May '25

Hello Stackers, Next week is milestone-1 of the 2025.2 ("Flamingo") release cycle [1]. This is typically a bug-targeting milestone across OpenStack project teams. During the last week, the OpenStack Technical Committee adopted a resolution to extend the purview of the OpenStack Vulnerability Management Team (VMT) to cover all OpenStack project teams [2]. This means that all of the user- and operator-facing deliverables within OpenStack will follow the Vulnerability Management process [3] consistently, and project teams no longer need to explicitly opt-in. To make this work in reality, project teams must ensure that their bug trackers are configured to explicitly share private security bugs with the VMT, and ensure that core security contacts are up to date. I'll share more on this in a dedicated post to this mailing list. Alongside this resolution, the TC also approved the creation of "grain-ui" as an OpenStack project under the telemetry team, and approved the retirement of the "openstack-helm-infra" project because the code is now part of the "openstack-helm" project. As you are aware, the OpenInfra Summit is coming to France this October [4]. The call for papers is open until 2025-06-13. Funding exists through the OpenInfra Foundation's Travel Support Program. If you're interested in applying, please do so by the deadline: 2025-07-09 [5]. === Weekly Meeting === The TC held its weekly meeting on 2025-04-29 in OFTC's #openstack-tc IRC channel [6]. We noted a change in affiliation with regard to Ghanshyam Maan (gmaan), who is now employed at Red Hat. This change does not negatively affect the TC's affiliation diversity requirement [7]. The TC caught up on several prior action items. It was noted that the base upgrade test jobs have been modified to now test upgrading from the 2025.1 ("Epoxy") release to the current release, and this change should have been reflected in all project-specific test jobs. SLURP (N-2→N upgrade) jobs are configured as voting by default but can be opted out of by projects, since the release under development, 2025.2 ("Flamingo"), isn't a SLURP release. We agreed that Sylvain Bauza (bauzas) and myself will serve as the VMT's TC liaisons. Our role will be to bridge communication between the VMT and project teams to coordinate security fixes. Discussing the i18n SIG's challenges with translation infrastructure provided us with a segue into OpenStack Special Interest Groups (SIG) Activity & Maintenance Processes. SIGs do not maintain any OpenStack deliverables; however, several of them maintain code repositories on OpenDev [8] that could be useful. In many cases, there isn't a clear process to onboard new contributors or maintainers, and communication breaks down if existing SIG chairs are unresponsive. We concluded that SIGs should have better documented succession and onboarding paths to avoid stagnation. We would like to do a health check of SIGs over the release and embrace any new ways to make SIGs sustainable. The next meeting of the OpenStack TC is today, 2025-05-06. This will be our monthly Video+IRC meeting. You're welcome to join any medium that you prefer. We'll take notes as usual and upload the meeting's recording to the TC's channel on YouTube [9]. Please find important meeting information such as the venue or the agenda on the meeting's wiki page [10]. I hope you'll be able to join us there. === Governance Proposals === ==== Merged ==== - Retire openstack-helm-infra repo | https://review.opendev.org/c/openstack/governance/+/947611 - Add openstack/grain-ui | https://review.opendev.org/c/openstack/governance/+/948252 - [resolution] Extend scope of VMT to cover all project teams | https://review.opendev.org/c/openstack/governance/+/944817 === Upcoming Events === - 2025-05-06: (Informal) OpenInfra Board discussion on AI policy: https://etherpad.opendev.org/p/board-informal-ai-contribution - 2025-05-13: OpenInfra Board Meeting: https://board.openinfra.org/ - 2025-05-22: OpenInfra UG Meetup, Sweden: https://www.meetup.com/openinfra-user-group-sweden/events/306139678/ - 2025-06-28: OpenInfra+Cloud Native Day, Vietnam: https://www.vietopeninfra.org/void2025 Thank you very much for reading! On behalf of the OpenStack TC, Goutham Pacha Ravi (gouthamr) OpenStack TC Chair [1] 2025.2 "Flamingo" Release Schedule: https://releases.openstack.org/flamingo/schedule.html [2] TC Resolution to extend the VMT's scope: https://governance.openstack.org/tc/resolutions/20250317-extend-scope-VMT-c… [3] VMT process: https://security.openstack.org/vmt-process.html [4] OpenInfra Summit 2025, Paris-Saclay: https://summit2025.openinfra.org/ [5] OpenInfra Summit Travel Support Program Application: https://openinfrafoundation.formstack.com/forms/openinfra_tsp [6] TC Meeting IRC Log 2025-04-29: https://meetings.opendev.org/meetings/tc/2025/tc.2025-04-29-17.00.html [7] OpenStack TC Affiliation Diversity Requirement: https://governance.openstack.org/tc/reference/charter.html#tc-diversity-req… [8] Repositories maintained by OpenStack SIGs: https://governance.openstack.org/tc/reference/sig-repos.html [9] OpenStack TC on YouTube: https://www.youtube.com/@openstack-tc [10] TC Meeting Agenda, 2025-05-06: https://wiki.openstack.org/wiki/Meetings/TechnicalCommittee#Next_Meeting

1 0

[Cinder] : Group API query
by Harsh Ailani 06 May '25

06 May '25

Hello team, In one of our customer requirements we came across a use-case to move a volume from one group to another directly. Currently user needs to first remove the volume from the current group and add it to the new group. Wanted to know, if there is one direct cinder API or method to directly move the volume from one group to another ? Thanks and Regards, Harsh Ailani

3 9

Student Opportunity - CMU Summer Semester 2025
by Kendall Nelson 05 May '25

05 May '25

Hello All! Are any of you interested in mentoring students to work on any projects you have on the backlog? This is what I have from one of the Profs of the course: ------- To remind you of the ask from last year (and *call out differences for this year*): - We are looking for projects that a team of 4-5 students could tackle together with at least two mentors for each project. As we saw last year, mentors can certainly overlap projects if appropriate and they have the time. (Life happens and having the built-in mentor redundancy helps. I’ve had mentors get laid off, change jobs, and take summer vacation.) - Mentors are expected to meet student teams once a week for an hour (via any video conference setup folks want to use), and to be available by email during the rest of the week to answer any urgent questions. - *This summer we are running the class from 13 May to 31 July (11 weeks)*. - *We want to try teaching concurrently in both campuses Doha, Qatar (GMT+3) and Pittsburgh (GMT-5), USA. The entire course will be taught virtually this year, without a classroom.* I certainly did something similar a few years ago when I was teaching at Hopkins (20 students) with another group in Galway (16 students). The morning class in Pitt will be the afternoon in Doha. - *We likely have 15-20 students in each location, so if you had on the order of 4-5 team projects with mentors that fit the format, like last year, that would be fantastic. * - *We are considering going so far as to choose the teams across time zones to get them working remotely from the start.* Last year, after six weeks together in class and daily stand-ups, the students scattered home away from Doha, and all of them worked remotely the last four weeks. They proved they could work remotely together. Of course, the relationships with mentors have always been remote. The profs in Doha and Pitt want to try remote from the beginning. (I have a few concerns but I’m also always up to experiment on students.) - We post the projects on the first day of class and will organize the teams in that first couple of days, so student teams are introduced to their mentors in the first week of class and expected to organize that first meeting to begin the project learning curve. That’s when mentors point students at any tutorials and bootstrap materials, recommended getting started materials, etc. - We have set the expectations with the students that they will be spending 20-40 hours of time per week on the project. It is an internship-like experience. - *Two co-teachers run classes on three days a week for 50 minutes, and I will guest lecture a collection of classes. *(Last year, there was just the real professor and I.) - The three of us will provide a coaching session with each team to ensure they are working with the mentors well. - Students generally have Windows or Mac laptops, but we have teaching assistants on each site that we can start to prep any other access to resources they might need. - As with last year, mentors have a lot of freedom to experiment. Some have run joint sessions if they are mentoring more students for learning curves. Some have run Slack or Discord channels. ------ So, who is interested in working with students? Please let me know if you have any additional questions. -- Kendall Nelson *Senior Upstream Developer Advocate* The OpenInfra Foundation <https://openinfra.dev/>

1 1

[barbican] Weekly meeting canceled 5-5-2025
by Grzegorz Grasza 05 May '25

05 May '25

Hi all, Due to conflicts and time off, we have to cancel the Barbican weekly meeting today (5-5-2025). Please reach out via IRC if you need anything. Thanks! / Greg

1 0

[kolla][OpenStackAnsible] 2025.1 Epoxy release for cycle-trailing projects
by Elõd Illés 05 May '25

05 May '25

Hi teams, This is a gentle reminder that the following projects with cycle trailing release model haven't done their 2025.1 Epoxy release yet: [kolla] ansible-collection-kolla kayobe kolla-ansible kolla [OpenStackAnsible] openstack-ansible-roles openstack-ansible Note, that next week we'll ask for cutting stable/2025.1 branches, as kind of a Feature Freeze, 3 weeks prior the 2025.1 Epoxy Cycle-Trailing Release Deadline on June 5th, 2025 [1] (exactly one month away). [1] https://releases.openstack.org/flamingo/schedule.html#f-cycle-trail Thanks in advance, Előd Illés irc: elodilles @ #openstack-release

1 0

[tacker] Skip next IRC meeting
by Yasufumi Ogawa 05 May '25

05 May '25

Hi Tacker team, Due to a holiday in Japan and most of us will not join, I would like to skip IRC meeting tomorrow. Thanks, Yasufumi

1 0

[manila] Share Encryption Collab Review meeting on 2025-05-08
by Goutham Pacha Ravi 03 May '25

03 May '25

Hello Zorillas and interested stackers, We're hosting a discussion on meetpad during the weekly Manila Team meeting slot [1][2] next week, 2025-05-08 (Thursday) to further discuss the design of share encryption. The discussion will follow a proposed specification [3], and seeks to align a number of folks that are implementing the API/UX portions, back end storage driver pieces and functional tests. The meetpad discussion will be at 15:30 UTC. We'll have our weekly IRC meeting as usual at 1500 UTC. Thanks, and hope to see you there, Goutham Pacha Ravi (gouthamr) [1] Manila Team Meeting: https://meetings.opendev.org/#Manila_Team_Meeting [2] ICS for this collab review: https://gist.github.com/gouthampacha/0eaaad7bb2eec343dd56cf15f2f87f9a [3] Design Spec for share encryption:

1 0