November 2024 - openstack-discuss

OpenStack Cinder - Volume-ID in cinder DB
by Harsh Ailani 12 Nov '24

12 Nov '24

Hello contributors, I am working on clone operation for a volumegroup and in that the volume ID auto-generated by OpenStack is not the one with which the clone volumes are being created on my IBM storage backend. This is as per design that IBM Storage backend will take the source-volume ID in the source-volumegroup and add a suffix to it while cloning. Now since there is a mismatch in the IDs between cinder and storage backend, because of which I am unable to perform any volume operations on this cloned volume. So, I was wondering if I can change the volume ID in cinder DB? I know that it’s a unique key and there is a limit of 36 characters for this field in cinder DB, but since I cannot change the volume in storage backend, I wanted to check if there is a way I can alter the volume ID in cinder DB? Kindly share your thoughts/suggestions on this matter. Thanks and Regards, Harsh Ailani

2 4

[OpenInfra Days NA] Recording/Slides from Workshop From zero to code
by Kaster, Jörn 12 Nov '24

12 Nov '24

Hello together, I have seen that on the OpenInfra Days NA there was a Session named 'From zero to code: A hands-on guide to making your first OpenStack code contribution' [1]. Unfortunately I can't attend the OpenInfraDays NA. Now the open question is the following: Is there any Recording/Slides or other Material available? Thank you for any information. Mit freundlichen Grüßen / Best Regards i.A. Jörn Kaster Cloud Platform Engineer Note: My work hours may not be your work hours. Please do not feel the need to respond during a time that is not convenient for you. EPG – Ehrhardt Partner Group [EPG_Logo_withClaim_rgb_transparent-background.png]<https://www.epg.com/de/> __________________________________________ EPX Ehrhardt + Partner Xtended GmbH Alte Römerstraße 3 56154 Boppard-Buchholz Germany Phone: (+49) 67 42 / 87 27 0 Fax: (+49) 67 42 / 87 27 50 E-Mail: info(a)epg.com Internet: www.epg.com __________________________________________ CEO: Marco Ehrhardt, Markus Derksen Commercial register Koblenz HRB 22546 Registered office: Boppard

1 0

[tc][all] OpenStack Technical Committee Weekly Summary and Meeting Agenda (2025.1/R-20)
by Goutham Pacha Ravi 12 Nov '24

12 Nov '24

Hello Stackers, This week is milestone-1 of the 2025.1 ("Epoxy") release cycle [1]. We've been hard at work implementing the action items from the recent Virtual Project Teams Gathering (PTG) [2]. This past week, we didn’t pursue any new governance changes, though several remain in draft as follow-ups from the PTG. We’re also excited about the extended deadline for the Call for Proposals for the OpenInfra Days event at SCaLE 2025. The new deadline is Nov 15, 2024 [3]. === Weekly Meeting === Last week, the Technical Committee met concurrently on video [4] and IRC [5]. Allison Price (aprice) and Jimmy McArthur (jimmymcarthur) from the OpenInfra VMware Migration Working Group [6] joined us to discuss the group's charter and activities so far. The working group’s goal is not only to address gaps and workarounds but also to influence the future development of OpenStack software. The group has published a whitepaper [7] for operators planning to migrate cloud workloads. The TC expressed interest in collaboration between the working group and the current maintainers of the VMware hypervisor driver in OpenStack Nova. We also highlighted low contributor numbers in several critical project teams mentioned in the whitepaper, including those maintaining OpenStack Compute High Availability (Masakari), OpenStack Storage Backup and Restore (Freezer), OpenStack Resource Optimization (Watcher), and OpenStack Database Service (Trove). The working group plans to hold regular meetings soon and will share details via this mailing list. The TC reviewed the maintenance status of the OpenStack documentation styling extension for Python Sphinx, openstackdocstheme. This library is outdated, as it depends on older versions, but there aren’t enough maintainers to review changes, which could impact documentation across all projects. Ideally, maintainers of this library should have expertise in Sphinx and JavaScript, making the TC a less-than-ideal group to support it. We encourage migrating documentation away from this custom tool, as Sphinx styling extensions have advanced and may now suffice without heavy customization. While there’s no immediate migration plan, we’re looking for volunteers to lead this effort. In the meantime, we’ll raise concerns with the Oslo project team about the lack of reviews for openstackdocstheme changes. The next meeting of the OpenStack Technical Committee is today (2024-11-12) on OFTC's #openstack-tc channel. You can find the agenda on the Meeting wiki [8]. I hope you'll be able to join us. === Governance Proposals === ==== Open for Review ==== Propose to select the eventlet-removal community goal | https://review.opendev.org/c/openstack/governance/+/931254 Add Cinder Huawei charm | https://review.opendev.org/c/openstack/governance/+/867588 Thank you very much for reading! On behalf of the OpenStack TC, Goutham Pacha Ravi (gouthamr) OpenStack TC Chair [1] 2025.1 "Epoxy" Release Schedule: https://releases.openstack.org/epoxy/schedule.html [2] Superuser compilation of PTG summaries: https://superuser.openinfra.dev/articles/october-2024-openinfra-ptg-summary/ [3] OpenInfra Days CFP: https://lists.openinfra.dev/archives/list/foundation@lists.openinfra.dev/me… [4] TC Meeting Video Recording, 2024-11-05: https://youtu.be/o_w3OutGEfM [5] TC Meeting IRC Log 2024-11-05: https://meetings.opendev.org/meetings/tc/2024/tc.2024-11-05-18.04.log.html [6] OpenStack VMware Migration Group: https://www.openstack.org/vmware-migration-to-openstack/ [7] VMware Migration to OpenStack White Paper: https://www.openstack.org/vmware-migration-to-openstack-white-paper [8] TC Meeting Agenda, 2024-11-12: https://wiki.openstack.org/wiki/Meetings/TechnicalCommittee#Next_Meeting

1 0

[openstack-ansible] Victoria - enable neutron plugin Subnet_dns_publish_fixed_ip
by Rambo Rambo 11 Nov '24

11 Nov '24

Hi Team I am running openstack-ansible Victoria release and would need to activate automatic dns records creation when a new instance is created. So, I am looking for the way to enable neutron plugin Subnet_dns_publish_fixed_ip. After which I can enable dns_publish_fixed_ip in the subnet properties Regards Rambo

2 3

[telemetry] Event Pipeline Config and Use Cases
by Fernandes Figueiredo, Alex 11 Nov '24

11 Nov '24

Hello Openstack Telemetry Team, I'm currently working on re-enabling the Openstack Telemetry services for the StarlingX Openstack (STX-O) project. Openstack telemetry was integrated into STX-O long time ago, but it has been disabled for the past few years. Since the last time telemetry was used by STX-O, it seems that important updates happened, including: * Ceilometer API deprecation * Ceilometer Collector deprecation * Panko (event storage) deprecation My questions are mainly related to the last one (Panko deprecation): * Since Panko is deprecated and there is no actual replacement for event storage [1], is there any other possible telemetry use case to justify keeping the event pipeline enabled and generating data? * Are there any aodh alarming mechanisms that relies on events generated by Ceilometer event pipelines? If so, which "sinks.publishers" should be used to ensure the integration between the Ceilometer event pipeline and the aodh alarm service? * If event pipeline is enabled, what are the publishers currently supported for events? * The current event pipeline configuration (event_pipeline.yaml) in STX-O defines the "sinks.publishers" as "gnocchi://". However, based on Ceilometer doc, it looks like Gnocchi only supports samples/meters, it doesn't support events. Is my interpretation of the documentation correct or does Gnocchi support events and can be really used as a publisher for the event pipeline? [1] OpenInfra summit Vancouver 23 - Openstack Observability revamped<https://youtu.be/AhQJEBR5BUg?si=JbBuLG2iaaDZW7Iu&t=1590> Thank you in advance for any possible help! I'm looking forward to getting Openstack telemetry up and running on StarlingX Openstack. Best regards, Alex Figueirêdo.

3 4

[nova][neutron] Persistent PCI topology for network interfaces
by jan＠uponu.com 11 Nov '24

11 Nov '24

Hi, we have come came across the issue that some appliances rely on the PCI topology to assign network interfaces and not the mac address. This is the a problem, because the PCI topology changes, when an instance (the libvirt config) is recreated. Is there a way to make the network interface order persistent in the PCI topology and if not can someone give me a hint on the steps that would need to be done in nova? I think this could be an important feature in general. Thanks Jan ------ E-Mail: jan(a)uponu.com

2 1

[neutron] Bug deputy report (week starting on November 4)
by Elvira Garcia Ruiz 11 Nov '24

11 Nov '24

Hi folks! Here you have the list of bugs reported for Neutron during last week (04 Nov - 10 Nov) Many new RFEs this week. High ------ - https://bugs.launchpad.net/neutron/+bug/2086740 Deadlock when metadata agent starts Fix by Jakub https://review.opendev.org/c/openstack/neutron/+/934188 [Merged] - https://bugs.launchpad.net/neutron/+bug/2087824 random router related test failures since wsgi switch Unassigned Medium ----------- - https://bugs.launchpad.net/neutron/+bug/2087822 When security group rules have the same normalized CIDR, neutron-ovn-db-sync-util has a wrong log message. Unassigned - https://bugs.launchpad.net/neutron/+bug/2086787 Neutron fails with postgres on agents Unassigned Low ------ - https://bugs.launchpad.net/neutron/+bug/208678 ovn_db_revision_check is not required anymore Patch proposed by Sharat Sharma https://review.opendev.org/c/openstack/neutron/+/934241 Wishlist ---------- - https://bugs.launchpad.net/neutron/+bug/2086776 [RFE] Configurable Agent Termination on OVS Restart Unassigned - Reporter has updated the RFE description - https://bugs.launchpad.net/neutron/+bug/2086794 [RFE] L3 Agent Readiness Status for HA Routers Unassigned - https://bugs.launchpad.net/neutron/+bug/2086799 [RFE] OVS Agent Synchronization Status for Container Environments Unassigned - https://bugs.launchpad.net/neutron/+bug/2087541 [RFE] Add OVN Ganesha Agent to setup path for VM connectivity to NFS-Ganesha Assigned to Amir Nikpour Best regards, Elvira

1 0

[openstacksdk] Flakey unit test makes assumptions about the speed of API results
by Michael Still 09 Nov '24

09 Nov '24

So I've spent a day or so debugging an occasionally flakey unit test that I see in the py3 test suite for openstacksdk. I'm at the point that I want to check that I actually understand what it's doing before proposing a fix. Specifically, openstack.tests.unit.test_stats.TestStats.test_servers_no_detail periodically fails to find a statsd value: self.assert_reported_stat( 'openstack.api.compute.GET.servers.200', value='0', kind='ms' ) This assertion is effectively saying that a GET to the servers compute API should return a HTTP 200, and take 0 milliseconds to execute. The unit test is failing because sometimes it doesn't take 0 milliseconds. In my debugging for example, it takes 4 milliseconds in my environment every now and then: Considering key openstack.api.compute.GET.servers.200, value 4.000000|ms Value 4.000000, kind ms Value 0 != 4.000000 Am I understanding this correctly? It seems to me that perhaps we should be asserting the call took less than a threshold (5 milliseconds) instead of just insisting it always be instantaneous? Michael

1 0

[neutron][ptg] 2025.1 Epoxy PTG summary
by Brian Haley 09 Nov '24

09 Nov '24

Hi all, First, sorry for taking so long to get this out, was traveling the week after the PTG. Second, thanks for all that attended the meetings, we had a good turnout for Neutron, the Ironic cross-team and some eventlet discussions that were important to the team. For complete notes see the etherpad [0] but a summary is below. And please comment if you feel I missed anything. Thank you, -Brian ### Dalmatian retrospective ### Everyone felt we were able to improve our CI stability, despite a few spikes during the cycle. We were also able to improve the OVN parity gap with ML2/OVS, and were able to move OVN to supporting WSGI (thanks Rodolfo!) Lack of developers and reviewers continues to be an issue, and we have struggled to reduce our bug backlog significantly. ### Eventlet deprecation ### We have started to identify and document changes needed to deprecate eventlet [1]. We will start creating bugs to track all this work. ### Spring cleanup ### At the beginning of every cycle, we always try and cleanup the code base, for example, removing deprecated and dead code, doc updates, job consolidation. We have made progress on those already. We had a discussion on two items that have been marked experimental in the code - Linuxbridge support and IPv6 prefix delegation - neither of which is well maintained or tested in our CI. I will send a follow-up on next steps for these in the coming week. ### Code base modernization ### As python 3.9 is now the minimal version, we are going through the steps of running pyupgrade/autopep8 on both the neutron and neutron-lib trees to adopt any new conventions. In the process we've been fixing a number of pylint warnings that is still ongoing. The mypy job is now voting, and we have started adding type hints to the code (thanks Ihar!). ### Ironic cross-project ### The high-level discussion point was on Neutron better supporting baremetal (of course). There are some issues with trunk ports as well as external network Single Source of Truth (to help better integrate Neutron with DC networking). An RFE will be proposed for either creating a pre-commit hook to help validate a VNI is okay to use, or to be able to modify the VNI range via the API, as it is currently fixed in the conf file. The other topic was around implementing an additional DHCP backend, as dnsmasq has not been actively maintained, and there was at least one issue Ironic was seeing with it crashing. The consensus was to start supporting Kea, an RFE will be created. See etherpad for further information. ### ML2/OVS future ### As two of the major distros have moved to make OVN their default, there has been less of an emphasis on ML2/OVS development over the past few cycles, and as such, there are an increasing number of bugs in the backlog. There are still a large number of clouds running ML2/OVS as seen in the 2024 User Survey [2], so there is some risk there. As a team we wanted to make sure we communicated this concern, and also wanted to make it known that users doing new deployments, or migrating from older ones (eg Linuxbridge) strongly consider using OVN. We will attempt to give any help we can in this area as migration and deployment issues arise. ### Distributed metadata ### This has been ongoing work from Liu to better support distributed metadata in ML2/OVS, similar to what is done in OVN. Patches have been updated accordingly after discussion. ### Performance/scale testing ### Some recent bugs have shown there are some places where Neutron has DB scaling issues. Any we've found so far have been fixed, but there is a need to do some targeted testing to see if there are others. We also discussed changing our rally testing to try and better detect such issues. My take away was we should be vigilant when any DB change is proposed to make sure it doesn't regress things, and to start looking at places we don't use 'selectin' to see if a simple change would help. ### Gate stability ### Similar to the weekly Neutron CI meeting, we discusses issues in the gate, from things related to the WSGI switch, scenario and fullstack failures. Please see the etherpad for specific bug links. ### RFE discussions ### On-demand topic to discuss any in-progress or proposed RFE changes, please see etherpad for more information. [0] https://etherpad.opendev.org/p/oct2024-ptg-neutron [1] https://etherpad.opendev.org/p/neutron-eventlet-deprecation [2] https://www.openstack.org/analytics

1 0

[ironic/neutron] Using neutron to bind a specific port on an ironic node
by Michael Sherman 08 Nov '24

08 Nov '24

So, my question is: Is there a way for an end-user, launching an instance with nova, to specify which ironic port neutron should use? For context, a subset of our ironic nodes have relatively heterogenous network interfaces, but are configured as being attached to the same physical networks. Examples include nodes with both Broadcom and Mellanox 25g interfaces attached to the same switch, but also nodes with both 1g, 25g, and 100g NICs attached to different switches, but with the same vlans available. In our configuration we’re using Neutron with openvswitch, networking-baremetal, and networking-generic-switch. From initial observations, it seems like if we *don’t* specify, neutron allocates ports in the same order that they’re returned from “baremetal port list”. What we’d like to be able to do is let our users send some API call to bind neutron port X based on some property of the ironic port, ether UUID, tags for speed or vendor, or something else we’re able to define, but it’s not clear to me what information is even available to neutron to use in say, a binding profile? Any pointers would be appreciated. Thanks! -Mike Sherman

1 0

[release] Release countdown for week R-20
by Thierry Carrez 08 Nov '24

08 Nov '24

Development Focus ----------------- The Epoxy-1 milestone is next week, on November 14th! Project team plans for the 2025.1 Epoxy cycle should now be solidified. General Information ------------------- Libraries need to be released at least once per milestone period. Next week, the release team will propose releases for any library which had changes but has not been otherwise released since the 2024.2 Dalmatian release. PTL's or release liaisons, please watch for these and give a +1 to acknowledge them. If there is some reason to hold off on a release, let us know that as well, by posting a -1. If we do not hear anything at all by the end of the week, we will assume things are OK to proceed. NB: If one of your libraries is still releasing 0.x versions, start thinking about when it will be appropriate to do a 1.0 version. The version number does signal the state, real or perceived, of the library, so we strongly encourage going to a full major version once things are in a good and usable state. Upcoming Deadlines & Dates -------------------------- Epoxy-1 milestone: November 14th, 2024 2025.1 Epoxy final release: April 2nd, 2025 -- Thierry Carrez (ttx)

1 0

[kolla][OSA][release] 2024.2 Dalmatian cycle-trailing release deadline
by Elõd Illés 08 Nov '24

08 Nov '24

Hello teams with deliverables following the cycle-trailing release model! This is just a reminder to wrap up your trailing deliverables for 2024.2 Dalmatian. The release deadline is *December 5th, 2024.* [1]. If things are ready sooner than that though, all the better for our downstream consumers. Some of them have been delivered already, so let's list here the ones that still need a release before the deadline: kayobe openstack-ansible-roles openstack-ansible [1] https://releases.openstack.org/epoxy/schedule.html#e-cycle-trail Thanks in advance, Előd Illés irc: elodilles @ #openstack-release

1 0

[all] CFP for OpenInfra Days North America Open for One More week!
by Jeremy Stanley 08 Nov '24

08 Nov '24

For those of you who may not be subscribed to the OpenInfra Foundation mailing list[*], just a heads up that we have another week until the OpenInfra Days North America planning committee will begin reviewing talks, so the CFP deadline has been extended to November 15. If you want to present in Pasadena, California this March, get your proposal in ASAP![**] [*] https://lists.openinfra.dev/archives/list/foundation@lists.openinfra.dev/me… [**] https://forms.gle/81QV3SFV6ixSdsBq8 -- Jeremy Stanley

1 0

[neutron] Drivers meeting canceled Friday, November 8th
by Brian Haley 08 Nov '24

08 Nov '24

Hello Neutrinos: I'm going to cancel tomorrow's drivers meeting as there is no agenda and I will be offline. I will triage incoming ones next week so plan to meet next Friday. Have a nice weekend! -Brian

1 0

[Cinder][Ceph] Volumetria errada no cluster ceph
by lfsilva＠binario.cloud 07 Nov '24

07 Nov '24

Hello, I'm experiencing an issue with the volume usage of the Ceph cluster, which is currently used in OpenStack for volumes. I am working with Ceph version Octopus (15.2.17). When I run the ceph df command, I get this output: --- RAW STORAGE --- CLASS SIZE AVAIL USED RAW USED %RAW USED hdd 120 TiB 33 TiB 87 TiB 87 TiB 72.72 TOTAL 120 TiB 33 TiB 87 TiB 87 TiB 72.72 --- POOLS --- POOL ID PGS STORED OBJECTS USED %USED MAX AVAIL device_health_metrics 1 32 69 MiB 101 137 MiB 0 11 TiB images 2 256 1.3 TiB 166.90k 2.5 TiB 10.72 11 TiB vms 3 64 574 KiB 21 2.5 MiB 0 11 TiB volumes 4 2048 41 TiB 3.94M 82 TiB 79.42 11 TiB backups 5 1024 407 GiB 111.85k 818 GiB 3.63 11 TiB And when I run the rbd du -p volumes command, I get this total: NAME PROVISIONED USED <TOTAL> 14 TiB 8.3 TiB This "volumes" pool is currently set to replica 2, and mirroring is not enabled. I have checked for any locked or deleted snapshots but found none. I also ran a ceph osd pool deep-scrub volumes, but it didn't resolve the issue. Has anyone encountered this problem before? Could someone provide assistance?

3 21

[kolla] Vacation - kevko
by Michal Arbet 07 Nov '24

07 Nov '24

Hello, I would like to inform you that I will be on vacation from November 7th, 2024, to November 23rd, 2024. Thank you, Michal Arbet Michal Arbet Openstack Engineer Ultimum Technologies a.s. Na Poříčí 1047/26, 11000 Praha 1 Czech Republic +420 604 228 897 michal.arbet(a)ultimum.io *https://ultimum.io <https://ultimum.io/>* LinkedIn <https://www.linkedin.com/company/ultimum-technologies> | Twitter <https://twitter.com/ultimumtech> | Facebook <https://www.facebook.com/ultimumtechnologies/timeline>

1 0

Pointers to FOSDEM devrooms CfPs
by Felix Kronlage-Dammers 07 Nov '24

07 Nov '24

Hi, in the last couple weeks the Call for Participations for the various FOSDEM[1] devrooms have been published. In case not everyone knows FOSDEM: FOSDEM is _the_ yearly European free and open source software event where people from around the globe meet and discuss. Aside from the main track FOSDEM consists mainly of many devrooms which focus on certain topics. I wanted to quickly highlight a few of the relevant devrooms for the OpenStack audience and would like to encourage to submit talks :) An overview of the devrooms can be seen here: https://fosdem.org/2025/schedule/ The CfP for the room is linked from the details page of each devroom. Virtualization and Cloud Infrastructure: https://lists.fosdem.org/pipermail/fosdem/2024q4/003574.html Confidential Computing: https://falder.org/fosdem25-cfp Software Defined Storage: https://lists.fosdem.org/pipermail/fosdem/2024q4/003563.html Monitoring and Observability: https://docs.google.com/document/d/1KQmNqzN3cO29ZZUSpd83REvJY_QhbvxIYr4IRBr… The SCS projects organizes, together with the german ZenDiS and the french DINUM, a room with the topic ‘Government Collaboration’. As part of the room we would love to receive proposals that cover: - Standardization and certification of digital sovereign solutions and infrastructure - Automating deployment and operations of such solutions on sovereign infrastructure - Examples and success stories of digital sovereign cloud computing by the public sector The CfP for this room is here: https://lists.fosdem.org/pipermail/fosdem/2024q4/003608.html Am sure there are more relevant devrooms :) An easy overview is also the mailing list archive: https://lists.fosdem.org/pipermail/fosdem/2024q4/thread.html regards felix [1] <url: https://fosdem.org/2025/>

1 0

Question about Package Inclusion and User Creation in Kolla Base Image
by Amir Hossein Ahmadi 06 Nov '24

06 Nov '24

Hello OpenStack Community, I'm currently reading through the OpenStack Kolla source code, and I noticed a few design choices in the kolla/base image that I’d like to understand better: - The kolla/base image includes a number of packages (such as Apache) that don’t appear to be necessary for most containers. - Additionally, user and group accounts for all services are created in the base image, even though they aren’t used by all containers. Could someone explain the reasoning behind including these components in the base image? Is there any documentation available that provides insight into the design approach for Kolla images? Thank you in advance for your help and guidance. Best regards, Amir Hossein, Burna innovation Bridge

1 0

[publiccloud-sig] Reminder: next meeting November 6th (tomorrow) - 0800 UTC
by Felix Kronlage-Dammers 05 Nov '24

05 Nov '24

Hi everyone, This Wednesday, tomorrow, (November 6th) the next meeting of the publiccloud-sig is going to happen. We’re going to do a quick re-cap from PTG as well as the ALASCA Summit that happened last week. Furthermore I’d like to see wether we can spark some activity around the discoverability service that we’ve been thinking and discussing about in the past. We’ll be meeting at 0800 UTC here: https://meetpad.opendev.org/publiccloud-sig <https://meetpad.opendev.org/publiccloud-sig> See also here for all other details: https://wiki.openstack.org/wiki/PublicCloudSIG <https://wiki.openstack.org/wiki/PublicCloudSIG> See you tomorrow! felix -- Felix Kronlage-Dammers Sovereign Cloud Stack — standardized, built and operated by many Ein Projekt der Open Source Business Alliance - Bundesverband für digitale Souveränität e.V. Tel.: +49-30-206539-205 | Matrix: @fkronlage:matrix.org | fkr(a)osb-alliance.com

1 0

debugging eventlet services in vscode.
by Sean Mooney 05 Nov '24

05 Nov '24

hi folks i recently had to debug a problematic interaction between eventlet and a native thread in watcher. while i probably could have printed my way though that eventually i decided to try and get debugging working in vscode instead. in this case i happened to be looking at watcher but this should apply to any eventlet service installed with devstack. in my .vscode directoy i created a launch.json that looks like this. { // Use IntelliSense to learn about possible attributes. // Hover to view descriptions of existing attributes. // For more information, visit: https://go.microsoft.com/fwlink/?linkid=830387 "version": "0.2.0", "configurations": [ { "name": "Python Debugger: watcher-applier", "type": "debugpy", "request": "launch", "program": "/opt/stack/data/venv/bin/watcher-applier", "console": "integratedTerminal", "args": [ "--config-file", "/etc/watcher/watcher.conf" ], "gevent": true, "env" : { "PYTEST_CURRENT_TEST": "true" } }, { "name": "Python Debugger: watcher-decision-engine", "type": "debugpy", "request": "launch", "program": "/opt/stack/data/venv/bin/watcher-decision-engine", "console": "integratedTerminal", "args": [ "--config-file", "/etc/watcher/watcher.conf" ], "gevent": true, "env" : { "PYTEST_CURRENT_TEST": "true" } } ] } there are a few odd things about this config. first the program i am launching is the console script run in the devstack@* systemd service file. you can find what that is using systemctl show devstack@<name> to make this work without conflict do `sudo systemctl stop devstack@<name>` first. after your done debugging you can just start the systemd service again. second when using vscode to view the source of a project select the devstack global virtual env as the project virtual env. third im telling vscode to enable gevent support. now this will raise a warning/error as gevent is not installed but you can ignore that. for those of you not familiar with gevent its a fork of eventlet that actually has much better integration with debuggers due to its wider usage outside of openstack. eventlet and gevent both use greenlet and the eventlet implementation is close enough that when you enable gevent debugging it mostly works. forth to fix the "mostly" part, specifically the fact that dajngo will explode on import, we set "PYTEST_CURRENT_TEST": "true" in the terminal env. PYTEST_CURRENT_TEST is an undocumented environment variable intended only for unit testing that is check by eventlet https://github.com/eventlet/eventlet/blob/8637820f468268ffb0b8504561ea4772d… when we set that to true we skip this `isinstance(o, rlock_type)` check https://github.com/eventlet/eventlet/blob/8637820f468268ffb0b8504561ea4772d… which crashes the debugpy debugger because it tries to load setting form the danjgo module which get loaded into the python interpreter as a side effect of debugpy. i have not tried this on other project yet but it enables me to single step debug, run to breakpoints and many of the normal debug action like using the debug console to inspect the current function scope. all of the above worked without any code change to watcher or running in a specal debug mode in the service code. we occasionally get questions on this list about how to debug openstack services and i usually respond with "its possible to do, is often fragile and need a lot of context to do right" in this case it seam to work reasonable ok in the limited usage i have made of it so far. given the current incantation is pretty simple and non invasive i decided to share this in case others find it useful. keep in mind however most service are distributed systems so if you leave it stopped on a break point for 5 mins while you inspect things, what ever you were originally doing will probably time out and it might fail in odd ways but for local development that may still be fine. please don't try this in production. i have not tried this with a wsgi app. that probably will work fine with the eventlet wsgi web-server via the console scripts i.e. nova-api binary but probably wont work for uwsig/Apache. regards sean

2 1

[cinder][PTG] 2025.1 Epoxy PTG summary
by Jon Bernard 05 Nov '24

05 Nov '24

Hello everyone, thanks for a very productive PTG for the Epoxy cycle. Below is a brief summary of the highlights. The raw etherpad notes are here [1] and both the summary and transcribed etherpad notes are captured in our wiki here [2]. # Retrospective We spent the first few minutes discussing the state the Dalmatian release, what went well, not so well, and how we can improve in the Epoxy release cycle. The most notable point raised is that we have two less cores this cycle. This adds additional strain to an already sizable review queue. We will be looking to cultivate and elevate any qualified contributors if the opportunity arises. # NetApp We covered 6 different topics related to NetApp's feature development plans for the Epoxy cycle in an effort to make sure we're all aligned. Below is a summary of the full set: * Certificate-based authentication Right now only basic auth is implemented, which requires a username and password. There is a strong desire to add support for certificate-based auth. Discussions covered possible approaches and a few helpful links to existing implementations that can be used as a reference. * HA Active-Active support for NVMe driver This will allow users to configure NVMe driver backends in cinder clustered environments. We covered the failover process and which methods needed to be implemented. The related use cases were discussed as well as testing scenarios. We do not have replication and failover coverage in our upstream gate, so special care is needed in testing and verification on the developer side at the moment. Mention of Rally for testing as a possibility. * SVM scoping support for NVMe Desire to add support to the NVMe driver to SVM scoping. At the moment the driver fails to initialize. Work is ongoing to add full support for this feature. Nothing concerning was raised, looking forward to this for Epoxy. * ASA r2 support ASA r2 systems use storage units and consistency groups to simplify storage management and data protection. The REST API is different, and some features are restricted. We'll see how this unfolds in this cycle, good to have this on the radar as we go forward. * Replication - Snapmirror Sync Support Synchronous mirroring achieves a recovery point objective (RPO) of zero lost data by having a copy of important data available if a disaster happens on one of the two storage arrays. The copy is identical to production data at every moment because each time a write is done to the primary volume, a write is done to the secondary volume. The host does not receive an acknowledgment that the write was successful until the secondary volume is successfully updated with the changes that were made on the primary volume. * Metrocluster Support Metro cluster is a clustering solution with synchronous replication to deliver continuous availability, immediately duplicating all of the mission-critical data on a transaction-by-transaction basis. MetroCluster configurations enhance the built-in high availability and non disruptive operations of NetApp hardware and ONTAP storage software, providing an additional layer of protection for the entire storage and host environment. A few changes beyond driver-specific updates were discussed, no blockers. # Handling MYPY Updates We support mypy in our gate jobs, but having the version unpinned causes occasional breakages. We pin hacking and similar tools, and we agreed that mypy should be included. The caveat being we need to remember to periodically unpin and update these version numbers to stay current. # New volume driver LVM+Clone Jan Horstmann proposed a derivative of our LVM driver to add support for using device mapper clone target for transparently migrating volumes. This would all an admin to leverage local storage for performance and still be able to live migrate instances across compute hosts. A POC/Demo is hopefully coming soon, no fundamental blockers were identified, very interested to see how this comes together. # Reviews for backup service It is very difficult to get reviews for cinder-backup related changes. Tobias asked if any cores can focus some time on cinder-backup specifically (an potential backports). We agreed that mention of the backup review queue and review dashboard at the weekly meeting may help raise awareness. # New Location API adoption Rajat proposed a patch that adds the call for the `add_image_location` method which triggers the new location API workflow that is more secure and robust than the old location workflow. This call is made when glance is using cinder as a backend and we want to perform an optimized "upload volume to image" operation. Additional patches are up that add support for testing this feature. We may still need some input from the Glance team, to be followed up on in our weekly meeting. # Adding `service` query parameter to AZ API Tobias proposed an API change, adding a query parameter to an existing API. Specifically, adding `service` query parameter to os-availability-zone API. The patch is thought to be complete and ready for review. # Adding a backup summary API Tobias proposed adding a backup summary API (same way that is available for volumes). He also found a volume summary API bug when implementing this feature. The patch is ready for review. # Horizon Feature Gap We had the privilege to meet with the Horizon team and discuss their efforts on closing existing feature gaps. Several specifics were discussed and clarity was achieved. Concerns were raised over possible incompatibility issues between the openstack client and the SDK. Further, Horizon updates microversion only when necessary and value was identified in reviewing microversions to see if any additional functionality can be leveraged. # Optimize RBD update from Cinder to Glance Currently the path to upload a cinder volume to glance image is not optimized when cinder and glance are both using RBD. We can leverage RBD layering (or copy-on-write cloning) to complete the operation in seconds irrespective of the volume size (since no actual copy is happening). There are 3 features implemented that were needed to support this feature: 1. RBD clone dependency deletion: We have landed changes in glance and cinder that handle the RBD dependency chains so dependency from volumes to images pools shouldn't be an issue 2. Service role: Currently the attachment delete API already uses service role + service token and we can use similar changes for this feature 3. New location APIs: This new glance feature eliminates the security risk as we will be calculating the image hash even in this optimized path (in background so our operation performance won't be affected) * Previously several blockers, appear to be all resolved now * Significant performance increase * Glance -> Cinder already supported * Fallback to unoptimized codepath * Related, recent patches to improve RBD cinder/glance, need to be considered/reviewed Eric has committed to help with the reviews. # State of volume replication Some time was spent discussing the current state of replication. We do not have testing coverage in tempest, so the current state of any particular driver implementation is difficult to know. An idea was raised to poll vendors for use cases and testing results. There is a general lack of clarity for this feature across projects and what our overall disaster recovery solution is. # Image Encryption (Glance Cross-Project) The goal with this effort is to standardize encrypted images between Glance, Cinder and Nova, in an effort to make them consistent and accessible to users. Here is a recap: 1. We merged a patchset in os-brick so far that extends key handling to support more types of Barbican keys for image encryption in addition to Cinder's existing implementation (which strictly uses binascii.hexlify() conversion). 2. Patchset for Glance has been written to add Secret Consumer API from Barbican for encrypted images and to handle the new standardized format. 3. Patchset for Cinder has been written, making it able to produce and use the new standardized format + additionally support qcow2+LUKS in addition to raw LUKS format as well as non-hexlified keys from Barbican. After the initial discussion with Glance, we agreed on *not* introducing any new container_format or disk_format and instead handle this with `os_encrypt_*` properties instead (e.g. os_encrypt_format=luks); this is how the patchsets are currently implemented. However, due to the recent CVEs around image format parsing and conversion, valid concerns were raised that this approach might lead to new vulnerabilities due to potential conflicts and ambiguities between container_format, disk_format and the encryption attributes. We need to agree on a new way of handling the classification of encrypted images in the metadata. We have at least two cases to consider: 1. Raw LUKS (as currently produced by Cinder from encrypted volumes) 2. qcow2+LUKS (as produced by Nova for Ephemeral Storage); both could also be produced by the user (see the gist link in the reference section). # Volume type metadata progress We received a progress update on the volume type metadata effort. Consensus that the direction is sound. A new API endpoint in addition to changes to the response of volume type show / create will require a new microversion. The tempest test still need to be adjusted. Patches need review. # Status of OpenAPI effort Very little traction on initial patches, needs help getting reviews for the first set of patches. Patches also cleanup technical debt and fix several bugs, would be very good to have. The next set of patches is more meaningful, but being held until initial reviews are available. # Removal of uWSGI scripts (stephenfin) The patch to remove uWSGI scripts has been up and needs review. Relatively small patch. Update: this has been reviewed and merged. # Eventlet A dedicated etherpad was created to help document current status. Cinder is still in the investigatory phase. Not only do we use eventlet directly, some drivers such as RBD use C libraries that spawn native threads. We must take care in how we handle these cases. Our goal is only preparatory changes in Epoxy (because it is a SLURP release), and hold big changes for F (which most people won't upgrade to, so will give us more time to detect & fix issues) [1] https://etherpad.opendev.org/p/epoxy-ptg-cinder [2] https://wiki.openstack.org/wiki/CinderEpoxyPTGSummary -- Jon

1 0

[tc][all] OpenStack Technical Committee Weekly Summary and Meeting Agenda (2025.1/R-21)
by Goutham Pacha Ravi 05 Nov '24

05 Nov '24

Hello Stackers, Thanks to all moderators and volunteers for compiling virtual PTG notes and sharing summaries through this mailing list. A lot of exciting work seems to be tracked through the action items from these summaries. I hope you were able to review the TC's PTG summary [1]. Next week is milestone-1 of the 2025.1 "Epoxy" release cycle [2]. This is typically a bug-targeting milestone for project teams. The following is a summary of this past week's happenings within the OpenStack Technical Committee. === Weekly Meeting === During the past week, the OpenStack Technical Committee met over IRC [3]. We followed up on action items from weeks preceding the Project Teams Gathering. We finalized PTL and DPL appointments for the "mistral" and "watcher" project teams. We spent some time reviewing action items related to translating documentation and user strings within OpenStack services. Automated jobs posting translation updates from Zanata are broken due to incompatibility with newer Python versions. Migration to Weblate remains a priority and the i18n team may require more support to complete this migration. We're collaborating over IRC to identify the needs and next steps. The TC also discussed progress on the community goal to migrate integration testing to Python 3.12 and to the latest LTS version of the operating system used in DevStack test jobs. Like me, you may have wondered why we’re combining these efforts. The reason is that it allows us to have a simpler test matrix despite the number of Python versions and OS distributions that OpenStack software is tested with. While we maintain unit test jobs for each supported Python version, it is impractical to duplicate each integration job for every OS distribution available. Therefore, we choose the latest LTS (e.g., Ubuntu 24.04 LTS / "Noble Numbat"), which includes the upper bound of the supported Python versions (Python 3.12). Additionally, each project will have at least one test job for the minimum supported Python version (Python 3.9), so you'll see a test job for the previous LTS release of the OS that will begin in this release and continue running as it transitions to a "stable" release branch. The next meeting of the OpenStack Technical Committee is on 2024-11-05 (today) at 1800 UTC. This is our monthly video meeting too! This meeting will occur simultaneously on Video (Zoom) and IRC (OFTC's #openstack-tc channel). Allison Price (aprice) and Jimmy McArthur (jimmymcarthur) from the OpenStack VMware Migration Working Group [4] will be joining us to discuss the working group's efforts. Meeting information, including other agenda items, is on the meeting wiki [5]. I hope you'll be able to join us. Please remember that you can always propose items to the TC's weekly meeting agenda. === Governance Proposals === ==== Merged ==== Add Watcher DPL for Epoxy | https://review.opendev.org/c/openstack/governance/+/933018 Adding Axel Vanzaghi as PTL for Mistral | https://review.opendev.org/c/openstack/governance/+/927962 ==== Open for Review ==== Propose to select the eventlet-removal community goal | https://review.opendev.org/c/openstack/governance/+/931254 === How to Contact the TC === You can reach the TC in several ways: 1. **Email:** Send an email with the tag [tc] on this mailing list. 2. **IRC:** Use the 'tc-members' keyword on the #openstack-tc IRC channel on OFTC. 3. **Weekly Meeting:** Join us every Tuesday at 1800 UTC [5]. Thank you very much for reading! On behalf of the OpenStack TC, Goutham Pacha Ravi (gouthamr) OpenStack TC Chair [1] 2025.1 "Epoxy" TC PTG Summary: https://lists.openstack.org/archives/list/openstack-discuss@lists.openstack… [2] 2025.1 "Epoxy" Release Schedule: https://releases.openstack.org/epoxy/schedule.html [3] TC Meeting IRC Log 2024-10-29: https://meetings.opendev.org/meetings/tc/2024/tc.2024-10-29-18.00.log.html [4] OpenStack VMware Migration Group: https://www.openstack.org/vmware-migration-to-openstack/ [5] TC Meeting Agenda, 2024-11-05: https://wiki.openstack.org/wiki/Meetings/TechnicalCommittee#Next_Meeting

1 0

Re: GPU PCI passthrough woes.
by Andy Speagle 05 Nov '24

05 Nov '24

Aye, that seems to have done it. Strange that this wasn't necessary for my old 1080 cards. Thanks! On Wed, 2024-10-30 at 08:12 +0000, Alexander Dibbo - STFC UKRI wrote: > ⚠️CAUTION⚠️ > This email originated outside of Toyon's network. Do not download > attachments or click links unless you know the content is safe. > > Hi Andy, > > We've seen many issues with pci passthrough and I suspect that the > l40s will have a seen similar issues with the a100s > > Here is an alias string that works for our a100s: > `alias = {"vendor_id": "10de", "product_id": "20f1", "device_type": > "type-PF", "name": "nvidia-tesla-a100-pcie-vga"}` > > I believe it is the `device_type` that will be your issue > > Regards > > Alexander Dibbo – Cloud Architect / Cloud Operations Group Leader > For STFC Cloud Documentation visit > https://urldefense.us/v3/__https://stfc.atlassian.net/wiki/spaces/CLOUDKB/o… > To raise a support ticket with the cloud team please email > cloud-support(a)stfc.ac.uk > To receive notifications about the service please subscribe to our > mailing list at: > https://urldefense.us/v3/__https://www.jiscmail.ac.uk/cgi-bin/webadmin?A0=S… > To receive fast notifications or to discuss usage of the cloud please > join our Slack: > https://urldefense.us/v3/__https://stfc-cloud.slack.com/__;!!A4KLn-w15EM4BR… > > -----Original Message----- > From: Andy Speagle <aspeagle(a)toyon.com> > Sent: 30 October 2024 02:15 > To: openstack-discuss(a)lists.openstack.org > Subject: GPU PCI passthrough woes. > > Hey Folks, > > I could use a little assistance getting GPU passthrough working. I > had > this working already for one flavor of nvidia gpu... and I've added > some hosts with a much newer gpu... > > I've updated the pci_alias and pci_passthrough variables and those > seem > to be getting set properly in nova.conf > > passthrough_whitelist = [{"vendor_id":"10de", > "product_id":"1b06"},{"vendor_id":"10de", "product_id":"26b9"}] > > alias = {"name": "gpu", "product_id": "1b06", "vendor_id": "10de"} > alias = {"name": "gpu-l40s", "product_id": "26b9", "vendor_id": > "10de"} > > I believe I have all of the iommu stuff configured and have the pci- > stub module entries... dmesg output shows that the GPUs are being > claimed by the stub module. > > $ openstack flavor show t1.small_gpu_l40s > +----------------------------+--------------------------------------+ > > Field | Value | > +----------------------------+--------------------------------------+ > > OS-FLV-DISABLED:disabled | False | > > OS-FLV-EXT-DATA:ephemeral | 0 | > > access_project_ids | None | > > description | None | > > disk | 0 | > > id | af70c94e-0026-4a39-bc1e-dfb93b286a54 | > > name | t1.small_gpu_l40s | > > os-flavor-access:is_public | True | > > properties | pci_passthrough:alias='gpu-l40s:1' | > > ram | 2048 | > > rxtx_factor | 1.0 | > > swap | 0 | > > vcpus | 1 | > +----------------------------+--------------------------------------+ > > Yet... I can't seem to get an instance to run using that new > flavor... > keeps complaining about there not being enough hosts available. > > Fault: > code: 500 > created: 2024-10-30T02:04:33Z > message: "No valid host was found. There are not enough hosts > available." > details: | > Traceback (most recent call last): > File "/usr/lib/python3/dist- > packages/nova/conductor/manager.py", > line 1580, in schedule_and_build_instances > host_lists = self._schedule_instances(context, > request_specs[0], > File "/usr/lib/python3/dist- > packages/nova/conductor/manager.py", > line 940, in _schedule_instances > host_lists = self.query_client.select_destinations( > File "/usr/lib/python3/dist- > packages/nova/scheduler/client/query.py", line 41, in > select_destinations > return self.scheduler_rpcapi.select_destinations(context, > spec_obj, > File "/usr/lib/python3/dist-packages/nova/scheduler/rpcapi.py", > line 160, in select_destinations > return cctxt.call(ctxt, 'select_destinations', **msg_args) > File "/usr/lib/python3/dist- > packages/oslo_messaging/rpc/client.py", line 189, in call > result = self.transport._send( > File "/usr/lib/python3/dist- > packages/oslo_messaging/transport.py", line 123, in _send > return self._driver.send(target, ctxt, message, > File "/usr/lib/python3/dist- > packages/oslo_messaging/_drivers/amqpdriver.py", line 689, in send > return self._send(target, ctxt, message, wait_for_reply, > timeout, > File "/usr/lib/python3/dist- > packages/oslo_messaging/_drivers/amqpdriver.py", line 681, in _send > raise result > nova.exception_Remote.NoValidHost_Remote: No valid host was > found. > There are not enough hosts available. > > Traceback (most recent call last): > File "/usr/lib/python3/dist- > packages/oslo_messaging/rpc/server.py", line 241, in inner > return func(*args, **kwargs) > > File "/usr/lib/python3/dist- > packages/nova/scheduler/manager.py", > line 223, in select_destinations > selections = self._select_destinations( > > File "/usr/lib/python3/dist- > packages/nova/scheduler/manager.py", > line 250, in _select_destinations > selections = self._schedule( > > File "/usr/lib/python3/dist- > packages/nova/scheduler/manager.py", > line 416, in _schedule > self._ensure_sufficient_hosts( > > File "/usr/lib/python3/dist- > packages/nova/scheduler/manager.py", > line 455, in _ensure_sufficient_hosts > raise exception.NoValidHost(reason=reason) > > nova.exception.NoValidHost: No valid host was found. There are > not > enough hosts available. > > Any clues on how/where to dig into this more to see what might be > missing? Thanks. > > -- > Andy Speagle > -- Andy Speagle Sr. Site Reliability Engineer Toyon Research Corporation 316.617.2431

3 2

Is HPET recommended for x86 instance?
by Jiatong Shen 05 Nov '24

05 Nov '24

Hello Community Experts, I found that to enable the HPET timer for x86 guests, the operator should set hw_time_hpet on the given image. By default, HPET is disabled. My question is what are the pros and cons for using HPET? Thanks. -- Best Regards, Jiatong Shen

2 3

[cloudbase-init] This project doesn't seem to be well managed.
by dogyun kim 04 Nov '24

04 Nov '24

Hi, I wanted to contribute to that project, so we created a CR. https://review.opendev.org/c/x/cloudbase-init/+/933629 But no one managed this project, so i can't get review. If you have an administrator, I would like to ask you to confirm it. Thanks

2 1