- openstack-discuss - lists.openstack.org

[nova] NVLink passthrough
by Pavlo Shchelokovskyy 05 Nov '25

05 Nov '25

Hi all, This is a continuing thread from 3 years ago here: https://lists.openstack.org/pipermail/openstack-discuss/2022-June/029273.ht… We've recently faced the same issue re Nvidia NVLink / NVSwitch, and I wanted to share how we are solving it. Given: - server with 8x Nvidia H200 pGPUs aimed to be used as PCI passthrough - separate pGPUs can be connected with NVLink but not any with any - there are 4 specific combinations to use when pairing 2 gpus - gpu1+gpu3, gpu2+gpu4, gpu5+gpu7, gpu6+gpu8 - and there are only 2 possible combinations of pairing 4 gpus - gpu{1..4} and gpu{5..8} The requirement is to have a single flavor that would ask for 2 GPUs, and the scheduling would pick only appropriate pair where NVLink between GPUs is possible, and compute will enable/disable NVLink for those GPUs as appropriate (same for 4 GPUs flavor). The compute part is solved by a libvirt hook provided by Nvidia here https://github.com/NVIDIA/libvirt-hooks/. Given the correct devices are chosen by Nova to be passed into the VM, the hook will enable NVLink between them. However, that scheduling part is currently rather impossible with Nova as-is. The best one can do is assign separate trait to appropriate pairs and quartets of GPUs, and then have 4 flavors for 2-GPU configuration and 2 flavors for the 4-GPU one. This if course is very inefficient and brittle, and can not work in a self-service cloud. What we are trying to do is the following: - add a new flavor extra spec `placement:same_subtree`. The value will be passed directly to placement's allocation candidates API. - in order for this to be usable in a flavor, we need predictable resource group names generated for PCI devices. Right now they are auto-generated from request-id, which won't work. - hence we extended `pci_passthrough:alias` spec to have an optional `:group_prefix` part. Then, what we do is reshape the PCI resource provider tree a bit. We introduce two extra layers of providers, let's call them `group4` and `group2`, each group has a specific trait, for example `CUSTOM_PCI_GROUP_OF_2` and `CUSTOM_PCI_GROUP_OF_4`. Two `group4` providers have compute nodes as a parent, and are parent to two `group2` providers each. Then we re-parent PCI resource providers to `group2` providers, so that each `group2` and `group4` tree corresponds to a set of GPUs that can be connected via NVLink. Luckily enough, resource tracker of nova-compute does not care about the parent provider, as long as the name and ID of PCI resource providers are not changed, so the reshape has no effect on the resource tracker. We also need to use all this with Aggregate Extra Spec filter active, so we've duplicated the `group_policy` flavor extra spec to `placement:group_policy` one. And of course, for all of this to work, one has to enable "PCI in placement" feature too https://docs.openstack.org/nova/2025.2/admin/pci-passthrough.html#pci-track… As a result, assuming the PCI alias for the GPUs is `h200` and its resource class is `PGPU`, we can create the following flavor extra spec: pci_passthrough:alias=h200:2:_nvlink2 trait_groupof2:CUSTOM_PCI_GROUP_OF_2=required placement:group_policy=none placement:same_subtree=_nvlink2-0,_nvlink2-1,_groupof2 that will result in the following placement query: GET /allocation_candidates ?resources_nvlink2-0=PGPU:1 &resources_nvlink2-1=PGPU:1 &required_groupof2=CUSTOM_PCI_GROUP_OF_2 &group_policy=none &same_subtree=_nvlink2-0,_nvlink2-1,_groupof2 That will select only a pair of GPUs that are children of a single provider with `CUSTOM_PCI_GROUP_OF_2` trait - which is exactly what we need! You can find this work proposed as a patch at https://review.opendev.org/966205. This is definitely just a beginning of discussion, and most certainly would require a spec, so I am eager to hear your thoughts and comments :-) Cheers, pas-ha -- Dr. Pavlo Shchelokovskyy Principal Software Engineer Mirantis Inc www.mirantis.com

2 1

[horizon][PTG] 2026.1 Gazpacho PTG summary
by Tatiana Ovchinnikova 05 Nov '25

05 Nov '25

Hello everyone, A huge thank you to everyone who joined the OpenStack Horizon team for the Gazpacho PTG, who organized sessions, presented topics, and actively contributed to the discussions! Thank you for making the PTG a success! The full list of sessions and topics can be found on our Etherpad [1] Here is the summary of the topics: # Flamingo Retrospective A review of the previous Flamingo cycle, highlighting positive developments like switching to SDK for certain services, improving TOTP setup, and improving region/domain selection. # Gazpacho Cycle Planning Planning for the current Gazpacho cycle, establishing AngularJS removal as the top priority. Other key goals include investigating the removal of old integration tests and planning for Django 5.2 support. # Open Discussion A session for discussing various in-progress code reviews. Topics included a discussion on a feature to add changing instance password support. # Django 5.2 Support Discussion on the plan to adopt Django 5.2, the latest LTS release. # Tests Review of current test stability and plans for test code cleanup. The team noted that both horizon-integration-pytest and horizon-ui-pytest are currently stable and plans to remove old integration tests from the repository are moving forward. We also discussed progress on implementing Manila UI integration tests as a template for other plugin teams. # Towards New OpenStack contributors: An OpenInfra User Group Kenya Story A presentation by Agnes Nduta to share the experiences and strategies of the OpenInfra User Group Kenya in successfully onboarding new contributors into the OpenStack ecosystem. # Contributor/Maintainer Surveys and Metrics Analysis A review of current project health using metrics and analysis from contributor and maintainer surveys. # How to create a Cursor MCP for analyzing Horizon code A walk-through demonstrating the steps required to set up a Cursor Maintainer Candidate Program (MCP) for analyzing Horizon code. # Multi-Realm Support and Multi-Region Support discussion A dedicated cross project session to discuss the implementation and implications of providing Multi-Realm and Multi-Region support within the Horizon dashboard. More details can be found on the separate Etherpad [2]. [1] https://etherpad.opendev.org/p/horizon-gazpacho-ptg [2] https://etherpad.opendev.org/p/multi-realm With kind regards, Tatiana

1 0

Nova error for fetch or server create after upgrade
by Ahmad Milad Pour 05 Nov '25

05 Nov '25

Hello there, I did upgrade the packages and repo, and something were upgraded: BEFORE: root@gregion83:~# apt-cache policy nova-api | head -n 10 nova-api: Installed: 3:29.2.0-0ubuntu1.1 AFTER: root@gregion83:~# apt-cache policy nova-api | head nova-api: Installed: 3:31.0.0-0ubuntu1.2~cloud0 I also did add "DEBUG=True" under "default" section of /etc/nova/nova.conf in my region and tried to create server, but I cannot find any important logs or details to see what's wrong: ==> /var/log/nova/nova-api.log <== 2025-11-05 14:42:44.524 2960 INFO nova.osapi_compute.wsgi.server [-] 185.137.25.83 "GET /v2.1 HTTP/1.1" status: 200 len: 790 time: 0.0164449 2025-11-05 14:42:45.615 2960 ERROR nova.api.openstack.wsgi [None req-892920aa-c4be-4aea-abab-aa8e9fe8c7e0 64c74c80ec9a4a31ae27b82af846b36a 906db1c682bc4e3db 822c2e6dbc9cb6a - - default default] Unexpected exception in API method: neutronclient.common.exceptions.InternalServerError: Request Failed: internal serve r error while processing your request. Neutron server returns request_ids: ['req-80f3ccda-11ee-4417-a27c-2437540e6ee0'] 2025-11-05 14:42:45.615 2960 ERROR nova.api.openstack.wsgi exception_handler_v20(status_code, error_body) 2025-11-05 14:42:45.615 2960 ERROR nova.api.openstack.wsgi File "/usr/lib/python3/dist-packages/neutronclient/v2_0/client.py", line 90, in exception_handler_v20 2025-11-05 14:42:45.615 2960 ERROR nova.api.openstack.wsgi raise client_exc(message=error_message, 2025-11-05 14:42:45.615 2960 ERROR nova.api.openstack.wsgi neutronclient.common.exceptions.InternalServerError: Request Failed: internal server error while processing your request. 2025-11-05 14:42:45.615 2960 ERROR nova.api.openstack.wsgi Neutron server returns request_ids: ['req-80f3ccda-11ee-4417-a27c-2437540e6ee0'] 2025-11-05 14:42:45.615 2960 ERROR nova.api.openstack.wsgi 2025-11-05 14:42:45.628 2960 INFO nova.api.openstack.wsgi [None req-892920aa-c4be-4aea-abab-aa8e9fe8c7e0 64c74c80ec9a4a31ae27b82af846b36a 906db1c682bc4e3db822c2e6dbc9cb6a - - default default] HTTP exception thrown: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible. <class 'neutronclient.common.exceptions.InternalServerError'> 2025-11-05 14:42:45.632 2960 INFO nova.osapi_compute.wsgi.server [None req-892920aa-c4be-4aea-abab-aa8e9fe8c7e0 64c74c80ec9a4a31ae27b82af846b36a 906db1c682bc4e3db822c2e6dbc9cb6a - - default default] 185.137.25.83 "GET /v2.1/servers/c18f669d-1338-42dd-947f-c40217238450 HTTP/1.1" status: 500 len: 660 time: 1.1018338 My last openstack version was 6.6.0, and I did upgrade after another email I sent "How to install Flamingo openstack in Ubuntu 24". Before upgrade, everything was working fine, but now I have issue. Would you please help me for this? Regards, Ahmad

2 3

How to install Flamingo openstack in Ubuntu 24
by Ahmad Milad Pour 05 Nov '25

05 Nov '25

Hello there, I use Ubuntu 24 and I did try to install openstack, but it seems it's not the latest one. root@gregion83:~# cat /etc/os-release PRETTY_NAME="Ubuntu 24.04.3 LTS" NAME="Ubuntu" VERSION_ID="24.04" VERSION="24.04.3 LTS (Noble Numbat)" VERSION_CODENAME=noble ID=ubuntu ID_LIKE=debian HOME_URL="https://www.ubuntu.com/" SUPPORT_URL="https://help.ubuntu.com/" BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" PRIVACY_POLICY_URL=" https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" UBUNTU_CODENAME=noble LOGO=ubuntu-logo root@gregion83:~# openstack --version openstack 6.6.0 root@gregion83:~# apt-cache policy nova-api | head -n 10 nova-api: Installed: 3:29.2.0-0ubuntu1.1 Candidate: 3:29.2.0-0ubuntu1.2 Version table: 3:29.2.0-0ubuntu1.2 500 500 http://archive.ubuntu.com/ubuntu noble-updates/main amd64 Packages *** 3:29.2.0-0ubuntu1.1 100 100 /var/lib/dpkg/status 3:29.0.1-0ubuntu1.4 500 500 http://security.ubuntu.com/ubuntu noble-security/main amd64 Packages root@gregion83:~# apt-cache policy openstack-dashboard | head -n 10 openstack-dashboard: Installed: 4:24.0.1-0ubuntu0.24.04.1 Candidate: 4:24.0.1-0ubuntu0.24.04.1 Version table: *** 4:24.0.1-0ubuntu0.24.04.1 500 500 http://archive.ubuntu.com/ubuntu noble-updates/main amd64 Packages 100 /var/lib/dpkg/status 4:24.0.0-0ubuntu1 500 500 http://archive.ubuntu.com/ubuntu noble/main amd64 Packages root@gregion83:~# curl -s -H "X-Auth-Token: $(openstack token issue -c id -f value)" http://$HOSTNAME:8774 | jq { "versions": [ { "id": "v2.0", "status": "SUPPORTED", "version": "", "min_version": "", "updated": "2011-01-21T11:33:21Z", "links": [ { "rel": "self", "href": "http://gregion83.domain.com:8774/v2/" } ] }, { "id": "v2.1", "status": "CURRENT", "version": "2.96", "min_version": "2.1", "updated": "2013-07-23T11:33:21Z", "links": [ { "rel": "self", "href": "http://gregion83.com:8774/v2.1/" } ] } ] } That's the information I thought is good for the help. And that's the error I have: root@gregion83:~# openstack server rebuild --image 608349cf-8743-4339-8e2c-e707f89be64f --no-user-data c18f669d-1338-42dd-947f-c40217238450 --os-compute-api-version 2.54 or greater is required to support the --no-user-data option root@gregion83:~# openstack server rebuild --image 608349cf-8743-4339-8e2c-e707f89be64f --user-data /dev/null c18f669d-1338-42dd-947f-c40217238450 --os-compute-api-version 2.54 or greater is required to support the --user-data option I see this link from which I can install: https://docs.openstack.org/horizon/2025.2/install/install-ubuntu.html How to resolve this issue? Regards, Ahmad

3 4

[kolla-ansible] Upgrade from 2024.1 to 2024.1 neutron-l3-agent unhealthy
by hamid.lotfi＠gmail.com 04 Nov '25

04 Nov '25

Hi everyone, I’m hitting a weird issue after a kolla-ansible upgrade and would appreciate any pointers. Environment kolla-ansible 18.8.1 rabbitmq 4.1.3 (deployed by Kolla) upgrade path: 2024.1 → 2025.1 What happened Ran kolla-ansible -i … upgrade from 2024.1 to 2025.1. The upgrade finished without obvious errors in kolla logs. Right after the upgrade every neutron-l3-agent container on all control/compute nodes is marked unhealthy (no ERROR/FATAL lines in the container logs). All tenant instances lose external connectivity. Simply docker restart neutron_l3_agent on each node is enough: containers become healthy and external access is restored instantly. No other OpenStack service shows the same behaviour. Any idea why the agents start in “unhealthy” state until manually restarted? Happy to collect more data if you tell me what to look for.

1 0

[ops][largescale-sig][scientific-sig] Ops Radio Hour on October 31, 2025 at the PTG
by Ildiko Vancsa 04 Nov '25

04 Nov '25

Hi OpenStack community, I'm reaching out to you with a friendly reminder that the next OpenStack Ops Radio Hour[1] is scheduled to happen on __October 31st, 2025 at 1300UTC,__ which is during the PTG! The call series is a casual setting for operators, engineers, and contributors to come together, share knowledge, and collaborate on the complexities of managing and deploying OpenStack environments. The main topic of the next gathering will be to discuss how to refactor Ops-related activities. See Thierry's email for more details: https://lists.openstack.org/archives/list/openstack-discuss@lists.openstack… If you have further topics you would like to dive into during the call, please add them to this etherpad: https://etherpad.opendev.org/p/openstack_ops_radiohour A few reminders: - The group on the call agreed to try out Matrix as a chat platform for more synchronous communication outside calls + You can access the Matrix room here: https://matrix.to/#/#openstack-ops:opendev.org<https://matrix.to/#/%23openstack-ops:opendev.org> + Further details are here: https://lists.openstack.org/archives/list/openstack-discuss@lists.openstack… - If you would like to read and catch up on notes from previous calls, you can find them in this etherpad: https://etherpad.opendev.org/p/openstack_ops_radiohour See you on the call on October 31st! Bets Regards, Ildikó [1] https://wiki.openstack.org/wiki/Operations/Meetups

1 1

[horizon] Cancelling weekly meeting tomorrow
by Tatiana Ovchinnikova 04 Nov '25

04 Nov '25

Hello team, I don't have anything to share after the PTG, so I'm cancelling the IRC meeting tomorrow, November 5th. Let's save some time for the reviews and see you next week! With kind regards, Tatiana

1 0

[ironic][all] Making core teams more accessible
by Jay Faulkner 04 Nov '25

04 Nov '25

I'm going to preface this with a comment that I'm not completely sold on this idea yet, but I think it's worth a larger discussion and might be a seed to help with discussion. I've tagged Ironic as this proposal is for Ironic specifically, but I'm very, very interested in the larger community opinion for this kind of approach so I also tagged all. One of the toughest parts about walking the path to becoming core is how fuzzy the targets are. With Ironic recently having added a two-tier review system, with reviewers and approvers separate, we have an opportunity to allow a much, much bigger tent for the ironic-reviewers group without risking code getting merged without approval by one of our folks with more experience and context in OpenStack and Ironic as a whole. So here's the suggestion -- we come up with a static criteria and period with a clear bar to "if you do this, you are an ironic-reviewer". For instance, we could say (I'm not sure about these numbers): if you do an average of Y (5?) reviews/week and have merged a major Ironic feature or multiple bugfix patches, you would automatically be a reviewer. Obviously, we'd have to keep safeguards to prevent abuse; in our community in the past we had a rule that changes required review from two separate corporate entities -- in this case, I'd suggest a waiting period to prevent quick-merging by someone with a single perspective. Exceptions would be, as in the current system, for emergency CI fixes and documentation improvements. To be clear: the safeguards shouldn't be seen as a lack of trust, but maybe instead as an acknowledgement that structural controls hold value even if there's no reason to assume mistrust. Full proposal: * ironic-approvers (+2 and Workflow +1) continues to operate as usual, with admission being based on technical merit, trust, and review frequency. * ironic-reviewers (+2 only) becomes a data-driven group, where reviewers are added/removed based on individual performance metrics TBD (likely reviews/commits/bugs fixed). -- I'll note that today, there's not really any set of reasonable metrics that would add additional reviewers to this group. * Code changes would either need a review from two separate company-representatives, or to have sat for a short waiting period to ensure other folks had a chance to look at it. * Urgent CI fixes, trivial changes, and documentation updates would still only need one review to land. What do folks think? Would this be a good carrot for existing non-cores to up their review and upstream contribution commitments? - Jay Faulkner G-Research OSS

6 6

[nova] Patch for Calico with libvirt >= 9.5.0
by Nell Jerram 04 Nov '25

04 Nov '25

I apologise if this is the wrong place to ask, but is there something I can do to encourage review and approval for https://review.opendev.org/c/openstack/nova/+/960284 ? Many thanks - Nell

2 2

[manila] Cancelling Nov 6th weekly meeting
by Carlos Silva 04 Nov '25

04 Nov '25

Hello Zorillas and interested stackers, As discussed in the previous upstream weekly meeting, we're cancelling this week's meeting, so we have some extra time to go through PTG action items before we regroup. The next meeting will be on Nov 13th. Thank you, carloss

1 0