October 2024 - openstack-discuss - lists.openstack.org

GPU PCI passthrough woes.
by Andy Speagle 30 Oct '24

30 Oct '24

Hey Folks, I could use a little assistance getting GPU passthrough working. I had this working already for one flavor of nvidia gpu... and I've added some hosts with a much newer gpu... I've updated the pci_alias and pci_passthrough variables and those seem to be getting set properly in nova.conf passthrough_whitelist = [{"vendor_id":"10de", "product_id":"1b06"},{"vendor_id":"10de", "product_id":"26b9"}] alias = {"name": "gpu", "product_id": "1b06", "vendor_id": "10de"} alias = {"name": "gpu-l40s", "product_id": "26b9", "vendor_id": "10de"} I believe I have all of the iommu stuff configured and have the pci- stub module entries... dmesg output shows that the GPUs are being claimed by the stub module. $ openstack flavor show t1.small_gpu_l40s +----------------------------+--------------------------------------+ | Field | Value | +----------------------------+--------------------------------------+ | OS-FLV-DISABLED:disabled | False | | OS-FLV-EXT-DATA:ephemeral | 0 | | access_project_ids | None | | description | None | | disk | 0 | | id | af70c94e-0026-4a39-bc1e-dfb93b286a54 | | name | t1.small_gpu_l40s | | os-flavor-access:is_public | True | | properties | pci_passthrough:alias='gpu-l40s:1' | | ram | 2048 | | rxtx_factor | 1.0 | | swap | 0 | | vcpus | 1 | +----------------------------+--------------------------------------+ Yet... I can't seem to get an instance to run using that new flavor... keeps complaining about there not being enough hosts available. Fault: code: 500 created: 2024-10-30T02:04:33Z message: "No valid host was found. There are not enough hosts available." details: | Traceback (most recent call last): File "/usr/lib/python3/dist-packages/nova/conductor/manager.py", line 1580, in schedule_and_build_instances host_lists = self._schedule_instances(context, request_specs[0], File "/usr/lib/python3/dist-packages/nova/conductor/manager.py", line 940, in _schedule_instances host_lists = self.query_client.select_destinations( File "/usr/lib/python3/dist- packages/nova/scheduler/client/query.py", line 41, in select_destinations return self.scheduler_rpcapi.select_destinations(context, spec_obj, File "/usr/lib/python3/dist-packages/nova/scheduler/rpcapi.py", line 160, in select_destinations return cctxt.call(ctxt, 'select_destinations', **msg_args) File "/usr/lib/python3/dist- packages/oslo_messaging/rpc/client.py", line 189, in call result = self.transport._send( File "/usr/lib/python3/dist- packages/oslo_messaging/transport.py", line 123, in _send return self._driver.send(target, ctxt, message, File "/usr/lib/python3/dist- packages/oslo_messaging/_drivers/amqpdriver.py", line 689, in send return self._send(target, ctxt, message, wait_for_reply, timeout, File "/usr/lib/python3/dist- packages/oslo_messaging/_drivers/amqpdriver.py", line 681, in _send raise result nova.exception_Remote.NoValidHost_Remote: No valid host was found. There are not enough hosts available. Traceback (most recent call last): File "/usr/lib/python3/dist- packages/oslo_messaging/rpc/server.py", line 241, in inner return func(*args, **kwargs) File "/usr/lib/python3/dist-packages/nova/scheduler/manager.py", line 223, in select_destinations selections = self._select_destinations( File "/usr/lib/python3/dist-packages/nova/scheduler/manager.py", line 250, in _select_destinations selections = self._schedule( File "/usr/lib/python3/dist-packages/nova/scheduler/manager.py", line 416, in _schedule self._ensure_sufficient_hosts( File "/usr/lib/python3/dist-packages/nova/scheduler/manager.py", line 455, in _ensure_sufficient_hosts raise exception.NoValidHost(reason=reason) nova.exception.NoValidHost: No valid host was found. There are not enough hosts available. Any clues on how/where to dig into this more to see what might be missing? Thanks. -- Andy Speagle

2 1

[Horizon] Cancelling weekly IRC meeting tomorrow
by Tatiana Ovchinnikova 30 Oct '24

30 Oct '24

Hello team, Thank you for joining the vPTG last week! Since there's a week of chaos because of the time change, and part of the team is on PTO, I'm canceling our weekly meeting tomorrow. See you all next Wednesday! With kind regards, Tatiana

1 0

[i18n][ptg] I18n SIG vPTG summary
by Ian Y. Choi 30 Oct '24

30 Oct '24

Hi all, I would like to share with you a summary of the recent discussions held during the 2025.1 Epoxy PTG regarding the Internationalization (I18n) Special Interest Group (SIG). During the period, the I18n SIG explored approaches to improve multilingual support, discussing Weblate migration, AI-based translation implementation, sustainable documentation translation strategies, and addressing current translation tool issues. Special thank you to TC and Nova team for elaborate discussions. Please refer to the below brief summary from Etherpad: https://etherpad.opendev.org/p/oct2024-ptg-i18n . 1. Overall Translation Status discussion - Clarifying that StringFreezes applies to Horizon and dashboard plugin projects [1] - Confusion exists about the current state of translations. Horizon translations were added months ago but not proposed by the tooling. : No translation proposals on the master branch after April 19, 2024. [2] : Horizon removed Python 3.8 support [3], while translation jobs depend on Ubuntu Bionic. 2. Towards sustainable Documentation Translation - Discussion on utilizing AI machine translation: : Proposal to use Weblate's machine translation feature. : Consideration of open-source tools like LibreTranslate [4]. : Concerns raised about legal issues. - Emphasis on the necessity of human reviewers: : Suggestion to create a 'language-moderator' group in Gerrit? : Need to add voting labels for translation patches. 3. Weblate Migration - Ongoing migration of translated project files from Zanata to Weblate. - Integration with Zuul is the second-phase goal. - Difficulties encountered due to API usage issues. 4. Weblate Issues - There was an authentication issue with Weblate and openinfraid (note: but it appears to be resolved now. Thank you @rosmaita!) 5. AI Translation Approach - Step-by-step ways a. Model selection b. Community review of the model c. Model utilization d. Model + RAG (Reviewed/Translated OpenStack documents, glossary) utilization - Plan to conduct PoC with Nova RST documents. 6. Action items - To start a 3-month PoC to test AI translation with Nova RST files (ianychoi) - Zuul investigation for Weblate migration (seongsoocho) - Horizon Zuul job issue check (discussion with other teams) Thank you all, /Ian [1] https://review.opendev.org/c/openstack/releases/+/930929 [2] https://review.opendev.org/q/owner:proposal-bot+project:openstack/horizon+b… [3] https://docs.openstack.org/releasenotes/horizon/2024.2.html#upgrade-notes [4] https://github.com/LibreTranslate/LibreTranslate

1 0

[all[qa] Tempest dropping python 3.8 support
by Ghanshyam Mann 30 Oct '24

30 Oct '24

Hello Everyone, As you know, services and Oslo libs are dropping Python 3.8 in this cycle. Tempest is branchless, and we use Tempest master to test all supported stable branches (currently 2023.1 to 2024.2). Because Python 3.8 is still supported in stable/2023.1 to stable/2024.1 Initially, we thought of continuing the py 3.8 support in Tempest, but without Oslo libs, Tempest cannot support it. One way is to keep the python 3.8 constraints separately (either a separate file in Tempest or separate constraints for python3.8 in requirements), which stops Tempest from adding any new features from oslo/deps. Another way (this is what we did in the previous Python version bump) is to drop the Python 3.8 support, and if anyone wants to test the cloud with Python 3.8, they can install and run Tempest on Python >=3.9 virtual env. In upstream testing, we also install and run Tempest from the virtual environment. or use the Tempest 41.0.0 (the last release which supports py3.8) to test python 3.8 cloud. This is what we did during py3.6/3.7 support removal. - https://review.opendev.org/c/openstack/tempest/+/933398 -gmann

1 1

Re: [Cinder][Ceph] Volumetria errada no cluster ceph
by Luiz Fernando Silva 30 Oct '24

30 Oct '24

I am sending the file with the ceph osd df tree and ceph pg ls-by-pool volumes output as an attachment. ceph -s: cluster: id: ccdcc86e-c0e7-49ba-b2cc-f8c162a42f91 health: HEALTH_OK services: mon: 3 daemons, quorum hc-node02,hc-node04,hc-node03 (age 5w) mgr: hc-node02(active, since 4M) osd: 54 osds: 54 up (since 14h), 54 in (since 3M) task status: data: pools: 5 pools, 3424 pgs objects: 4.25M objects, 43 TiB usage: 88 TiB used, 32 TiB / 120 TiB avail pgs: 3424 active+clean io: client: 16 MiB/s rd, 5.7 MiB/s wr, 856 op/s rd, 483 op/s wr Thank you very much, we will adjust to replica 3.

1 0

Re: [barbican][vault]
by Andy Speagle 30 Oct '24

30 Oct '24

On Thu, 2024-10-24 at 18:06 -0700, melanie witt wrote: > On 10/24/24 16:43, Andy Speagle wrote: > > Hey Folks, > > > > I'm having some trouble getting barbican to generate and/or store > > keys > > in our hashicorp vault in our charmed openstack environment. I have > > issues with both TPM and SSL related keys. > > > > I get extremely generic errors back that I can't seem to figure out > > how > > to troubleshoot: > > > > 'Traceback (most recent call last):\n File "/usr/lib/python3/dist- > > packages/nova/conductor/manager.py", line 688, in build_instances\n > > scheduler_utils.populate_retry(\n File "/usr/lib/python3/dist- > > packages/nova/scheduler/utils.py", line 998, in populate_retry\n > > raise > > exception.MaxRetriesExceeded(reason=msg)\nnova.exception.MaxRetries > > Exce > > eded: Exceeded maximum number of retries. Exceeded max scheduling > > attempts 3 for instance 0c50b2d7-8ac5-41a2-90ab-1e33f7aa1c22. Last > > exception: Key manager error: Internal Server Error: Secret > > creation > > failure seen - please contact site administrator.\n' > > Just FYI this is the generic MaxRetriesExceeded error which means > that > the server action failed on nova-compute(s). So you'll want to look > in > your nova-compute.log for the error that triggered the retries and > eventual MaxRetriesExceeded. > > > I'm using barbican 14.0.2 in yoga/stable ... anyone have an ideas > > how > > to troubleshoot this? > > I'm guessing in nova-compute.log you will see an error 403 forbidden > for > creating a barbican secret. If so, the most common reason I have seen > for this is if you have in your barbican.conf > [oslo.policy]enforce_new_defaults = False, then secret creates > require > the requester to have the 'creator' role assigned to them (you can do > this by 'openstack role add <options> <role>' [1]). > > When [oslo.policy]enforce_new_defaults = True, the 'creator' role is > not > required. Yup... nailed it. I really appreciate the insight. :) > The tl;dr is 'enforce_new_defaults' means to use the newer secure > RBAC > policies and the secure RBAC changes are shown in the Wallaby release > notes:

1 0

[neutron] Moving CI Meeting to Monday 1500 UTC
by Yatin Karel 29 Oct '24

29 Oct '24

Hello Neutron Team !! As discussed/agreed over the last Neutron CI meeting, pushed the changes[1][2] for the new timing. [1] https://review.opendev.org/c/opendev/irc-meetings/+/933637 [2] https://etherpad.opendev.org/p/neutron-ci-meetings -- Thanks and Regards Yatin Karel

1 0

[qa][ptg] QA 2025.1 vPTG Summary
by Martin Kopec 29 Oct '24

29 Oct '24

Hi everyone, here is a summary of the discussions. The full agenda can be found here [1]. == Dalmatian (2024.2) Retrospective == At the beginning of the cycle a devstack optimization change was merged [2] that uses the OSC service. If you haven't already, give it a try and enable it in your jobs. If it doesn't work for your project, let us know. == Migrating upstream CI/CD jobs to Ubuntu Noble (24.04) == Details about the migration can be found in [3] and [4] together with the testing status of all projects. We have time until Nov 29th to test the newer Ubuntu version and deal with any issues, after that, QA would like to proceed with the migration. == Triage previous and write new priority items == The list of the priority items planned for this cycle as well as the ones which were carried over from the previous cycles, can be found here [5]. == Open Discussion == There have been issues with a few jobs and tests that made an assumption of having the amount of certain resources, such as CPUs, guaranteed. However, that's not the case and the amount of the resources is not guaranteed in the upstream CI. No tests or jobs should make this assumption. Apart from CPUs, the same applies on floating ips, public addresses etc. [1] https://etherpad.opendev.org/p/oct2024-ptg-os-qa [2] https://review.opendev.org/c/openstack/devstack/+/676016 [3] https://lists.openstack.org/archives/list/openstack-discuss@lists.openstack… [4] https://etherpad.opendev.org/p/migrate-to-noble [5] https://etherpad.opendev.org/p/qa-epoxy-priority Thank you all for attending the PTG and contributing to the discussion, -- Martin Kopec Red Hat EMEA IM: kopecmartin

1 0

[nova][ptg] 2025.1 Epoxy PTG summary
by Sylvain Bauza 29 Oct '24

29 Oct '24

(resending the email as the previous one was blocked to an attached etherpad backup txtfile larger than the max size) Hey all, First, thanks for having joined us if you were in the vPTG. We had 15-20 people every day for our nova sessions, I was definitely happy to see new folks :-) If you want to see our PTG etherpad, please rather look at https://etherpad.opendev.org/p/r.4f297ee4698e02c16c4007f7ee76b7c1 instead of the main nova etherpad as I don't want that the etherpad would have a wrong traduction or having some paragraphs to be removed. As I say every cycle, just take a coffee (or a tea) now as the summary will be large. ### Dalmatian retrospective and Epoxy planning ### 6 of 15 approved blueprints were eventually implemented. We also merged more than 31 bugfixes during Dalmatian. We agreed to be explaining on the IRC channel when we have meetings for discussing some feature series (like the one we did every week for the manila/virtiofs series) and providing some public invitations. We could do this again this cycle for other features, we'll see. We will also try to have a periodic integration-compute job that pulls OSC and SDK from master. Our Epoxy deadlines will be : two spec review days (R-16, R-2), a soft spec approval freeze by R-16 and then hard spec approval freeze by R-12. That means that contributors really need to provide their specs before mid-December. Bauzas (me) will add these deadlines into the Epoxy schedule : https://releases.openstack.org/epoxy/schedule.html ### vTPM live migration ### We agreed on the fact that a vTPM live-migration feature is a priority for Epoxy given Windows 11. artom will create a spec proposing an image metadata property saying 'do I want to share my secret with nova service user ?' and also providing a new `nova-manage image_property set migratable_something` command so operators could migrate the existing instances for getting the Barbican secrets, if really the operators wants. ### Unified limits wrap-up ### We already have two changes needing to be merged before we can modify the default quota driver (in order to default to use unified limits). We agreed on reviewing both patches (one for treating unset limits as unlimited, the other about adding a nova-manage command for automatically creating nova limits) but we also discussed about a latter patch that would eventually say which nova resources need to be eventually set (so we *have to* enforce them anyway). melwitt agreed on working on that latter patch. ### per-process health checks ### We already had one series and we discussed it again. Gibi agreed on taking over it and he will re-propose the existing spec as it is. We also discussed the first checks we would have, like RPC failures and DB connection issues, we'll review those when they are in Gerrit. ### sustainable computing (a.k.a. power mgmt) ### When someone (I won't say who [1]) implemented power management in Antelope, this was nice but we eventually found a long list of bugs that we fixed. Since we don't really want to reproduce that experience, we had this kind of post-mortem where we eventually agreed on two things that could avoid reproducing that problem : a weekly periodic job will run whitebox tempest plugins [2] with nova-compute restarts also covered by a whitebox tempest plugin. Nobody is committed against those two actions but we hope to identify someone soon. As a side note, gibi mentioned RAPL MSR support [3], notifying us that we would have to support that in a later release (as the libvirt implementation is not merged yet) ### nvidia's vGPU vfio-pci variant driver support ### Long story short, since the linux kernel removed some feature in release 5.18 (IOMMU backend support for vfio-mdev) this impacted the nvidia driver which now detects that and then creates vfio-pci devices instead of vfio-mdev devices (as vGPUs). This has a dramatic impact on Nova as we relied on the vfio-mdev framework for abstracting virtual GPUs. By the next release, Nova will need to inventorize the GPUs by rather looking at SRIOV virtual functions which are specific to the nvidia driver (we call them vfio-pci variant driver resources). The nova PTG session focused on the required efforts to do so. We agreed on the fact it will require operators to propose different flavors for vGPU where they would require distinct resource classes (all but VGPU). Fortunately, we'll reuse existing device_spec PCI config options [4] where the operator would define custom resource classes which would match the PCI addresses of the nvidia-generated virtual functions (don't freak out, we'll also write documentation). We'll create another device type (something like type-VF-migratable) for describing such specific nvidia VFs. Accordingly the generated domain XML will correctly write the device description (amending the "managed=no" flag for that device). There will be an upgrade impact: existing instances will need to be resized to that new flavor (or instances will need to be shelved, updated for changing the embedded flavor and unshelved). In order to be on par with existing vGPU features, we'll also need to implement vfio-pci live-migration by detecting the VF type on the existing SRIOV live-migration. Since that effort is quite large, bauzas will incept a subteam of interested parties that would help him implement all of those bits in the short timeframe that is one upstream cycle. ### Graceful shutdowns ### A common pitfall that was told by tobian-urdin is when you want to stop nova-compute services. In general, before stopping the service, we should be sure that all RPC calls are done, which means we would no longer accept RPC calls after asking to stop the nova-compute and just awaiting the current calls to be done before stopping the service. For that, we need to create a backlog spec for discussing that design and we would also need to modify oslo.service for unsubscribing the RPC topics. Unfortunately, this cycle we won't have any contributor for working on it, but gibi could try to at least document this. ### horizon-nova x-p session ### We mostly discussed the Horizon feature gaps [5]. The first priority would be Horizon to use OpenStackSDK instead of novaclient, but then supporting all of the new Nova API microversions. Unfortunately, we are no sure that we could have Horizon contributors that could fix those, but if you're a contributor and you want to help Horizon to be better, maybe you could do this ? If so, please ping me. ### Ironic-nova x-p session ### We didn't really have topics for this x-p session. We just quickly discussed some points, like Graphical Console support. Nothing really worth noting, maybe just that it would be nice that we could have readonly graphical console. We were just happy to say that the ironic driver now works better thanks to some features that were merged last cycles. Kudos to those who did them. ### HPC/AI optimized hypervisor "slices" ### A large topic to explain, I'll try to keep it short. Basically, how Nova slices the NUMA affinity between guests is nice but hard for HPC usecases where sometimes you need to better explain how to slice the NUMA dependent devices depending on the various PCI topologies. Eventually, we agreed on some POC that johnthetubaguy could work on by trying to implement a specific virt driver that would do something different from the existing NUMA affinities. ### Cinder-nova x-p session ### Multiple topics were discussed there. First, abishop wanted to enhance cinder's retyping of in-use boot volumes which means that the Nova os-attachments API to get a new parameter. We said that he needs to create a new spec and we agreed on the fact that the cinder contributors need to discuss with QEMU folks to know about the qemu writes. We also discussed about a new nova spec which is about adding burst length support to Cinder QoS [6]. We said that we need to both (nova and cinder) review this spec. About left residues when detaching a volume, we also agreed on the fact this is not a security flaw and the fact that os-brick should delete them, not nova (even if nova need to ask os-brick to look at that, either by a periodic run or when attaching/detaching). whoami-rajat will provide a spec for it. ### Python 3.13 support ### We discussed a specific issue for py3.13, the fact that the crypt module is no longer in stlib for py3.13, which impacts nova due to some usage in nova.virt.disk.api module for passing an admin password for file injection. Given file injection is deprecated, we have three possibilities: either removing admin password file injection (or even file injection as a whole), adding the new separate crypt package in upper-constraints or using oslo_uitls.secretutils module. bauzas (me) will provide an email to openstack-discuss for asking operators whether they are OK with deprecating file injection or just admin password injection and then we'll see the direction. bauzas or sean-k-mooney will also try to have py3.13 non-voting jobs for unittests/functtests. ### Eventlet removal steps in Nova ### I won't explain why we need to remove eventlet, you already know, right ? We rather discussed about the details in our nova components, including nova-api, nova-compute and other nova services. We agreed on removing direct eventlet imports where possible, move nova entrypoints that don't use eventlet to separate modules that don't monkeypatch the stdlib, look at what we can do with all our scatter_gather methods which asynchronously calling cells DB for using threads instead and check whether those calls are blocking on DB (and not on the MQ side). Gibi will shepherd that effort and provide some audit on the eventlet usage in order to avoid any unexpected but unfortunate late discoveries. ### Libvirt image backend refactor ### If you like spaghettis, you should pay attention to the libvirt image backend code. Lots of assumptions and conditionals make any change to that module hard to be written and hard to be reviewed, leading to errorprone situations like the ones we had when fixing some recent CVEs. We all agreed on the quite urgent necessity to refactor that code and melwitt proposed a multi-stage effort about that. We agreed on the proposal for the first two steps with some comments, leading to future revisions of the proposal's patches. The crucial bits with the refactor are about test coverage. ### IOThreads tuning for libvirt instances ### An old spec was already proposed for defining iothreads to guests. We agreed on reviving that spec, where a config option would define either no iothread or one iothread per instance (with a potential for a latter option value to be "one iothread per disk"). Depending on whether emulator_thread_policy is provided in the flavor/image, we would set the iothread on that policy or we would put the iothread floating over the shared CPU set. If no shared CPUs are configured but the operator wants iothreads, nova-compute would refuse to start. lajoskatona will work on such an implementation that will be designed in a blueprint that doesn't require a spec. ### OpenAPI schemas progress ### Nothing specific to say here, bauzas and gmann will review the series this cycle. That's it. I'm gone, I'm dead [7] (a cyclist metaphor) but I eventually skimmed the very large nova etherpad. Of course, 99% of chances that I'll write some notes incorrectly, so please correct if I'm wrong, I won't feel offended, just tired. Thanks all (and I hope your coffee or tea was good) -Sylvain [1] https://geek-and-poke.com/geekandpoke/2013/11/24/simply-explained [2] https://opendev.org/openstack/whitebox-tempest-plugin [3] https://www.qemu.org/docs/master/specs/rapl-msr.html [4] https://docs.openstack.org/nova/latest/configuration/config.html#pci.device… [5] https://etherpad.opendev.org/p/horizon-feature-gap#L69 [6] https://review.opendev.org/c/openstack/nova-specs/+/932653 [7] https://www.youtube.com/watch?v=HILcYXf8yqc

1 0

[neutron] How to disable Floating IP?
by yardalgedal＠gmail.com 29 Oct '24

29 Oct '24

Hi, Openstack Team, Is there a way how to temporarily disable floating ip? Like we have admin_state_up for many resources, but we don't have it for floating ip. admin_state_up of floating_port_id not working in Wallaby and I'm not sure it's working in later releases.

2 1