October 2024 - openstack-discuss - lists.openstack.org

Re: [Cinder][Ceph] Volumetria errada no cluster ceph
by Luiz Fernando Silva 29 Oct '24

29 Oct '24

I am sending the file with the ceph osd df tree and ceph pg ls-by-pool volumes output as an attachment. ceph -s: cluster: id: ccdcc86e-c0e7-49ba-b2cc-f8c162a42f91 health: HEALTH_OK services: mon: 3 daemons, quorum hc-node02,hc-node04,hc-node03 (age 5w) mgr: hc-node02(active, since 4M) osd: 54 osds: 54 up (since 14h), 54 in (since 3M) task status: data: pools: 5 pools, 3424 pgs objects: 4.25M objects, 43 TiB usage: 88 TiB used, 32 TiB / 120 TiB avail pgs: 3424 active+clean io: client: 16 MiB/s rd, 5.7 MiB/s wr, 856 op/s rd, 483 op/s wr Thank you very much, we will adjust to replica 3.

1 0

Re: [barbican][vault]
by Andy Speagle 29 Oct '24

29 Oct '24

On Thu, 2024-10-24 at 18:06 -0700, melanie witt wrote: > On 10/24/24 16:43, Andy Speagle wrote: > > Hey Folks, > > > > I'm having some trouble getting barbican to generate and/or store > > keys > > in our hashicorp vault in our charmed openstack environment. I have > > issues with both TPM and SSL related keys. > > > > I get extremely generic errors back that I can't seem to figure out > > how > > to troubleshoot: > > > > 'Traceback (most recent call last):\n File "/usr/lib/python3/dist- > > packages/nova/conductor/manager.py", line 688, in build_instances\n > > scheduler_utils.populate_retry(\n File "/usr/lib/python3/dist- > > packages/nova/scheduler/utils.py", line 998, in populate_retry\n > > raise > > exception.MaxRetriesExceeded(reason=msg)\nnova.exception.MaxRetries > > Exce > > eded: Exceeded maximum number of retries. Exceeded max scheduling > > attempts 3 for instance 0c50b2d7-8ac5-41a2-90ab-1e33f7aa1c22. Last > > exception: Key manager error: Internal Server Error: Secret > > creation > > failure seen - please contact site administrator.\n' > > Just FYI this is the generic MaxRetriesExceeded error which means > that > the server action failed on nova-compute(s). So you'll want to look > in > your nova-compute.log for the error that triggered the retries and > eventual MaxRetriesExceeded. > > > I'm using barbican 14.0.2 in yoga/stable ... anyone have an ideas > > how > > to troubleshoot this? > > I'm guessing in nova-compute.log you will see an error 403 forbidden > for > creating a barbican secret. If so, the most common reason I have seen > for this is if you have in your barbican.conf > [oslo.policy]enforce_new_defaults = False, then secret creates > require > the requester to have the 'creator' role assigned to them (you can do > this by 'openstack role add <options> <role>' [1]). > > When [oslo.policy]enforce_new_defaults = True, the 'creator' role is > not > required. Yup... nailed it. I really appreciate the insight. :) > The tl;dr is 'enforce_new_defaults' means to use the newer secure > RBAC > policies and the secure RBAC changes are shown in the Wallaby release > notes:

1 0

[neutron] Moving CI Meeting to Monday 1500 UTC
by Yatin Karel 29 Oct '24

29 Oct '24

Hello Neutron Team !! As discussed/agreed over the last Neutron CI meeting, pushed the changes[1][2] for the new timing. [1] https://review.opendev.org/c/opendev/irc-meetings/+/933637 [2] https://etherpad.opendev.org/p/neutron-ci-meetings -- Thanks and Regards Yatin Karel

1 0

[qa][ptg] QA 2025.1 vPTG Summary
by Martin Kopec 29 Oct '24

29 Oct '24

Hi everyone, here is a summary of the discussions. The full agenda can be found here [1]. == Dalmatian (2024.2) Retrospective == At the beginning of the cycle a devstack optimization change was merged [2] that uses the OSC service. If you haven't already, give it a try and enable it in your jobs. If it doesn't work for your project, let us know. == Migrating upstream CI/CD jobs to Ubuntu Noble (24.04) == Details about the migration can be found in [3] and [4] together with the testing status of all projects. We have time until Nov 29th to test the newer Ubuntu version and deal with any issues, after that, QA would like to proceed with the migration. == Triage previous and write new priority items == The list of the priority items planned for this cycle as well as the ones which were carried over from the previous cycles, can be found here [5]. == Open Discussion == There have been issues with a few jobs and tests that made an assumption of having the amount of certain resources, such as CPUs, guaranteed. However, that's not the case and the amount of the resources is not guaranteed in the upstream CI. No tests or jobs should make this assumption. Apart from CPUs, the same applies on floating ips, public addresses etc. [1] https://etherpad.opendev.org/p/oct2024-ptg-os-qa [2] https://review.opendev.org/c/openstack/devstack/+/676016 [3] https://lists.openstack.org/archives/list/openstack-discuss@lists.openstack… [4] https://etherpad.opendev.org/p/migrate-to-noble [5] https://etherpad.opendev.org/p/qa-epoxy-priority Thank you all for attending the PTG and contributing to the discussion, -- Martin Kopec Red Hat EMEA IM: kopecmartin

1 0

[nova][ptg] 2025.1 Epoxy PTG summary
by Sylvain Bauza 29 Oct '24

29 Oct '24

(resending the email as the previous one was blocked to an attached etherpad backup txtfile larger than the max size) Hey all, First, thanks for having joined us if you were in the vPTG. We had 15-20 people every day for our nova sessions, I was definitely happy to see new folks :-) If you want to see our PTG etherpad, please rather look at https://etherpad.opendev.org/p/r.4f297ee4698e02c16c4007f7ee76b7c1 instead of the main nova etherpad as I don't want that the etherpad would have a wrong traduction or having some paragraphs to be removed. As I say every cycle, just take a coffee (or a tea) now as the summary will be large. ### Dalmatian retrospective and Epoxy planning ### 6 of 15 approved blueprints were eventually implemented. We also merged more than 31 bugfixes during Dalmatian. We agreed to be explaining on the IRC channel when we have meetings for discussing some feature series (like the one we did every week for the manila/virtiofs series) and providing some public invitations. We could do this again this cycle for other features, we'll see. We will also try to have a periodic integration-compute job that pulls OSC and SDK from master. Our Epoxy deadlines will be : two spec review days (R-16, R-2), a soft spec approval freeze by R-16 and then hard spec approval freeze by R-12. That means that contributors really need to provide their specs before mid-December. Bauzas (me) will add these deadlines into the Epoxy schedule : https://releases.openstack.org/epoxy/schedule.html ### vTPM live migration ### We agreed on the fact that a vTPM live-migration feature is a priority for Epoxy given Windows 11. artom will create a spec proposing an image metadata property saying 'do I want to share my secret with nova service user ?' and also providing a new `nova-manage image_property set migratable_something` command so operators could migrate the existing instances for getting the Barbican secrets, if really the operators wants. ### Unified limits wrap-up ### We already have two changes needing to be merged before we can modify the default quota driver (in order to default to use unified limits). We agreed on reviewing both patches (one for treating unset limits as unlimited, the other about adding a nova-manage command for automatically creating nova limits) but we also discussed about a latter patch that would eventually say which nova resources need to be eventually set (so we *have to* enforce them anyway). melwitt agreed on working on that latter patch. ### per-process health checks ### We already had one series and we discussed it again. Gibi agreed on taking over it and he will re-propose the existing spec as it is. We also discussed the first checks we would have, like RPC failures and DB connection issues, we'll review those when they are in Gerrit. ### sustainable computing (a.k.a. power mgmt) ### When someone (I won't say who [1]) implemented power management in Antelope, this was nice but we eventually found a long list of bugs that we fixed. Since we don't really want to reproduce that experience, we had this kind of post-mortem where we eventually agreed on two things that could avoid reproducing that problem : a weekly periodic job will run whitebox tempest plugins [2] with nova-compute restarts also covered by a whitebox tempest plugin. Nobody is committed against those two actions but we hope to identify someone soon. As a side note, gibi mentioned RAPL MSR support [3], notifying us that we would have to support that in a later release (as the libvirt implementation is not merged yet) ### nvidia's vGPU vfio-pci variant driver support ### Long story short, since the linux kernel removed some feature in release 5.18 (IOMMU backend support for vfio-mdev) this impacted the nvidia driver which now detects that and then creates vfio-pci devices instead of vfio-mdev devices (as vGPUs). This has a dramatic impact on Nova as we relied on the vfio-mdev framework for abstracting virtual GPUs. By the next release, Nova will need to inventorize the GPUs by rather looking at SRIOV virtual functions which are specific to the nvidia driver (we call them vfio-pci variant driver resources). The nova PTG session focused on the required efforts to do so. We agreed on the fact it will require operators to propose different flavors for vGPU where they would require distinct resource classes (all but VGPU). Fortunately, we'll reuse existing device_spec PCI config options [4] where the operator would define custom resource classes which would match the PCI addresses of the nvidia-generated virtual functions (don't freak out, we'll also write documentation). We'll create another device type (something like type-VF-migratable) for describing such specific nvidia VFs. Accordingly the generated domain XML will correctly write the device description (amending the "managed=no" flag for that device). There will be an upgrade impact: existing instances will need to be resized to that new flavor (or instances will need to be shelved, updated for changing the embedded flavor and unshelved). In order to be on par with existing vGPU features, we'll also need to implement vfio-pci live-migration by detecting the VF type on the existing SRIOV live-migration. Since that effort is quite large, bauzas will incept a subteam of interested parties that would help him implement all of those bits in the short timeframe that is one upstream cycle. ### Graceful shutdowns ### A common pitfall that was told by tobian-urdin is when you want to stop nova-compute services. In general, before stopping the service, we should be sure that all RPC calls are done, which means we would no longer accept RPC calls after asking to stop the nova-compute and just awaiting the current calls to be done before stopping the service. For that, we need to create a backlog spec for discussing that design and we would also need to modify oslo.service for unsubscribing the RPC topics. Unfortunately, this cycle we won't have any contributor for working on it, but gibi could try to at least document this. ### horizon-nova x-p session ### We mostly discussed the Horizon feature gaps [5]. The first priority would be Horizon to use OpenStackSDK instead of novaclient, but then supporting all of the new Nova API microversions. Unfortunately, we are no sure that we could have Horizon contributors that could fix those, but if you're a contributor and you want to help Horizon to be better, maybe you could do this ? If so, please ping me. ### Ironic-nova x-p session ### We didn't really have topics for this x-p session. We just quickly discussed some points, like Graphical Console support. Nothing really worth noting, maybe just that it would be nice that we could have readonly graphical console. We were just happy to say that the ironic driver now works better thanks to some features that were merged last cycles. Kudos to those who did them. ### HPC/AI optimized hypervisor "slices" ### A large topic to explain, I'll try to keep it short. Basically, how Nova slices the NUMA affinity between guests is nice but hard for HPC usecases where sometimes you need to better explain how to slice the NUMA dependent devices depending on the various PCI topologies. Eventually, we agreed on some POC that johnthetubaguy could work on by trying to implement a specific virt driver that would do something different from the existing NUMA affinities. ### Cinder-nova x-p session ### Multiple topics were discussed there. First, abishop wanted to enhance cinder's retyping of in-use boot volumes which means that the Nova os-attachments API to get a new parameter. We said that he needs to create a new spec and we agreed on the fact that the cinder contributors need to discuss with QEMU folks to know about the qemu writes. We also discussed about a new nova spec which is about adding burst length support to Cinder QoS [6]. We said that we need to both (nova and cinder) review this spec. About left residues when detaching a volume, we also agreed on the fact this is not a security flaw and the fact that os-brick should delete them, not nova (even if nova need to ask os-brick to look at that, either by a periodic run or when attaching/detaching). whoami-rajat will provide a spec for it. ### Python 3.13 support ### We discussed a specific issue for py3.13, the fact that the crypt module is no longer in stlib for py3.13, which impacts nova due to some usage in nova.virt.disk.api module for passing an admin password for file injection. Given file injection is deprecated, we have three possibilities: either removing admin password file injection (or even file injection as a whole), adding the new separate crypt package in upper-constraints or using oslo_uitls.secretutils module. bauzas (me) will provide an email to openstack-discuss for asking operators whether they are OK with deprecating file injection or just admin password injection and then we'll see the direction. bauzas or sean-k-mooney will also try to have py3.13 non-voting jobs for unittests/functtests. ### Eventlet removal steps in Nova ### I won't explain why we need to remove eventlet, you already know, right ? We rather discussed about the details in our nova components, including nova-api, nova-compute and other nova services. We agreed on removing direct eventlet imports where possible, move nova entrypoints that don't use eventlet to separate modules that don't monkeypatch the stdlib, look at what we can do with all our scatter_gather methods which asynchronously calling cells DB for using threads instead and check whether those calls are blocking on DB (and not on the MQ side). Gibi will shepherd that effort and provide some audit on the eventlet usage in order to avoid any unexpected but unfortunate late discoveries. ### Libvirt image backend refactor ### If you like spaghettis, you should pay attention to the libvirt image backend code. Lots of assumptions and conditionals make any change to that module hard to be written and hard to be reviewed, leading to errorprone situations like the ones we had when fixing some recent CVEs. We all agreed on the quite urgent necessity to refactor that code and melwitt proposed a multi-stage effort about that. We agreed on the proposal for the first two steps with some comments, leading to future revisions of the proposal's patches. The crucial bits with the refactor are about test coverage. ### IOThreads tuning for libvirt instances ### An old spec was already proposed for defining iothreads to guests. We agreed on reviving that spec, where a config option would define either no iothread or one iothread per instance (with a potential for a latter option value to be "one iothread per disk"). Depending on whether emulator_thread_policy is provided in the flavor/image, we would set the iothread on that policy or we would put the iothread floating over the shared CPU set. If no shared CPUs are configured but the operator wants iothreads, nova-compute would refuse to start. lajoskatona will work on such an implementation that will be designed in a blueprint that doesn't require a spec. ### OpenAPI schemas progress ### Nothing specific to say here, bauzas and gmann will review the series this cycle. That's it. I'm gone, I'm dead [7] (a cyclist metaphor) but I eventually skimmed the very large nova etherpad. Of course, 99% of chances that I'll write some notes incorrectly, so please correct if I'm wrong, I won't feel offended, just tired. Thanks all (and I hope your coffee or tea was good) -Sylvain [1] https://geek-and-poke.com/geekandpoke/2013/11/24/simply-explained [2] https://opendev.org/openstack/whitebox-tempest-plugin [3] https://www.qemu.org/docs/master/specs/rapl-msr.html [4] https://docs.openstack.org/nova/latest/configuration/config.html#pci.device… [5] https://etherpad.opendev.org/p/horizon-feature-gap#L69 [6] https://review.opendev.org/c/openstack/nova-specs/+/932653 [7] https://www.youtube.com/watch?v=HILcYXf8yqc

1 0

[neutron] How to disable Floating IP?
by yardalgedal＠gmail.com 28 Oct '24

28 Oct '24

Hi, Openstack Team, Is there a way how to temporarily disable floating ip? Like we have admin_state_up for many resources, but we don't have it for floating ip. admin_state_up of floating_port_id not working in Wallaby and I'm not sure it's working in later releases.

2 1

[tc][all] OpenStack Technical Committee Weekly Meeting Agenda (2025.1/R-22)
by Goutham Pacha Ravi 28 Oct '24

28 Oct '24

Hello Stackers, As you're aware, we concluded the virtual Project Teams Gathering event last week. This week, we’ll start addressing the to-do lists we created. The Technical Committee met for six hours, in addition to participating in several discussions with various project teams. I’m working on sharing a summary of the week’s proceedings with you via this mailing list. In the meantime, we’ll be catching up on 2024-10-29 at the TC’s weekly IRC meeting in OFTC’s #openstack-tc channel. The agenda for this meeting is available on the Meeting Wiki [1]. I hope you’ll be able to join us. === Governance Proposals === ==== Merged ==== - Retire kuryr-kubernetes and kuryr-tempest-plugin | https://review.opendev.org/c/openstack/governance/+/922507 - Propose a pop-up team for eventlet-removal | https://review.opendev.org/c/openstack/governance/+/931978 ==== Open for Review ==== - Add Cinder Huawei charm | https://review.opendev.org/c/openstack/governance/+/867588 - Add watcher DPL for Epoxy | https://review.opendev.org/c/openstack/governance/+/933018 - Add Axel Vanzaghi as PTL for Mistral | https://review.opendev.org/c/openstack/governance/+/927962 - Propose to select the eventlet-removal community goal | https://review.opendev.org/c/openstack/governance/+/931254 Thank you very much for reading! On behalf of the OpenStack TC, Goutham Pacha Ravi (gouthamr) OpenStack TC Chair P.S: OpenStack TC PTG YouTube Playlist is shaping up here [2]. Please feel free to browse through [1] TC Meeting Agenda, 2024-10-29: https://wiki.openstack.org/wiki/Meetings/TechnicalCommittee#Next_Meeting [2] https://youtube.com/playlist?list=PLhwOhbQKWT7XGjIwT0mtPpixpuY-tKYoh&featur…

1 0

[glance] Hold your rechecks
by Abhishek Kekane 28 Oct '24

28 Oct '24

Hi Team, Currently glance gate is failing for glance-cinder-multistore-import job with quota exceeding issue. The quota issue seems to be occurring from a recent change in devstack (removing volumev3 endpoint) causing issues with openstack client commands (set quotas). The details of the issue and fix are in this release patch https://review.opendev.org/c/openstack/releases/+/933578 Until this release patch is merged, kindly avoid posting recheck on your patch. Thanks & Best Regards, Abhishek Kekane

1 1

[Neutron] Bug Deputy Oct 21 - 27
by Miro Tomaska 28 Oct '24

28 Oct '24

Hello All, Here is the report for the week from Oct 21 - 27. We had a busy week. *Critical:* https://bugs.launchpad.net/neutron/+bug/2085462 - [OVN] "test_trunk_subport_lifecycle" unstable Assigned: Rodolfo Alonso https://bugs.launchpad.net/neutron/+bug/2085543 - [OVN] Port device_owner is not set in the Trunk subport Assigned: Yatin Karel *High*: https://bugs.launchpad.net/neutron/+bug/2084977 - Deleting a security group fails "cannot find port_group" Unassigned https://bugs.launchpad.net/neutron/+bug/2085422 - [OVN] "test_overlapping_sec_grp_rules" is unstable Unassigned https://bugs.launchpad.net/neutron/+bug/2084977 - Deleting a security group fails with "Cannot find Port_Group with name=pg_<id>" when neutron-api is run by httpd + mod_wsgi Assigned: Terry Willson https://bugs.launchpad.net/neutron/+bug/2085422 - test_overlapping_sec_grp_rules" is unstable Unassigned *Medium*: https://bugs.launchpad.net/neutron/+bug/2085491 - How to run neutron fwaasV2 ovn driver Unassigned *Low*: https://bugs.launchpad.net/neutron/+bug/2085099 -[OVN] E/W traffic fragmentation for routed traffic Assigned: Rodolfo Alonso https://bugs.launchpad.net/neutron/+bug/2085447 - Fast exit ``_ensure_external_network_default_value_callback`` if network not external Assigned to Rodolfo Alonso *Undecided:* https://bugs.launchpad.net/neutron/+bug/2085421 - Tempest failing at teardown -- Miro Tomaska Senior Software Engineer

1 0

#eventlet-removal - PTG October 2024 Summary
by Herve Beraud 28 Oct '24

28 Oct '24

Last week was the OpenStack PTG. The Eventlet removal initiative ran two PTG sessions. The first session [1] was a presentation of the initiative, where the second session [2] was a technical open discussion. Please find out below the main key points and the proposed answers. ## Key Points The first key point made during these PTG sessions is that without consistency in our choices and decisions we will create more complexity than anything else. It is crucial to avoid having multiple ways to start services across different projects. A unified approach will simplify maintenance and enhance user experience. Leading us to the following questions: - how to remove the WSGI features provided by Eventlet and how to move to another HTTP server (WSGI/ASGI); - how to manage tasks. The WSGI/HTTP server discussion just come in time with the proposal made by Stephen Finucane to Deprecate/remove non-uWSGI deployment mechanisms from Devstack: https://lists.openstack.org/archives/list/openstack-discuss@lists.openstack… We think that this topic overlaps with the WSGI aspects of Eventlet removal, for this reason, with the aim to answer these questions we propose to create specialized working groups (see the section below). We also made comparisons between available alternatives, here is an example with the SGI question https://wiki.openstack.org/wiki/SGI-servers-comparison. Find out all our comparisons in the official wiki page: https://wiki.openstack.org/wiki/Eventlet-removal#The_HTTP_SGI_Working_Group ## Specialized working groups The main role of these working groups is to avoid having multiple ways to manage things. Groups would be responsible for producing a unified approach that will simplify maintenance and enhance user experience. Our discussions have brought out 3 specialized working groups: - The HTTP SGI Working Group - The Tasks/Process Working Group - The Architecture, Interoperability and Technical Standardization Working Group If you are interested in one or many of these specialized working groups, then, please raise your hand and join the discussions. Please also take a look to the official page: https://wiki.openstack.org/wiki/Eventlet-removal#Specialized_Working_Groups… ## Latest Updates - Glance took full advantage of the PTG discussion. They got an overview of their WSGI migration options and they gain a better understanding of their current situation (https://etherpad.opendev.org/p/oct2024-ptg-glance#L277) and the Glance team were able to set up a migration plan: https://etherpad.opendev.org/p/oct2024-ptg-glance#L340 - Ironic Python Agent (IPA), is also experimenting about their Eventlet removal, their works is strongly related to the WSGI feature replacement: https://review.opendev.org/c/openstack/ironic-python-agent/+/924634 - The threading executor of oslo.messaging could shorten our migration path concerning the RPC aspects. For this reason we proposed to start the deprecation of the Eventlet executor within oslo.messaging. https://review.opendev.org/c/openstack/oslo.messaging/+/933399 - Another topic that arose during the PTG is how to manage timeouts. Some proposal have been made, they propose to rely on SQLAlchemy or Ceph to handle timeout management: https://etherpad.opendev.org/p/oct2024-ptg-eventlet-removal#L103 https://etherpad.opendev.org/p/oct2024-ptg-os-tc#L298 ## Important Dates Nov. 5th is the first IRC meeting on #openstack-eventlet-removal, do not hesitate to join us. We will cover all the topics addressed in this mailing thread. ## SLURP vs non-SLURP The PTG sessions came with timeline and cadence discussions. For this reason we made some proposal about workload regulation: https://wiki.openstack.org/wiki/Eventlet-removal#Workload_regulation_2 In substance we propose to base our cadence on SLURP and non-SLURP releases. Epoxy is a SLURP release, for this reason we should not expect breaking changes happening during this series, however, SLURP series are good opportunities to deprecate things, remove useless workaround, and start small clean up in preparation of wider changes. ## Final Thoughts We strongly encourage a collaborative selection of our candidate(s) for HTTP server replacement .This selection will dictate how we deal with server gateway interfaces for the coming decade, so we should focus our coming discussion on this specification, for this reason we invite you to join the specialized working groups. Thanks to all the people who joined the various discussions. Thank you for your dedication and hard work! [1] https://etherpad.opendev.org/p/oct2024-ptg-os-tc#L195 [2] https://etherpad.opendev.org/p/oct2024-ptg-eventlet-removal -- Hervé Beraud Senior Software Engineer at Red Hat irc: hberaud https://github.com/4383/

1 0