November 2024 - openstack-discuss

Re: GPU PCI passthrough woes.
by Andy Speagle 05 Nov '24

05 Nov '24

Aye, that seems to have done it. Strange that this wasn't necessary for my old 1080 cards. Thanks! On Wed, 2024-10-30 at 08:12 +0000, Alexander Dibbo - STFC UKRI wrote: > ⚠️CAUTION⚠️ > This email originated outside of Toyon's network. Do not download > attachments or click links unless you know the content is safe. > > Hi Andy, > > We've seen many issues with pci passthrough and I suspect that the > l40s will have a seen similar issues with the a100s > > Here is an alias string that works for our a100s: > `alias = {"vendor_id": "10de", "product_id": "20f1", "device_type": > "type-PF", "name": "nvidia-tesla-a100-pcie-vga"}` > > I believe it is the `device_type` that will be your issue > > Regards > > Alexander Dibbo – Cloud Architect / Cloud Operations Group Leader > For STFC Cloud Documentation visit > https://urldefense.us/v3/__https://stfc.atlassian.net/wiki/spaces/CLOUDKB/o… > To raise a support ticket with the cloud team please email > cloud-support(a)stfc.ac.uk > To receive notifications about the service please subscribe to our > mailing list at: > https://urldefense.us/v3/__https://www.jiscmail.ac.uk/cgi-bin/webadmin?A0=S… > To receive fast notifications or to discuss usage of the cloud please > join our Slack: > https://urldefense.us/v3/__https://stfc-cloud.slack.com/__;!!A4KLn-w15EM4BR… > > -----Original Message----- > From: Andy Speagle <aspeagle(a)toyon.com> > Sent: 30 October 2024 02:15 > To: openstack-discuss(a)lists.openstack.org > Subject: GPU PCI passthrough woes. > > Hey Folks, > > I could use a little assistance getting GPU passthrough working. I > had > this working already for one flavor of nvidia gpu... and I've added > some hosts with a much newer gpu... > > I've updated the pci_alias and pci_passthrough variables and those > seem > to be getting set properly in nova.conf > > passthrough_whitelist = [{"vendor_id":"10de", > "product_id":"1b06"},{"vendor_id":"10de", "product_id":"26b9"}] > > alias = {"name": "gpu", "product_id": "1b06", "vendor_id": "10de"} > alias = {"name": "gpu-l40s", "product_id": "26b9", "vendor_id": > "10de"} > > I believe I have all of the iommu stuff configured and have the pci- > stub module entries... dmesg output shows that the GPUs are being > claimed by the stub module. > > $ openstack flavor show t1.small_gpu_l40s > +----------------------------+--------------------------------------+ > > Field | Value | > +----------------------------+--------------------------------------+ > > OS-FLV-DISABLED:disabled | False | > > OS-FLV-EXT-DATA:ephemeral | 0 | > > access_project_ids | None | > > description | None | > > disk | 0 | > > id | af70c94e-0026-4a39-bc1e-dfb93b286a54 | > > name | t1.small_gpu_l40s | > > os-flavor-access:is_public | True | > > properties | pci_passthrough:alias='gpu-l40s:1' | > > ram | 2048 | > > rxtx_factor | 1.0 | > > swap | 0 | > > vcpus | 1 | > +----------------------------+--------------------------------------+ > > Yet... I can't seem to get an instance to run using that new > flavor... > keeps complaining about there not being enough hosts available. > > Fault: > code: 500 > created: 2024-10-30T02:04:33Z > message: "No valid host was found. There are not enough hosts > available." > details: | > Traceback (most recent call last): > File "/usr/lib/python3/dist- > packages/nova/conductor/manager.py", > line 1580, in schedule_and_build_instances > host_lists = self._schedule_instances(context, > request_specs[0], > File "/usr/lib/python3/dist- > packages/nova/conductor/manager.py", > line 940, in _schedule_instances > host_lists = self.query_client.select_destinations( > File "/usr/lib/python3/dist- > packages/nova/scheduler/client/query.py", line 41, in > select_destinations > return self.scheduler_rpcapi.select_destinations(context, > spec_obj, > File "/usr/lib/python3/dist-packages/nova/scheduler/rpcapi.py", > line 160, in select_destinations > return cctxt.call(ctxt, 'select_destinations', **msg_args) > File "/usr/lib/python3/dist- > packages/oslo_messaging/rpc/client.py", line 189, in call > result = self.transport._send( > File "/usr/lib/python3/dist- > packages/oslo_messaging/transport.py", line 123, in _send > return self._driver.send(target, ctxt, message, > File "/usr/lib/python3/dist- > packages/oslo_messaging/_drivers/amqpdriver.py", line 689, in send > return self._send(target, ctxt, message, wait_for_reply, > timeout, > File "/usr/lib/python3/dist- > packages/oslo_messaging/_drivers/amqpdriver.py", line 681, in _send > raise result > nova.exception_Remote.NoValidHost_Remote: No valid host was > found. > There are not enough hosts available. > > Traceback (most recent call last): > File "/usr/lib/python3/dist- > packages/oslo_messaging/rpc/server.py", line 241, in inner > return func(*args, **kwargs) > > File "/usr/lib/python3/dist- > packages/nova/scheduler/manager.py", > line 223, in select_destinations > selections = self._select_destinations( > > File "/usr/lib/python3/dist- > packages/nova/scheduler/manager.py", > line 250, in _select_destinations > selections = self._schedule( > > File "/usr/lib/python3/dist- > packages/nova/scheduler/manager.py", > line 416, in _schedule > self._ensure_sufficient_hosts( > > File "/usr/lib/python3/dist- > packages/nova/scheduler/manager.py", > line 455, in _ensure_sufficient_hosts > raise exception.NoValidHost(reason=reason) > > nova.exception.NoValidHost: No valid host was found. There are > not > enough hosts available. > > Any clues on how/where to dig into this more to see what might be > missing? Thanks. > > -- > Andy Speagle > -- Andy Speagle Sr. Site Reliability Engineer Toyon Research Corporation 316.617.2431

3 2

Is HPET recommended for x86 instance?
by Jiatong Shen 05 Nov '24

05 Nov '24

Hello Community Experts, I found that to enable the HPET timer for x86 guests, the operator should set hw_time_hpet on the given image. By default, HPET is disabled. My question is what are the pros and cons for using HPET? Thanks. -- Best Regards, Jiatong Shen

2 3

[cloudbase-init] This project doesn't seem to be well managed.
by dogyun kim 04 Nov '24

04 Nov '24

Hi, I wanted to contribute to that project, so we created a CR. https://review.opendev.org/c/x/cloudbase-init/+/933629 But no one managed this project, so i can't get review. If you have an administrator, I would like to ask you to confirm it. Thanks

2 1

[cloudbase-init] This project doesn't seem to be well managed.
by 김도균 04 Nov '24

04 Nov '24

3 2

[keystone] Weekly Meeting Canceled 06-Nov-2024
by Dave Wilde 04 Nov '24

04 Nov '24

Hello all, Due to some scheduling conflicts I have to cancel this weeks Keystone meeting. Please feel free to add to the agenda [0] or reach out in #openstack-keystone if you need anything from us. Thanks, /Dave 0: https://etherpad.opendev.org/p/keystone-weekly-meeting

1 0

[neutron] Bug deputy report (week starting on October 28)
by Bernard Cafarelli 04 Nov '24

04 Nov '24

Hey neutrinos, here is the list of bugs reported during 2024 week 44 While this was Halloween week, the number of reported bugs was not too scary, and bugs have patches or assignee High * [OVN] Revision number registers must be filtered by resource ID and type - https://bugs.launchpad.net/neutron/+bug/2085946 Patch by Rodolfo https://review.opendev.org/c/openstack/neutron/+/933752 W+1 Medium * ´openstack vpn service show´ command doesnt return external_v4_ip and external_v6_ip - https://bugs.launchpad.net/python-neutronclient/+bug/2086144 CLI issue for VPNaaS, Lajos looking into it * [ovn-octavia-provider] Load balancer enable/disable has a reverse effect - https://bugs.launchpad.net/neutron/+bug/2086194 Patch proposed https://review.opendev.org/c/openstack/ovn-octavia-provider/+/933879 * [OVN] The security group create command is not creating the SG rules revision number registers - https://bugs.launchpad.net/neutron/+bug/2086205 Patch proposed https://review.opendev.org/c/openstack/neutron/+/933969 Low * [neutron-tempest-plugin] Trunk tests are running out of VLAN IDs when creating a VLAN network - https://bugs.launchpad.net/neutron/+bug/2086602 Patch proposed https://review.opendev.org/c/openstack/neutron-tempest-plugin/+/934027 Wishlist * Reboot breaks VPN connection (OVN) - https://bugs.launchpad.net/neutron/+bug/2086189 Marked as wishlist for VPNaaS, as this is for redundancy on reboot -- Bernard Cafarelli

1 0

[openstack-ansible][ptg] Epoxy Project Team Gathering summary for OpenStack-Ansible
by Dmitriy Rabotyagov 04 Nov '24

04 Nov '24

Hi folks, Despite low participation in the PTG this time, we still had quite productive discussions around a variety of topics. You can find some summary on our future plans below: * We agreed to proceed with upgrading RabbitMQ to version 4.0 for Dalmatian. This means that HA (replicated) Queues won't be available anymore so Caracal will be the only release having support for both HA Queues and Quorum Queues. Deployments are expected to accomplish their migration to Quorum Queues (or just Classic Queues v2) on Caracal before proceeding further. * Implementing unified limits At this point of time, support for unified limits across services is not clear enough for us as well as how from operator perspective a mix of unified limits and classic quota will look like, as these 2 options are very distinct from each other. While one use case for their implementation would be vGPUs, we need an interested party for implementing them at this point. * Reviving Watcher support We have retired the Watcher role in Stein. With the project itself getting more traction these days, we agreed to proceed with reviving the role code to re-add support for the Watcher. * uWSGI and eventlet replacement A big topic throughout this PTG was replacement of eventlet with more native async approaches. However, uWSGI has no support for any of them right now. Since uWSGI is also in a maintenance mode, there is no expectations for adding async support to the project either. With that we need to start looking at alternatives, like uvicorn, which is already used in Skyline role. With that we potentially want to re-purpose the uWSGI role to be more generic at serving WSGI, meaning we will look into implementing service selection (uwsgi/gunicorn/uvicorn/etc) and renaming the role to address the more generic nature of it. * PKI role improvements This was another big topic we have discussed this time. There are multiple issues we need to address with the role. It could be split into multiple sub-topics: - Improving CA private key storage method on the deploy host to be able to store them encrypted with ansible-vault/sops or some other lookup mechanism - We need to implement better way for creating and storing private keys for generated certificates as transferring them from the deploy host is not really a best practice. - Adding support for hashicorp vault to store it in an encrypted manner. With that we should also add support to have CA in hashi-vault directly [1] and issuing certs from there. - Improve certs rollout process to allow short-term certificates to be issued and rotated regularly. This includes dynamically reloading libvirt config to address VNC consoles and reducing runtime for certificate rotation. * Would be great to have finally another round on ProxySQL efforts * Document of making user_secrets not a plain-texted file. Also address upgrade script failure in case user_secrets contain ansible-vault/sops/etc rather than plain text. * Document 2 ways of doing "pretty endpoints" - domain based and subpath based. * We need to work closely on deprecation of gather_subset starting with ansible-core 2.18, as there is no straightforward replacement. * Some repositories are missing deb822_repository migration, ie: frrouting, zun, qdrouterd [1] https://developer.hashicorp.com/vault/tutorials/secrets-management/pki-engi…

1 0

[neutron][ovn-octavia-provider][octavia][ovn] Octavia OVN DBs sync cmd
by Rico Lin 04 Nov '24

04 Nov '24

Hi all I would like to raise attention to the patch set that introduces Octavia OVN DBs sync cmd. patches: https://review.opendev.org/q/topic:%22add-ovn-db-sync-utils%22 With command `octavia-ovn-db-sync-util {--debug}` It should allows us to do the OVN DB sync similar to what `neutron-ovn-db-sync-util` does. And thanks to Fernando Royo, the patches are small and earlier to review in focused functions. Would be great if we could have some more reviews for it. *Rico Lin*

2 1

[tacker][ptg] 2025.1 Epoxy vPTG summary
by Yasufumi Ogawa 01 Nov '24

01 Nov '24

Hi tacker team, Thank you for joining the vPTG in the last week. I'd like to share the summary of our two days discussion for the proposals on the etherpad[1]. ## [recap] Revise FTs(yasufum) We have revised functional test from Bobcat and it's a recap session for looking back the remained issues. All the issues are going to be fixed in Epoxy. * Revise directory structure * Names of functions, classes and files too long for maintenance to be done this release * Combine scenarios into one to reduce the number of tests if possible. In process * Reduce setting up infra for the tests to be done this release * Revising scenarios and reducing setups are going to be started in this release. ## Plan for Improvement FT in Epoxy release (yasuum) We discussed the items about strategies for improving functional tests. To complete the goal of this proposal, we will take several steps to focus on dominant ones. (1) Divide time consuming tests, (2) Move tests less frequently updated to non-voting (3) Drop tests almost not updated from zuul (4) Rename each test scinarios and revise directory structure. We agreed with this proposal, but which of tests will be divided or dropped was not decided in the discussion. So, continue the discussion in the next IRC meeting. ## Add option to enable/disable Cinder volume recreation in v1 Heal request (koba) In the current v1 Heal request, the related Cinder Volume for the specified VDU is recreated although it can be specified not to recreate it. To support it in v1, Clients specify a flag in the additional_params to indicate whether to recreate the volume or not when making a Heal request. * Tacker conductor checks this flag to decide whether to mark the storage resources when executing STACK_UPDATE. Heal process works by recreating the resources marked "mark-unhealthy" during STACK_UPDATE. *Consider ensuring version compatibility and consistency with v2 operations. * Default behavior will be the same as now (recreating the Cinde Volume). * The name of the flag parameter in the additonal_params can be configured by the admin in tacker.conf. ## Add default_secret_key Option to [vim_keys] for Multi-Master Tacker Deployment (koba) ## Enhancement of the Ansible Driver (sample mgmt driver) (koba) Tacker currently supports only a single version of Ansible. As a result, when tenants include Playbooks in the VNF-Package that only work with specific versions of Ansible, there may be compatibility problems with the Playbooks. Tacker administrators must modify the Ansible Driver’s source code directly if they want to enforce options related to ansible-playbook execution, such as specifying log storage locations or custom callback plugins. These issues lack flexibility for both tenants and administrators. We agreed the proposed approaches for supporting several versions of ansible, but still not sure it's appropriate to define all the supported versions in tacker.conf and select which version will be applied by specifying it in the definition file in VNF package. It going to be discussed the way of selecting version in the next meeting. ## Remove legacy engine facade (yasufum) It's one of the community goal, so we'll fix it in this release. ## Remove sphinxcontrib-*diag from your (docs) dependencies (yasufum) It's similar to the previous item, and fix it in this release. ## Ensuring Tacker Compatibility with TST and SOL Specifications (Renu Rani) Agreed to the approach proposed on the etherpad. * Addressing the Gap Between TST and SOL Specifications: Raise bugs on TST and request the TST team to update schemas and variables according to SOL specifications. * Support of Compliance Tests for Tacker V2 a. Resolving Issues with Interconnected Test Cases in Tacker: Update Tacker's code to ensure compatibility with the interconnected test cases in v4.5.1-dev. b. Supporting Tacker for TST Release-4: Implement new Tacker compliance test cases for TST Release-4. c. Validating Compliance Test Cases with Tacker V2: Seek the Tacker team's opinion on whether to use Release 3 or Release 4 for validation. ## Enhancing Tacker Testing Coverage in the OpenStack Helm Check Pipeline (Shivam Shukla) To further enhance Tacker testing within the OpenStack Helm Check pipeline by adding more testcases to ensure thorough verification of Tacker deployments. Although the test scenarios were proposed, but we were not sure all the scenario are really needed because some of them are existing scenarios. We agreed to accept the proposal itself, but still not all the scenario should be introduced. So, continue to discuss for that. 1. VIM management (create, update and delete VIMs) 2. VNFD management (create and delete VNF packages) 3. VNF Instance management (create, update and delete VNF instances) 4. VNF LCM operation (instantiate, terminate, heal and scale) 5. Subscription management (create and delete subscription for VNFs) [1] https://etherpad.opendev.org/p/oct2024-ptg-tacker

1 0

[ironic] Bare Metal SIG
by Julia Kreger 01 Nov '24

01 Nov '24

Greetings folks, During this past week's PTG, the Ironic community participants reached consensus to wrap up the Bare Metal SIG. There are many reasons, but the bulk of the reasoning revolves around spreading word and collaborating outward as opposed to focusing on our own vertical of interaction. Having discussions in different contexts helps bring strength to discussions and we encourage contributors to engage in adjacent community and group discussions where relevant then build connections between communities. Communities, when working together, are always stronger when we build these sorts of connections or bonds. We hope to continue having meetups with operators as we always have, even if it may not bear the Bare Metal SIG name. As always, operators of any bare metal provisioning software are welcome to join our discussions at any time in OFTC IRC #openstack-ironic or right here on the mailing list. Among the accomplishments of the Baremetal SIG, was a whitepaper[0] devoted to the Ironic use case. Special thanks goes to Chris Hoge, previously with the Open Infra Foundation who helped drive us to the finish line on that effort. Beyond the whitepaper, and likely even harder was to bring people together in person on multiple occasions! A special thank goes to Arne Wiebalck who championed the Bare Metal SIG and helped steward several meetups including the most recent meetup at CERN in late Spring of this year. The success of that event speaks volumes about our community and the power of collaboration! Many thanks also goes to everyone who helped with the videos we have posted to youtube[1]! Without everyone pitching in, we wouldn't have been able to achieve success! -Julia, on behalf of the Ironic team. [0]: https://www.openstack.org/use-cases/bare-metal/how-ironic-delivers-abstract… [1]: https://www.youtube.com/playlist?list=PLKqaoAnDyfgoBFAjUvZGjKXQjogWZBLL_

1 0