February 2023 - openstack-discuss

[cloudkitty] question to the scope_key in cloudkitty
by Noah Elias Feldt 28 Jul '23

28 Jul '23

Hello! I am currently trying to get CloudKitty to work with the Prometheus Collector and PyScripts. I noticed that the scope_key for Prometheus can only be changed once in the global configuration file. This was a problem for me because we need different scope_keys for different metrics. Many of the metrics still need the scope key "tenant_id" and some others "project_id". I have found in the documentation of CloudKitty no way to change this for each metric, so I have internally changed something in the source code and added the function to set the scope_key for each metric individually. I just wanted to ask if you would be interested if I submit the small change upstream? Thank you. Regards Noah Feldt

3 3

[cinder] cinder-backup volume stuck in creating
by Satish Patel 27 Jul '23

27 Jul '23

Folks, I have configured nova and cinder with ceph storage. VMs running on ceph storage but now when i am trying to create a backup of cinder volume its getting stuck on creating and doing nothing. Logs also do not give any indication of bad. My cinder.conf [DEFAULT] enabled_backends = rbd-1 backup_driver = cinder.backup.drivers.ceph.CephBackupDriver backup_ceph_conf = /etc/ceph/ceph.conf backup_ceph_user = cinder-backup backup_ceph_chunk_size = 134217728 backup_ceph_pool = backups backup_ceph_stripe_unit = 0 backup_ceph_stripe_count = 0 restore_discard_excess_bytes = true osapi_volume_listen = 10.73.0.181 osapi_volume_listen_port = 8776 Output of "openstack volume service list" showing cinder-backup service is up but when i create a backup it's getting stuck in this stage and no activity. I am not seeing anything getting transferred to the ceph backups pool also. Any clue? or method to debug? # openstack volume backup list --all +--------------------------------------+------+-------------+----------+------+ | ID | Name | Description | Status | Size | +--------------------------------------+------+-------------+----------+------+ | bc844d55-8c5a-4bd3-b0e9-7c4c780c95ad | foo1 | | creating | 20 | +--------------------------------------+------+-------------+----------+------+

5 7

Re: [all] [oslo.messaging] Interest in collaboration on a NATS driver
by Tobias Urdin 22 Jul '23

22 Jul '23

Hello Kai, Great to hear that you’re interested! I’m currently just testing this as a POC directly in devstack so there is real-world testing (not should it yet I think, needs more development first). Best regards Tobias > On 30 Aug 2022, at 18:32, Kai Bojens <kb(a)kbojens.de> wrote: > > Am 29.08.22 um 15:46 schrieb Tobias Urdin: > >> If anybody, or any company, out there would be interested in collaborating in a project to bring this support and maintain it feel free to >> reach out. I’m hoping somebody will bite but atleast I’ve put it out there for all of you. > > Hi, > I am very much interested to take a closer look at your work and maybe contribute to it. Although I'm working with OpenStack for my employer at the moment, I'd do this in my spare time as I'm not sure that I can convince him to add another staging system with a full OpenStack installation just for developing a RabbitMQ replacement. That's not on our agenda as we are mostly users and not developers of OpenStack. > > As I'm pretty new to the messaging topic and had just heard of NATS and your idea, I'll first would have to dive into NATS and then I would take a closer look at your code and maybe try to create some documentation or tests. So, I can't promise anything but as I said I'm very interested in your approach as I also see the massive load from RabbitMQ. > > This brings me to the most important question: Where do you run and test your code? > > Greetings, > Kai

8 30

Experience with VGPUs
by Oliver Weinmann 22 Jun '23

22 Jun '23

Dear All, we are planning to have a POC on VGPUs in our Openstack cluster. Therefore I have a few questions and generally wanted to ask how well VGPUs are supported in Openstack. The docs, in particular: https://docs.openstack.org/nova/zed/admin/virtual-gpu.html explain quite well the general implementation. But I am more interested in general experience with using VGPUs in Openstack. We currently have a small YOGA cluster, planning to upgrade to Zed soon, with a couple of compute nodes. Currently our users use consumer cards like RTX 3050/3060 on their laptops and the idea would be to provide VGPUs to these users. For this I would like to make a very small POC where we first equip one compute node with an Nvidia GPU. Gladly also a few tips on which card would be a good starting point are highly appreciated. I know this heavily depends on the server hardware but this is something I can figure out later. Also do we need additional software licenses to run this? I saw this very nice presentation from CERN on VGPUs: https://indico.cern.ch/event/776411/contributions/3345183/attachments/18516… In the table they are listing Quadro vDWS licenses. I assume we need these in order to use the cards? Also do we need something like Cyborg for this or is VGPU fully implemented in Nova? Best Regards, Oliver

17 36

Kubernetes Conformance 1.24 + 1.25
by Kendall Nelson 06 Jun '23

06 Jun '23

Hello All! First of all, I want to say a huge thanks to Guilherme Steinmuller for all his help ensuring that OpenStack Magnum remains Kubernetes Certified [1]! We are certified for v1.24! However, in digging into the set of conformance tests this round, it appears that we will not be able to get 1.25 certified with the current Magnum. This means that in May when we have to re-cert (every 3 months they reach an EOL because that's k8s release cadence) we will not be able to maintain this which is the other part of why I wanted to write this email - to raise awareness and ask for help digging into what needs to happen to make sure we can retain this. Tangentially related, Guilherme has posted a patch with documentation for how to run the conformance tests for yourself and it could use some reviews [2]. And one last thing, I have begun conversation with the opendev folks about what it might take to setup jobs to run the conformance tests periodically so that we don't have to manually run them when it's time to recert. I am planning on submitting a forum topic for this for those interested :) Long live LOKI! - Kendall Nelson [1] https://github.com/cncf/k8s-conformance/pull/2446 [2] https://review.opendev.org/c/openstack/magnum/+/849156

5 10

[ironic][stable] Proposing EOL of ironic project branches older than Wallaby
by Jay Faulkner 02 Jun '23

02 Jun '23

Hi all, Ironic has a large amount of stable branches still in EM. We need to take action to ensure those branches are either retired or have CI repaired to the point of being usable. Specifically, I'm looking at these branches across all Ironic projects: - stable/queens - stable/rocky - stable/stein - stable/train - stable/ussuri - stable/victoria In lieu of any volunteers to maintain the CI, my recommendation for all the branches listed above is that they be marked EOL. If someone wants to volunteer to maintain CI for those branches, they can propose one of the below paths be taken instead: 1 - Someone volunteers to maintain these branches, and also report the status of CI of these older branches periodically on the Ironic whiteboard and in Ironic meetings. If you feel strongly that one of these branches needs to continue to be in service; volunteering in this way is how to save them. 2 - We seriously reduce CI. Basically removing all tempest tests to ensure that CI remains reliable and able to merge emergency or security fixes when needed. In some cases; this still requires CI fixes as some older inspector branches are failing *installing packages* in unit tests. I would still like, in this case, that someone volunteers to ensure the minimalist CI remains happy. My intention is to let this message serve as notice and a waiting period; and if I've not heard any response here or in Monday's Ironic meeting (in 6 days), I will begin taking action on retiring these branches. This is simply a start; other branches (including bugfix branches) are also in bad shape in CI, but getting these retired will significantly reduce the surface area of projects and branches to evaluate. I know it's painful to drop support for these branches; but we've provided good EM support for these branches for a long time and by pruning them away, we'll be able to save time to dedicate to other items. Thanks, Jay Faulkner

3 5

[magnum] Propose deprecation of Docker Swarm, CoreOS, fedora-ironic
by Jake Yip 28 Apr '23

28 Apr '23

Hi, Due to limited resources, Magnum team has decided to put most effort into supporting the most popular use case - Kubernetes on Fedora CoreOS. In light of that, we are seeking maintainers for the less well maintained drivers / COE - Docker Swarm, CoreOS and fedora-ironic drivers. The reasons are as follows: - Docker Swarm - It uses Fedora Atomic which has been EOL [1] - CoreOS - It is EOL [2] - fedora-ironic - Not tested, unsure if it is still working If anyone wants to take up maintenance of any of the above, we will be glad. Please let us know either through the ML or at the Magnum meetings. If there are no suitable maintainers, we will start the deprecation of these before end of Antelope. Regards, Jake [1]: https://projectatomic.io/blog/2019/11/fedora-atomic-host-nearing-eol/ [2]: https://www.redhat.com/en/technologies/cloud-computing/openshift/what-was-c… -- Jake Yip Magnum Team

2 3

Re: [all][stable][ptl] Propose to EOL Rocky series
by Elõd Illés 18 Apr '23

18 Apr '23

Hi, thanks for all the feedbacks from teams so far! @Zigo: Extended Maintenance process was created just for the same situation: to give space to interested parties to cooperate and keep things maintained even when stable releases are over their 'supported' lifetime. So it's good to see that there is interest in it! Unfortunately, with very old branches we've reached the state where gates can't be maintained and without a functional gate it's not safe to merge patches (yes, even security fixes) and they are just using resources (CI & maintainers' time). When gate is broken in such extent, then i think the community have to accept that it is not possible to merge patches confidently and needs to EOL that release. (Yes, it means that some vendors still have to do it for their own sake, testing it by their own, etc). Keeping things open in such state gives the false signal to vendors that 'it is maintained' however this is done without fully testing the changes. Another aspect is that code cannot be cleaned up until those old branches are still present (CI jobs, project configurations, etc) which gives pain for developers. So, however some vendors would appreciate probably to keep things open forever, for the community this is not beneficial and doable I think. Seeing that teams want to EOL their old branches is a proof that this is needed (unfortunately). Thanks, Előd Illés irc: elodilles @ #openstack-stable / #openstack-release ________________________________ From: Thomas Goirand <thomas(a)goirand.fr> Sent: Saturday, January 28, 2023 2:25 PM Cc: openstack-discuss(a)lists.openstack.org <openstack-discuss(a)lists.openstack.org> Subject: Re: [all][stable][ptl] Propose to EOL Rocky series Hi, I understand the gate is broken, however, Rocky is in Debian LTS, and I would like to keep the possibility to merge patches, even with the gate tests disabled. CVE-2022-47951 is an example why this is important... Thomas Goirand (zigo) On Jan 28, 2023 11:54, Radosław Piliszek <radoslaw.piliszek(a)gmail.com> wrote: > > Masakari is happy to EOL Rocky too. > > Radek > -yoctozepto > > On Fri, 27 Jan 2023 at 20:59, Jay Faulkner <jay(a)gr-oss.io> wrote: > > > > Thanks for doing all this cleanup work Elod. Ironic is OK with retirements of these shared resources up to Train. > > > > - > > Jay Faulkner > > > > On Fri, Jan 27, 2023 at 10:12 AM Elõd Illés <elod.illes(a)est.tech> wrote: > >> > >> Hi, > >> > >> Similarly like the Queens branch EOL proposal [1] I would like to propose > >> to transition every project's stable/rocky to End of Life: > >> > >> - gates are mostly broken > >> - minimal number of activity can be seen on this branch > >> - some core projects already transitioned their stable/rocky to EOL > >> recently (like ironic, neutron, nova) > >> - gate job definitions are still using the old, legacy zuul syntax > >> - gate jobs are based on Ubuntu Xenial, which is also beyond its public > >> maintenance window date and hard to maintain > >> > >> Based on the above, if there won't be any project who wants to keep open > >> their stable/rocky, then I'll start the process of EOL'ing Rocky stable > >> series as a whole. If anyone has any objection then please respond to > >> this mail. > >> > >> Thanks, > >> > >> Előd Illés > >> irc: elodilles @ #openstack-stable / #openstack-release > >> > >> [1] https://lists.openstack.org/pipermail/openstack-discuss/2022-October/031030… > >> >

3 4

DCN compute service goes down when a instance is scheduled to launch | wallaby | tripleo
by Swogat Pradhan 28 Mar '23

28 Mar '23

Hi, I currently have 3 compute nodes on edge site1 where i am trying to launch vm's. When the VM is in spawning state the node goes down (openstack compute service list), the node comes backup when i restart the nova compute service but then the launch of the vm fails. nova-compute.log 2023-02-26 08:15:51.808 7 INFO nova.compute.manager [req-bc0f5f2e-53fc-4dae-b1da-82f1f972d617 - - - - -] Running instance usage audit for host dcn01-hci-0.bdxworld.com from 2023-02-26 07:00:00 to 2023-02-26 08:00:00. 0 instances. 2023-02-26 08:49:52.813 7 INFO nova.compute.claims [req-3a1547ea-326f-4dd0-9127-7f4a4bdf1e45 b240e3e89d99489284cd731e75f2a5db 4160ce999a31485fa643aed0936dfef0 - default default] [instance: 0c62c1ef-9010-417d-a05f-4db77e901600] Claim successful on node dcn01-hci-0.bdxworld.com 2023-02-26 08:49:54.225 7 INFO nova.virt.libvirt.driver [req-3a1547ea-326f-4dd0-9127-7f4a4bdf1e45 b240e3e89d99489284cd731e75f2a5db 4160ce999a31485fa643aed0936dfef0 - default default] [instance: 0c62c1ef-9010-417d-a05f-4db77e901600] Ignoring supplied device name: /dev/vda. Libvirt can't honour user-supplied dev names 2023-02-26 08:49:54.398 7 INFO nova.virt.block_device [req-3a1547ea-326f-4dd0-9127-7f4a4bdf1e45 b240e3e89d99489284cd731e75f2a5db 4160ce999a31485fa643aed0936dfef0 - default default] [instance: 0c62c1ef-9010-417d-a05f-4db77e901600] Booting with volume c4bd7885-5973-4860-bbe6-7a2f726baeee at /dev/vda 2023-02-26 08:49:55.216 7 WARNING nova.cache_utils [req-3a1547ea-326f-4dd0-9127-7f4a4bdf1e45 b240e3e89d99489284cd731e75f2a5db 4160ce999a31485fa643aed0936dfef0 - default default] Cache enabled with backend dogpile.cache.null. 2023-02-26 08:49:55.283 7 INFO oslo.privsep.daemon [req-3a1547ea-326f-4dd0-9127-7f4a4bdf1e45 b240e3e89d99489284cd731e75f2a5db 4160ce999a31485fa643aed0936dfef0 - default default] Running privsep helper: ['sudo', 'nova-rootwrap', '/etc/nova/rootwrap.conf', 'privsep-helper', '--config-file', '/etc/nova/nova.conf', '--config-file', '/etc/nova/nova-compute.conf', '--privsep_context', 'os_brick.privileged.default', '--privsep_sock_path', '/tmp/tmpin40tah6/privsep.sock'] 2023-02-26 08:49:55.791 7 INFO oslo.privsep.daemon [req-3a1547ea-326f-4dd0-9127-7f4a4bdf1e45 b240e3e89d99489284cd731e75f2a5db 4160ce999a31485fa643aed0936dfef0 - default default] Spawned new privsep daemon via rootwrap 2023-02-26 08:49:55.717 2647 INFO oslo.privsep.daemon [-] privsep daemon starting 2023-02-26 08:49:55.722 2647 INFO oslo.privsep.daemon [-] privsep process running with uid/gid: 0/0 2023-02-26 08:49:55.726 2647 INFO oslo.privsep.daemon [-] privsep process running with capabilities (eff/prm/inh): CAP_SYS_ADMIN/CAP_SYS_ADMIN/none 2023-02-26 08:49:55.726 2647 INFO oslo.privsep.daemon [-] privsep daemon running as pid 2647 2023-02-26 08:49:55.956 7 WARNING os_brick.initiator.connectors.nvmeof [req-3a1547ea-326f-4dd0-9127-7f4a4bdf1e45 b240e3e89d99489284cd731e75f2a5db 4160ce999a31485fa643aed0936dfef0 - default default] Process execution error in _get_host_uuid: Unexpected error while running command. Command: blkid overlay -s UUID -o value Exit code: 2 Stdout: '' Stderr: '': oslo_concurrency.processutils.ProcessExecutionError: Unexpected error while running command. 2023-02-26 08:49:58.247 7 INFO nova.virt.libvirt.driver [req-3a1547ea-326f-4dd0-9127-7f4a4bdf1e45 b240e3e89d99489284cd731e75f2a5db 4160ce999a31485fa643aed0936dfef0 - default default] [instance: 0c62c1ef-9010-417d-a05f-4db77e901600] Creating image Is there a way to solve this issue? With regards, Swogat Pradhan

5 40

[all][ptg] Pre-PTG discussion: New Keystone Middleware Feature Supporting OAuth2.0 with External Authorization Service
by Hiromu Asahina 28 Mar '23

28 Mar '23

Hello Everyone, Recently, Tacker and Keystone have been working together on a new Keystone Middleware that can work with external authentication services, such as Keycloak. The code has already been submitted [1], but we want to make this middleware a generic plugin that works with as many OpenStack services as possible. To that end, we would like to hear from other projects with similar use cases (especially Ironic and Barbican, which run as standalone services). We will make a time slot to discuss this topic at the next vPTG. Please contact me if you are interested and available to participate. [1] https://review.opendev.org/c/openstack/keystonemiddleware/+/868734 -- Hiromu Asahina

4 11