January 2023 - openstack-discuss - lists.openstack.org

[ops] QOS on flavor breaking live migration from CentOS 7 to 8
by Jahson Babel 04 Jan '23

04 Jan '23

Hello, I'm trying to live migrate some VMs from CentOS 7 to Rocky 8. Everything run smoothly when there is no extra specs on flavors but things getting more complicated when those are fixed. Especially when using quota:vif_burst for QOS. I know that we aren't supposed to use this for QOS now but it's an old cluster and it was done that way at the time. So VMs kinda have all those specs tied to them. When live migrate a VM this show up in the nova's logs : driver.py _live_migration_operation nova.virt.libvirt.driver Live Migration failure: internal error: Child process (tc class add dev tapxxxxxxxx-xx parent 1: classid 1:1 htb rate 250000kbps ceil 2000000kbps burst 60000000kb quantum 21333) unexpected exit status 1: Illegal "burst" This bug cover the problem : https://bugs.launchpad.net/nova/+bug/1960840 So it's seems to be a normal behavior. Plus I forgot to mention that I'm on OpenStack Train version and the file mentioned in the launchpad is not present for this version. By using Rocky 8 I have to use an updated libvirt that won't accept the burst parameter we used to set. All available versions of libvirt on Rocky 8 have changed behavior concerning the burst parameter. I've done some testing to make things works including removing the extra_specs on flavors and in the DB, removing it through libvirt and trying to modify tc rules used by a VM but it didn't worked. I have not tried yet to patch Nova or Libvirt but I don't really know where to look for. The only thing that did work was to resize the VM to an identical flavor without the extra_specs. But this induce a complete reboot of the VM. I would like, if possible, to be able to live migrate the VMs which is quite easier. Is it possible to remove the extra_specs on the VMs and then live migrate ? Or should I just plan to resize/reboot all VMs without those extra_specs ? Any advise will be appreciated. Thank you for any help, Best regards. Jahson

2 2

Re: openstack-discuss Digest, Vol 51, Issue 6
by Kerem Çeliker 04 Jan '23

04 Jan '23

Hello Stephen, Yes, OpenStack does support USB passthrough. You can pass through USB devices to instances running in OpenStack by using the nova.virt.libvirt.vif.LibvirtGenericVIFDriver driver and the qemu:commandline option in the nova.conf file. Best, Kerem Çeliker keremceliker.medium.com Sent from my iPhone > On 4 Jan 2023, at 02:25, openstack-discuss-request(a)lists.openstack.org wrote: > > Send openstack-discuss mailing list submissions to > openstack-discuss(a)lists.openstack.org > > To subscribe or unsubscribe via the World Wide Web, visit > https://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-discuss > > or, via email, send a message with subject or body 'help' to > openstack-discuss-request(a)lists.openstack.org > > You can reach the person managing the list at > openstack-discuss-owner(a)lists.openstack.org > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of openstack-discuss digest..." > > > Today's Topics: > > 1. Re: [nova] Do openstack support USB passthrough (Sean Mooney) > 2. Re: [ops] QOS on flavor breaking live migration from CentOS 7 > to 8 (Sean Mooney) > 3. Re: [cloudkitty] Instances billing based on tags (Satish Patel) > 4. [nova][keystone] workload identity? (Mohammed Naser) > 5. Re: [cloudkitty] Instances billing based on tags > (Rafael Weing?rtner) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Tue, 03 Jan 2023 19:03:23 +0000 > From: Sean Mooney <smooney(a)redhat.com> > To: Stephen Finucane <stephenfin(a)redhat.com>, ??? > <hanguangyu2(a)gmail.com>, openstack-discuss > <openstack-discuss(a)lists.openstack.org> > Subject: Re: [nova] Do openstack support USB passthrough > Message-ID: > <765850d1c170a510d34dafb4253ab97528829351.camel(a)redhat.com> > Content-Type: text/plain; charset="UTF-8" > >> On Tue, 2023-01-03 at 14:54 +0000, Stephen Finucane wrote: >>> On Mon, 2022-12-26 at 10:04 +0000, ??? wrote: >>> Hi, all >>> >>> I want to ask if openstack support USB passthrough now? >>> >>> Or if I want the instance to recognize the USB flash drive on the >>> host, do you have any suggestions? >>> >>> Thanks, >>> Han >> >> This isn't supported in nova and probably never will be. The closest you can get >> is to passthrough an entire USB controller as suggested by this blog [1], but >> that's really a hack and I 100% would not use it in production. > > ya so we have discussed usb passthough supprot a few times and its somethign nova could add but there has > neither been the demand or desire to add it stongly enough in the core team to actully do it. > > the shorted path to enableing usb passthoug would likely be to add support to cyborg and then add support for that ot nova. > i am perhaps the most open of the nova cores to supporting usb passthough since i have wanted to add it in the past but > if we were to support it it would have to be similar to howe we support pci passhtough. static provisioning and likely only of > stateless devices which rules out usb falsh drives. > > usb gps recivers for precision time stamping was one of the usecause raised in the past which we were somewhat open too > the other was usb programmers/debuggers forh cases when vms where used in industral test and automation workflows. > > as stephen said the only way to do it today is to piggyback on pci passthough and passhtough a usb contoller not a single device. > > if we were to ever support this in nova directly i would proably extend the pci tracker or support other buses like usb or use the generic > resouce table created for persistent memory to model the devices. in eitehr case we would want this capablity to be placement native from the > start if we added this capablity so it would be more and less work then you might imagine to do this right. > less work if we maintain the requirement for statless devices only (ie no usb flash drives) more if you also need to handel multi tenancy and > move operation include data copying, erasure and or encypetion. > > i would not expect this to change in the next few release unless multiple operators provide feedback that this is a broadly deired capablity. > with out a top level generic device api for mutple type of devices (vgpu, usb, pci) that was decoupled form the flaovr or an abstraction like > the cyborg device-profile or pci alias it is hard to see a clean way to model this in our api. that is why enabling it in cyborg and then extneding > nova ot support device profiles with a device type of usb is the simplar solution form a nova perspecitve but that is non trivial from an operational > perspective as you requrie cyborg to utilise the feature. doing it via a usb_alias in the flavor has all the draw backs of the pci_alias, static > configuration that must match on all compute nodes and futher proliferation of flavor explostion. this is one of the reasons we have not added this in > the past. the work to do it in the libvirt driver would not be hard but the maintaince and operational overhead of using it for operators is non > trivial. > >> >> Stephen >> >> [1] https://egallen.com/openstack-usb-passthrough/ >> >> > > > > > ------------------------------ > > Message: 2 > Date: Tue, 03 Jan 2023 19:20:50 +0000 > From: Sean Mooney <smooney(a)redhat.com> > To: Jahson Babel <jahson.babel(a)cc.in2p3.fr>, > openstack-discuss(a)lists.openstack.org > Subject: Re: [ops] QOS on flavor breaking live migration from CentOS 7 > to 8 > Message-ID: > <b0f13cde177cd8b5f7e6e5b20445decab7dbc8b8.camel(a)redhat.com> > Content-Type: text/plain; charset="UTF-8" > > hi yes this is a know issue. > > so the simple answer is resize all affected vms instead of live migrating them > the longer answer is we have been dissing this internally at redhat on and off for > some time now. > https://bugs.launchpad.net/nova/+bug/1960840 is one example where this happens. > > there is another case for the cpu based quotas that happens when going form rhel/centos 8->9 > basically in the 8->9 change the cgroups implemantion changes form v1 to v2 > https://bugzilla.redhat.com/show_bug.cgi?id=2035518 > > when adressing that we did not have a good universal solution for instnace that hardcoded a value that > was incompatible with the cgroups_v2 api in the kernel except resize. > > in https://review.opendev.org/c/openstack/nova/+/824048/ we removed automatically adding the > cpu_shares cgroup option to enable booting vms with more then 8 cpus > > we did not come up with any option other then resize for the other quotas that were in a similar situation. > the one option that we considerd possibel to do was extend nova-mange to allow the embeded flaour to be updated > this would be similar to what we did to enable the image property to be modifed for chaing machine types. > > https://docs.openstack.org/nova/latest/cli/nova-manage.html#image-property-… > > we didcussed at the time that while we did not want to allow falvor extra specs to be modifed we might recondier that > if the quota issue forced our hand or we had a similar need due to foces beyond our contol. i.e. we needed to provide a way beyond > resize e.g. due ot operating system changes. what make image properties and flavor extra spec different is that image proerties can > only be updated by rebuild which is a destructive operation. extra specs are upsted by resize which is not a destructive operation. > that is one of the reasons we have special considertion to image properties and did not do the same for extra specs. > > if we allow the same for flavor extra specs you would still have to stop the instance, make the change and then migrate the instnace > resize automates that so it is generall a better fit. we were also conceren that adding it to nova manage would result in it being abused > to modify instnace in ways that were either invalid for the host(changing the numa toplogy, adding traits/resouce request not trackedcxd in placemnt) > or otherwise break the instnace in weird ways. that could happen via image properites too but its less likely. > > > >> On Tue, 2023-01-03 at 17:25 +0100, Jahson Babel wrote: >> Hello, >> >> I'm trying to live migrate some VMs from CentOS 7 to Rocky 8. >> Everything run smoothly when there is no extra specs on flavors but >> things getting more complicated when those are fixed. Especially when >> using quota:vif_burst for QOS. >> I know that we aren't supposed to use this for QOS now but it's an old >> cluster and it was done that way at the time. So VMs kinda have all >> those specs tied to them. >> >> When live migrate a VM this show up in the nova's logs : >> driver.py _live_migration_operation nova.virt.libvirt.driver? Live >> Migration failure: internal error: Child process (tc class add dev >> tapxxxxxxxx-xx parent 1: classid 1:1 htb rate 250000kbps ceil >> 2000000kbps burst 60000000kb quantum 21333) unexpected exit status 1: >> Illegal "burst" >> This bug cover the problem : https://bugs.launchpad.net/nova/+bug/1960840 >> So it's seems to be a normal behavior. Plus I forgot to mention that I'm >> on OpenStack Train version and the file mentioned in the launchpad is >> not present for this version. >> By using Rocky 8 I have to use an updated libvirt that won't accept the >> burst parameter we used to set. All available versions of libvirt on >> Rocky 8 have changed behavior concerning the burst parameter. >> >> I've done some testing to make things works including removing the >> extra_specs on flavors and in the DB, removing it through libvirt and >> trying to modify tc rules used by a VM but it didn't worked. >> I have not tried yet to patch Nova or Libvirt but I don't really know >> where to look for. >> The only thing that did work was to resize the VM to an identical flavor >> without the extra_specs. But this induce a complete reboot of the VM. I >> would like, if possible, to be able to live migrate the VMs which is >> quite easier. >> >> Is it possible to remove the extra_specs on the VMs and then live >> migrate ? Or should I just plan to resize/reboot all VMs without those >> extra_specs ? >> Any advise will be appreciated. >> >> Thank you for any help, >> Best regards. >> >> Jahson > > > > > ------------------------------ > > Message: 3 > Date: Tue, 3 Jan 2023 15:23:19 -0500 > From: Satish Patel <satish.txt(a)gmail.com> > To: Rafael Weing?rtner <rafaelweingartner(a)gmail.com> > Cc: OpenStack Discuss <openstack-discuss(a)lists.openstack.org> > Subject: Re: [cloudkitty] Instances billing based on tags > Message-ID: > <CAPgF-fpGyO-Z8bt7KMdP7CAhi9kne3136ptoE+8coURyFwi7yw(a)mail.gmail.com> > Content-Type: text/plain; charset="utf-8" > > Wow! very interesting, I will poke around and see how it's feasible. Very > curious how it will represent that data in horizon UI. Currently I am > seeing rates based on project_id so assuming it will show based on > customer_id. correct? > > > On Tue, Jan 3, 2023 at 8:13 AM Rafael Weing?rtner < > rafaelweingartner(a)gmail.com> wrote: > >> You can do that. Basically, you can start collecting those attributes you >> want for billing (e.g. tags) via Ceilometer dynamic pollster (that is the >> easiest way to achieve this). Then, you need to configure the resource type >> in Gnocchi to store this extra attribute, and of course, configure >> CloudKitty to collect/use it. Both in the metrics.yml and then in the >> hashmap or Pyscript rules. >> >>> On Mon, Jan 2, 2023 at 11:01 PM Satish Patel <satish.txt(a)gmail.com> wrote: >>> >>> Folks, >>> >>> We have an AWS project and in a single project we run multiple customers >>> so for billing we use tags. In short every vm instance has a tag (customer >>> name) and that way we can produce bills for each customer. >>> >>> Recently I am playing with openstack cloudkitty and it works with >>> standard cases like project based billing. But does it support tags based >>> billing just similar to what i have explained in the above aws example? >>> >>> >>> >> >> -- >> Rafael Weing?rtner >> >

1 0

[cinder] Bug Report from 01-04-2023
by Sofia Enriquez 04 Jan '23

04 Jan '23

This is a bug report from 12-21-2022 to 01-04-2023. Agenda: https://etherpad.opendev.org/p/cinder-bug-squad-meeting ----------------------------------------------------------------------------------------- High - https://bugs.launchpad.net/cinder/+bug/2000436 "250 unit test failures under Python 3.11." Unassigned. - https://bugs.launchpad.net/cinder/+bug/2000489 "Dell Powerstore won't delete volume when storage migrate." Unassigned. Medium - https://bugs.launchpad.net/cinder/+bug/2000724 "cinder should not send external events when extending online is called by glance." In Progress. Cheers, -- Sofía Enriquez she/her Software Engineer Red Hat PnT <https://www.redhat.com> IRC: @enriquetaso @RedHat <https://twitter.com/redhat> Red Hat <https://www.linkedin.com/company/red-hat> Red Hat <https://www.facebook.com/RedHatInc> <https://www.redhat.com>

1 0

2023 PTGs + Virtual Team Signup
by Kendall Nelson 04 Jan '23

04 Jan '23

Hello Everyone! The next PTG on the calendar will be our usual virtual format and it will take place March 27 -31 right after the 2023.1 release. The project signup survey is already ready to go! If your team is interested in participating, sign them up here[1]! Registration is also open now[2] We have also heard many requests from the contributor community to bring back in person events and also heard that with global travel reduction, it would be better travel wise for the community at large to combine it with the OpenInfra Summit[3]. So, an in-person PTG will take place Wednesday and Thursday! June 14 -15th. OpenInfra Summit registration will be required to participate, and contributor pricing and travel support[4] will be available. We will also be planning another virtual PTG in the latter half of the year. The exact date is still being finalized. This will give you three opportunities to get your team together in 2023: two virtual and one in-person. - Virtual: March 27-31 - In- Person: June 14-15 collocated with the OpenInfra Summit - Virtual: TBD, but second half of the year As any project in our community, you are free to take advantage of all or some of the PTGs this coming year. I, for one, hope to see you and your project at as many as make sense for your team! To make sure we are as inclusive as possible I will continue to encourage team moderators to write summaries of their discussions to gather and promote after the events as well. Stay tuned for more info as we get closer, but the team signup survey[1] has opened for the first virtual PTG and is ready for you! -Kendall (diablo_rojo) [1] https://openinfrafoundation.formstack.com/forms/march2023_vptg_survey [2] https://openinfra-ptg.eventbrite.com [3]https://openinfra.dev/summit/vancouver-2023 [4] https://openinfrafoundation.formstack.com/forms/openinfra_tsp

1 1

[all][tc] Technical Committee next weekly meeting on 2023 Jan 4 at 1600 UTC
by Ghanshyam Mann 04 Jan '23

04 Jan '23

Hello Everyone, The technical Committee's next weekly meeting is scheduled for 2023 Jan 4, at 1600 UTC. If you would like to add topics for discussion, please add them to the below wiki page by Tuesday, Jan 3 at 2100 UTC. https://wiki.openstack.org/wiki/Meetings/TechnicalCommittee#Next_Meeting -gmann

1 1

[cloudkitty] Instances billing based on tags
by Satish Patel 03 Jan '23

03 Jan '23

Folks, We have an AWS project and in a single project we run multiple customers so for billing we use tags. In short every vm instance has a tag (customer name) and that way we can produce bills for each customer. Recently I am playing with openstack cloudkitty and it works with standard cases like project based billing. But does it support tags based billing just similar to what i have explained in the above aws example?

2 3

[nova][keystone] workload identity?
by Mohammed Naser 03 Jan '23

03 Jan '23

Hi folks: I’m wondering if there’s anyone who’s had some thought or perhaps some work/progress/thoughts on workload identity (example service accounts for VMs) — is that something that’s really far away for us? Has someone thought about outlining what’s needed? Thanks Mohammed -- Mohammed Naser VEXXHOST, Inc.

1 0

[nova][placement] Specs freeze will be on Jan 12th
by Sylvain Bauza 03 Jan '23

03 Jan '23

Hey, As we didn't have a quorum for today's meeting, I prefer to move our deadline for freezing the Antelope accepted specs by one week, instead of this Thursday. This means we will continue to review open specs until January the 12th. That being said, as a reminder, our Feature Freeze is on February 16th [1] which means we will only have *five* weeks for reviewing all the implementations, so please start to work on your feature branch and don't wait for your spec to be accepted or we wouldn't have time for reviewing all the open features. Thanks, -Sylvain [1] https://releases.openstack.org/antelope/schedule.html#a-ff

1 0

Kolla-ansible Yoga Magnum csi-cinder-controllerplugin-0 CrashLoopBackOff
by Oliver Weinmann 03 Jan '23

03 Jan '23

Dear all, I have a strange issue with Magnum on Yoga deployed by kolla-ansible. I noticed this on a prod cluster and so I deployed a fresh cluster from scratch and here I'm facing the very same issue. When deploying a a new K8s cluster the csi-cinder-controllerplugin-0 pod is in CrashLoopBackOff state. kubectl get pods -A NAMESPACE NAME READY STATUS RESTARTS AGE kube-system coredns-56448757b9-62lxw 1/1 Running 0 11d kube-system coredns-56448757b9-6mvtr 1/1 Running 0 11d kube-system csi-cinder-controllerplugin-0 4/5 CrashLoopBackOff 3339 (92s ago) 11d kube-system csi-cinder-nodeplugin-88ttn 2/2 Running 0 11d kube-system csi-cinder-nodeplugin-tnxpt 2/2 Running 0 11d kube-system dashboard-metrics-scraper-67f57ff746-2vhgb 1/1 Running 0 11d kube-system k8s-keystone-auth-5djxh 1/1 Running 0 11d kube-system kube-dns-autoscaler-6d5b5dc777-mm7qs 1/1 Running 0 11d kube-system kube-flannel-ds-795hj 1/1 Running 0 11d kube-system kube-flannel-ds-p76rf 1/1 Running 0 11d kube-system kubernetes-dashboard-7b88d986b4-5bhnf 1/1 Running 0 11d kube-system magnum-metrics-server-6c4c77844b-jbpml 1/1 Running 0 11d kube-system npd-sqsbr 1/1 Running 0 11d kube-system openstack-cloud-controller-manager-c5prb 1/1 Running 0 11d I searched the mailing list and noticed that I reported the very same issue last year: https://lists.openstack.org/pipermail/openstack-discuss/2022-May/028517.html but then I was either able to fix it or not able to reproduce it. I searched the web and found other users having the same issue: https://storyboard.openstack.org/#!/story/2010023 https://github.com/kubernetes/cloud-provider-openstack/issues/1845 They suggest to change the csi_snapshotter_tag to v4.0.0. I haven't tried it yet, but I have the feeling that something in my deployment is not correct since it wants to use v1.2.2 which is a very old version. Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Pulling 26m (x3382 over 12d) kubelet Pulling image "quay.io/k8scsi/csi-snapshotter:v1.2.2" Warning BackOff 81s (x79755 over 12d) kubelet Back-off restarting failed container kubectl get nodes -o wide NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME k8s-admin-test-old-ygahdfhciitx-master-0 Ready master 12d v1.23.3 10.0.0.132 172.28.4.124 Fedora CoreOS 35.20220410.3.1 5.16.18-200.fc35.x86_64 docker://20.10.12 k8s-admin-test-old-ygahdfhciitx-node-0 Ready <none> 12d v1.23.3 10.0.0.160 172.28.4.128 Fedora CoreOS 35.20220410.3.1 5.16.18-200.fc35.x86_64 docker://20.10.12 I deployed on Rocky Linux 8.7 using latest Kolla-ansible for Yoga: [vagrant@seed ~]$ kolla-ansible --version 14.7.1 [vagrant@seed ~]$ cat /etc/os-release NAME="Rocky Linux" VERSION="8.7 (Green Obsidian)" ID="rocky" ID_LIKE="rhel centos fedora" VERSION_ID="8.7" PLATFORM_ID="platform:el8" PRETTY_NAME="Rocky Linux 8.7 (Green Obsidian)" ANSI_COLOR="0;32" LOGO="fedora-logo-icon" CPE_NAME="cpe:/o:rocky:rocky:8:GA" HOME_URL="https://rockylinux.org/" BUG_REPORT_URL="https://bugs.rockylinux.org/" ROCKY_SUPPORT_PRODUCT="Rocky-Linux-8" ROCKY_SUPPORT_PRODUCT_VERSION="8.7" REDHAT_SUPPORT_PRODUCT="Rocky Linux" REDHAT_SUPPORT_PRODUCT_VERSION="8.7" Can you please help or clarify? Best Regards, Oliver

1 0

[cloudkitty][horizon] change service name list in GUI
by Satish Patel 03 Jan '23

03 Jan '23

Folks, I am trying to configure rating for network data bytes in/out but what I have seen is that the ceilometer uses a different name than what cloudkitty GUI drop down-menu has. Ceilometer using "network.outgoing.bytes" and "network.incoming.bytes" Cloudkitty GUI drop-down list has "network.outgoing.bytes.rate" & "network.incoming.bytes.rate" I have tried to change it in cloudkitty metrics.yml file to match it with the ceilometer but the rating is not calculating until I change that in cloudkitty. Please advise how to change it in cloudkitty.

2 1