September 2021 - openstack-discuss

[nova]is there support for discard/trim in virtio-blk?
by Sven Kieske 03 Sep '21

03 Sep '21

Hi, according to our research discard/trim is commonly documented as "not supported by virtio-blk" in the wider community and in openstack nova[0]. I understand that this was not supported by the kernel or qemu when discard support was initially implemented[1] in openstack, but times have changed. Virtio-blk in upstream Kernel[2] and in qemu[3] does clearly support discard/trim, which we discovered thanks to StackExchange[4]. So my question is, has someone successfully used trim/discard with virtio-blk in nova provisioned vms? [1] makes me guess, that there is no codepath that works with virtio-blk, but I'm not sure I understood all the codepaths in nova yet. we are still working our way through it, though. We currently use the train release in conjunction with ceph rbd volumes for testing this feature but weren't yet able to use it successfully. Looking at nova's master branch not much seems to have changed regarding trim support. If this is no supported configuration currently, should I submit a blueprint to enable this feature? Would there be other devs/ops interested in this? Any help or pointers would be appreciated. [0]: https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L… (nova debug log saying discard is unsupported for virtio) [1]: https://blueprints.launchpad.net/nova/+spec/cinder-backend-report-discard (initial implementation of discard support in openstack) [2]: https://github.com/torvalds/linux/commit/1f23816b8eb8fdc39990abe166c10a18c1… (available since linux 5.0-rc1) [3]: https://github.com/qemu/qemu/commit/37b06f8d46fe602e630e4bdce24e80a3e0f70cc2 (available since qemu 4.0.0) [4]: https://unix.stackexchange.com/a/518223 -- Mit freundlichen Grüßen / Regards Sven Kieske Systementwickler / systems engineer Mittwald CM Service GmbH & Co. KG Königsberger Straße 4-6 32339 Espelkamp Tel.: 05772 / 293-900 Fax: 05772 / 293-333 https://www.mittwald.de Geschäftsführer: Robert Meyer, Florian Jürgens St.Nr.: 331/5721/1033, USt-IdNr.: DE814773217, HRA 6640, AG Bad Oeynhausen Komplementärin: Robert Meyer Verwaltungs GmbH, HRB 13260, AG Bad Oeynhausen Informationen zur Datenverarbeitung im Rahmen unserer Geschäftstätigkeit gemäß Art. 13-14 DSGVO sind unter www.mittwald.de/ds abrufbar.

4 4

Unable to attach a Volume to Openstack WIndows instance
by KK CHN 03 Sep '21

03 Sep '21

List, I am trying to import a multi disk Windows VM ( 100GB bootable disk and 600 GB data volume disk) exported from HyperV to Openstack(Ussuri, Qemu-KVM, Glance image store, Cinder volume with Ceph backend). I am able to import the First bootable disk(vhdx converted to qcow2 ) with Windows VM in the OpenStack and its up and running able to login. ONE BASIC queston I not perfomed the *"Virt IO Injection steps"* as I am not sure whether for existing Windows VMs exported from HyperV need this Injection or not . Some one can clarify or defend this statement * " That Any Windows VM with Volume disks attached to it running on Other HyperVisor( say HyperV in my case, oVirt) exported to openstack which uses qemu-kvm needed the VirtIO Injection must done before importing to OpenStack with KVM for these Windows VMs for up and running and for attaching volume disks which also exported from other hypervisors . " OR only performance improvement only VirtIO injection do, so that I can avoid VirtIO injection for importing WIndowsVMs with multi disk to OpenStack.* used this step to import the converted bootable disk to my openstack. # openstack image create “Your_Windows_VM_NAME_Blaah” --file Windows_VM_disk1.qcow2 --disk-format qcow2 --container-format bare --public --property hw_firmware_type=uefi --property os_secure_boot=required --property hw_disk_bus=ide I am trying to import the second disk # openstack image create “Your_Windows_VM_NAME_Blah” --file Windows_VM_disk2.qcow2 --disk-format qcow2 --public and tried to create Volume through horizon dashboard with this image and 600 GB space. But volume creation itself showing status creating for more than 1 hour for now . Is this usual behaviour or some this wrong ? Note: so this volume creation from image is taking long time. As trial and error method I tried to create a small plain volume and attach this to the running WIndows VM(Windows2012 RC). But failed. I tried to create a plain 20 GB volume from the Horizon GUI and its created with in 30 seconds.. And tried to attach this volume to the Windows VM disk1 which is running and not got attached. ( I shutdown the Windows VM and tried to attach) same result. not attached. A plain volume also unable to got attached to the Windows VM ? why ? But the large 600GB goes for long hours. ( yesterday also tried and it was in creating state for 5 to 10 hours and I was waiting with no use, then I used the cinder command to change the volume status to "available" and then removed the volume which had not yet changed the state from "creating" to ``available" . Why do these volume creations with large disk space go for hours ? Are there any short steps to make it fast ? Any hints most

1 0

Replacing swift with RGW
by Michel Niyoyita 02 Sep '21

02 Sep '21

Hello , I tried to replace Swift RGW but I face some errors 1) the object storage doesnot appear to the dashboard . but it appears in systems information and also it shows the rgw where it points . [image: image.png] the second error I try to run swift command through openstack cli but fails (kolla-open) [stack@kolla-open ~]$ swift -v stat -bash: swift: command not found (kolla-open) [stack@kolla-open ~]$ sudo yum -y install python-swiftclient Last metadata expiration check: 0:59:21 ago on Thu 02 Sep 2021 09:21:53 AM CAT. No match for argument: python-swiftclient Error: Unable to find a match: python-swiftclient (kolla-open) [stack@kolla-open ~]$ . Kindly advise Michel

1 1

[kolla-ansible] Running Docker Swarm with Kuryr networking
by Paolo Celati 02 Sep '21

02 Sep '21

Hi, long story short I have a 3 node Openstack cluster that I manage with kolla-ansible, and I'd like to run Docker Swarm on that as well. I am aware Magnum exists, but I'd first like to get my head around this simpler case. Seeing as I'd like to connect Docker containers from swarm compose files to Neutron networks I'm trying to set up Kuryr together with a swarm configuration. However the documentation is a little scarce <https://docs.openstack.org/kolla-ansible/latest/reference/containers/kuryr-…> and I'd prefer running everything on these three hosts, including etcd. If I follow the guide and pass --cluster-store and --cluster-advertise arguments to dockerd then I can't run Docker in Swarm mode because I get an error saying Swarm is incompatible with those options, and at the same time it's not clear from documentation how you are expected to do Kuryr+Swarm. I did initialise the Swarm cluster before trying to add Kuryr, so I don't know if perhaps doing this the other way works? Do you have ideas or advice with this scenario? If worst comes to worst I can set up an external etcd cluster on a separate non-Openstack cluster but I'd rather avoid that. Thanks in advance, Paolo

4 9

[nova][cinder][barbican] storing secrets in service project
by Pavlo Shchelokovskyy 02 Sep '21

02 Sep '21

Hi all, When Barbican is involved, and services like Nova and Cinder create secrets behind the scenes themselves, this results in problems deleting or otherwise managing some resources by cloud admins. For example as described in https://bugs.launchpad.net/cinder/+bug/1775372, admin can not delete an encrypted volume created by another user, and needs to resort to adding herself to the project first with appropriate roles. This stems IMO from the difference in default resource access policies between Barbican and other services that consume it. For example, while in Nova or Cinder, cloud admin by default can delete servers or volumes in other projects, in Barbican by default only secret creator or admin "in the same project" can delete a secret. I came up with at least 3 solutions (already mentioned in the issue above) but would like to discuss their validity, pros and cons, and especially how they affect the security model and the use cases that this Nova-Cinder-Barbican integration aims to solve. 1. Just ignore failure to delete a secret and log appropriate warning that a secret may be left orphaned. Obviously not the best solution, but rather simple one, also will protect from cases when for whatever reason a secret will be deleted from Barbican directly (e.g. by mistake), rendering volume undeletable for anyone. 2. Change default barbican policies to allow secret deletion for cloud admin (system scope?). Naturally has implications for the Barbican model for resources access, so may be not that appropriate. 3. Store the secrets in the service project instead of user's. Currently those secrets are generated by Nova or Cinder themselves, and user quite probably is not even aware that there was something created in Barbican on her behalf. The user also has no option to pass her own secrets to any of the involved API calls, so this is something happening completely behind the scenes. So do we really need to store these secrets in the user's project? Wouldn't it be better to store them in the service project, have the same service user defined in Nova and Cinder, so it can operate and access those secrets regardless of who manages the resource which depends on access to this secret? This also may solve the problem of encrypted local instance images in Nova, where upon host restart, Nova is not able to bring those instances up because it lacks the context capable of accessing those secrets. I think the third option is the best one, but I may be missing the actual use case meant behind this feature. Is it "keep the data encrypted at rest and in transit" or more strictly "encrypt data of different users with different keys and limit access to those keys" (for the case of breach etc) ? In the first case it seems storing all the auto-generated secrets in a service project would suffice... Please help me understand the best approach to take. Best regards, Pavlo.

4 4

[neutron] Drivers meeting - Friday 3.09.2021 - cancelled
by Slawek Kaplonski 02 Sep '21

02 Sep '21

Hi, Due to lack of agenda, let's cancel tomorrow's drivers meeting. See You on the meeting next week. -- Slawek Kaplonski Principal Software Engineer Red Hat

1 0

[tc][all]Please help to identify common pain points & video meeting scheduled next week
by Rico Lin 02 Sep '21

02 Sep '21

Dear all First, I thank Kendall and everyone who help with the Pain point elimination etherpad [1]. Two paths we plan to work on or would like to see how we can push it forward: 1. To identify common pain points from OpenStack. And see if we can target them in the following cycles. 2. To see if we can have a community-wide goal to encourage project teams to pick up their own pain point in the following cycles. *This ML is focused on the first path; To identify common pain points.* So I would like to encourage all to help to identify common pain points from it and also help put your feedbacks on the etherpad [1]. To identify common pain points, you can help to leave comments to pain points that you feel it is not an issue for a single project team but for a larger scope. We kind of recognized rabbitmq (or said RPC backend) might be one of the common pain points from that etherpad. And we wish to learn more from all of you. And we need operators' feedback mostly. Also, *we plan to put `identify common pain points` on schedule for the TC meeting next week (which is a video meeting on google meet). If you're interested, please help with etherpad, and join the TC meeting [2] next week.* [1] https://etherpad.opendev.org/p/pain-point-elimination [2] https://wiki.openstack.org/wiki/Meetings/TechnicalCommittee *Rico Lin* OIF Individual Board of directors, OpenStack TC, Multi-arch SIG chair, Heat PTL, Senior Software Engineer@EasyStack

1 1

[Ironic] No weekly meeting on Sep 06
by Iury Gregory 02 Sep '21

02 Sep '21

Hello Ironicers! During our weekly meeting this week (Aug 30) we decided to skip our next weekly meeting since it is a holiday in the US -- *Att[]'sIury Gregory Melo Ferreira * *MSc in Computer Science at UFCG* *Part of the ironic-core and puppet-manager-core team in OpenStack* *Software Engineer at Red Hat Czech* *Social*: https://www.linkedin.com/in/iurygregory *E-mail: iurygregory(a)gmail.com <iurygregory(a)gmail.com>*

1 0

[magnum] [kolla-ansible] [kayobe] [Victoria]
by Tony Pearce 02 Sep '21

02 Sep '21

Testing out Kubernetes with Magnum project, deployed via kayobe on Victoria we have deployed an auto scaling cluster and have run into a problem and I'm not sure how to proceed. I understand that the cluster tried to scale up but the openstack project did not have enough CPU resources to accommodate it (error= Quota exceeded for cores: Requested 4, but already used 20 of 20 cores). So the situation is that the cluster shows "healthy" and "UPDATE_FAILED" but also kubectl commands are failing [1]. What is required to return the cluster back to a working status at this point? I have tried: - cluster resize to reduce number of workers - cluster resize to increase number of workers after increasing project quota - cluster resize and maintaining the same number of workers When trying any of the above, horizon shows an immediate error "Unable to resize given cluster" but magnum logs and heat logs do not show any log update at all at that time. - using "check stack" and resume stack in the stack horizon menu gives this error [2] Investigating the kubectl issue, it was noted that some services had failed on the master node [3]. Manual start as well as reboot the node did not bring up the services. Unfortunately I dont have ssh access to the master and no further information has been forthcoming with regards to logs for those service failures so I am unable to provide anything around that here. I found this link [4] so I decided to delete the master node then run "check" cluster again but the check cluster just fails in the same way except this time it fails saying that it cannot find the master [5] while the previous error was that it could not find a node. Ideally I would prefer to recover the cluster - whether this is still possible I am unsure. I can probably recreate this scenario again. What steps should be performed in this case to restore the cluster? [1] kubectl get no Error from server (Timeout): the server was unable to return a response in the time allotted, but may still be processing the request (get nodes) [2] Resource CHECK failed: ["['NotFound: resources[4].resources.kube-minion: Instance None could not be found. (HTTP 404) (Request-ID: req-6069ff6a-9eb6-4bce-bb25-4ef001ebc428)']. 'CHECK' not fully supported (see resources)"] [3] [systemd] Failed Units: 3 etcd.service heat-container-agent.service logrotate.service [4] https://bugzilla.redhat.com/show_bug.cgi?id=1459854 [5] ["['NotFound: resources.kube_masters.resources[0].resources.kube-master: Instance c6185e8e-1a98-4925-959b-0a56210b8c9e could not be found. (HTTP 404) (Request-ID: req-bdfcc853-7dbb-4022-9208-68b1ab31008a)']. 'CHECK' not fully supported (see resources)"]. Kind regards, Tony Pearce

3 6

[all][tc] Technical Committee next weekly meeting on Sept 2nd at 1500 UTC
by Ghanshyam Mann 02 Sep '21

02 Sep '21

Hello Everyone, Technical Committee's next weekly meeting is scheduled for Sept 2nd at 1500 UTC. If you would like to add topics for discussion, please add them to the below wiki page by Wednesday, Sept 1st, at 2100 UTC. https://wiki.openstack.org/wiki/Meetings/TechnicalCommittee#Next_Meeting -gmann

2 2