March 2021 - openstack-discuss - lists.openstack.org

[ops][cinder][nova] os-brick upcoming releases
by Brian Rosmaita 12 Apr '21

12 Apr '21

Hello operators, You may have heard about a potential data-loss bug [0] that was recently discovered. It has been fixed in the upcoming wallaby release and we are planning to backport to all stable branches and do new os-brick releases from the releasable stable branches. In the meantime, the bug occurs if the multipath configuration option on a compute is changed while volumes are attached to instances on that compute. The possible data loss may occur when the volumes are detached (migration, volume-detach, etc.). Thus, before the new os-brick releases are available, the issue can nonetheless be averted by not making such a configuration change under those circumstances. The new os-brick releases will be: - victoria: 4.0.3 - ussuri: 3.0.6 - train: 2.10.6 The stein, rocky, and queens branches are in Extended Maintenance mode and are no longer released from, but critical fixes are backported to them when possible, though it may take a while before these are merged. [0] https://launchpad.net/bugs/1921381

1 1

[dev][cinder][keystone] Properly consuming system-scope in cinder
by Lance Bragstad 09 Apr '21

09 Apr '21

Hey folks, As I'm sure some of the cinder folks are aware, I'm updating cinder policies to include support for some default personas keystone ships with. Some of those personas use system-scope (e.g., system-reader and system-admin) and I've already proposed a series of patches that describe what those changes look like from a policy perspective [0]. The question now is how we test those changes. To help guide that decision, I worked on three different testing approaches. The first was to continue testing policy using unit tests in cinder with mocked context objects. The second was to use DDT with keystonemiddleware mocked to remove a dependency on keystone. The third also used DDT, but included changes to update NoAuthMiddleware so that it wasn't as opinionated about authentication or authorization. I brought each approach in the cinder meeting this week where we discussed a fourth approach, doing everything in tempest. I summarized all of this in an etherpad [1] Up to yesterday morning, the only approach I hadn't tinkered with manually was tempest. I spent some time today figuring that out, resulting in a patch to cinderlib [2] to enable a protection test job, and cinder_tempest_plugin [3] that adds the plumbing and some example tests. In the process of implementing support for tempest testing, I noticed that service catalogs for system-scoped tokens don't contain cinder endpoints [4]. This is because the cinder endpoint contains endpoint templating in the URL [5], which keystone will substitute with the project ID of the token, if and only if the catalog is built for a project-scoped token. System and domain-scoped tokens do not have a reasonable project ID to use in this case, so the templating is skipped, resulting in a cinder service in the catalog without endpoints [6]. This cascades in the client, specifically tempest's volume client, because it can't find a suitable endpoint for request to the volume service [7]. Initially, my testing approaches were to provide examples for cinder developers to assess the viability of each approach before committing to a protection testing strategy. But, the tempest approach highlighted a larger issue for how we integrate system-scope support into cinder because of the assumption there will always be a project ID in the path (for the majority of the cinder API). I can think of two ways to approach the problem, but I'm hoping others have more. First, we remove project IDs from cinder's API path. This would be similar to how nova (and I assume other services) moved away from project-specific URLs (e.g., /v3/%{project_id}s/volumes would become /v3/volumes). This would obviously require refactoring to remove any assumptions cinder has about project IDs being supplied on the request path. But, this would force all authorization information to come from the context object. Once a deployer removes the endpoint URL templating, the endpoints will populate in the cinder entry of the service catalog. Brian's been helping me understand this and we're unsure if this is something we could even do with a microversion. I think nova did it moving from /v2/ to /v2.0/, which was technically classified as a major bump? This feels like a moon shot. Second, we update cinder's clients, including tempest, to put the project ID on the URL. After we update the clients to append the project ID for cinder endpoints, we should be able to remove the URL templating in keystone, allowing cinder endpoints to appear in system-scoped service catalogs (just like the first approach). Clients can use the base URL from the catalog and append the admin project ID before putting the request on the wire. Even though the request has a project ID in the path, cinder would ignore it for system-specific APIs. This is already true for users with an admin role on a project because cinder will allow you to get volumes in one project if you have a token scoped to another with the admin role [8]. One potential side-effect is that cinder clients would need *a* project ID to build a request, potentially requiring another roundtrip to keystone. Thoughts? [0] https://review.opendev.org/q/project:openstack/cinder+topic:secure-rbac [1] https://etherpad.opendev.org/p/cinder-secure-rbac-protection-testing [2] https://review.opendev.org/c/openstack/cinderlib/+/772770 [3] https://review.opendev.org/c/openstack/cinder-tempest-plugin/+/772915 [4] http://paste.openstack.org/show/802117/ [5] http://paste.openstack.org/show/802097/ [6] https://opendev.org/openstack/keystone/src/commit/c239cc66615b41a0c09e031b3… [7] http://paste.openstack.org/show/802092/ [8] http://paste.openstack.org/show/802118/

4 8

[puppet] Proposing Alan Bishop (abishop) for puppet-cinder core and puppet-glance core
by Takashi Kajinami 08 Apr '21

08 Apr '21

Hello, I'd like to propose Alan Bishop (abishop) for the core team of puppet-cinder and puppet-glance. Alan has been actively involved in these 2 modules for a few years and has implemented some nice features like multiple backend support in glance, cinder s3 backup driver and etc, which expanded adoption of puppet-openstack. He has also provided good reviews on patches for these 2 repos based on his understanding about our code, puppet and serverspec. He is an active contributor to cinder and has deep knowledge about it. In addition He is also a core review in TripleO, which consumes our puppet modules, and mainly covers storage components like cinder and glance, so he is familiar with the way how these two components are deployed and configured. I believe adding him to our board helps us improve our review of these two modules. I'll wait for one week to hear any feedback from other core reviewers. Thank you, Takashi

6 6

[cinder] Review of tiny patch to add Ceph RBD fast-diff to cinder-backup
by Christian Rohmann 07 Apr '21

07 Apr '21

Hey everyone, I wrote a tiny patch to add the Ceph RDB feature of fast-diff to backups created by cinder-backup: * https://review.opendev.org/c/openstack/cinder/+/766856/ Could someone please take a peek and let me know of this is sufficient to be merged? Thanks and with kind regards, Christian

3 12

[docs] Project guides in PDF format
by Peter Matulis 07 Apr '21

07 Apr '21

Hello, I understand that there was a push [1] to make PDFs available for download from the published pages on docs.openstack.org for the various guides (Sphinx projects). Is there any steam left in this initiative? The next best thing is to have something at the repository level that people can work with. Currently, there are a limited number of guides that can be converted to PDF from the openstack-manuals repository. What is the suggested implementation for individual projects? Thanks, Peter Matulis [1]: https://etherpad.opendev.org/p/train-pdf-support-goal

5 11

[qa][hacking] Proposing new core reviewers
by Martin Kopec 06 Apr '21

06 Apr '21

Hi all, I'd like to propose Sorin Sbarnea (IRC: zbr) and Radosław Piliszek (IRC: yoctozepto) to hacking core. They both are doing a great upstream work among multiple different projects and volunteered to help us with maintenance of hacking project as well. You can vote/feedback in this email thread. If no objection by 6th of April, we will add them to the list. Regards, -- Martin Kopec

3 3

[all] Gate resources and performance
by Dan Smith 06 Apr '21

06 Apr '21

Hi all, I have become increasingly concerned with CI performance lately, and have been raising those concerns with various people. Most specifically, I'm worried about our turnaround time or "time to get a result", which has been creeping up lately. Right after the beginning of the year, we had a really bad week where the turnaround time was well over 24 hours. That means if you submit a patch on Tuesday afternoon, you might not get a test result until Thursday. That is, IMHO, a real problem and massively hurts our ability to quickly merge priority fixes as well as just general velocity and morale. If people won't review my code until they see a +1 from Zuul, and that is two days after I submitted it, that's bad. Things have gotten a little better since that week, due in part to getting past a rush of new year submissions (we think) and also due to some job trimming in various places (thanks Neutron!). However, things are still not great. Being in almost the last timezone of the day, the queue is usually so full when I wake up that it's quite often I don't get to see a result before I stop working that day. I would like to ask that projects review their jobs for places where they can cut out redundancy, as well as turn their eyes towards optimizations that can be made. I've been looking at both Nova and Glance jobs and have found some things I think we can do less of. I also wanted to get an idea of who is "using too much" in the way of resources, so I've been working on trying to characterize the weight of the jobs we run for a project, based on the number of worker nodes required to run all the jobs, as well as the wall clock time of how long we tie those up. The results are interesting, I think, and may help us to identify where we see some gains. The idea here is to figure out[1] how many "node hours" it takes to run all the normal jobs on a Nova patch compared to, say, a Neutron one. If the jobs were totally serialized, this is the number of hours a single computer (of the size of a CI worker) would take to do all that work. If the number is 24 hours, that means a single computer could only check *one* patch in a day, running around the clock. I chose the top five projects in terms of usage[2] to report here, as they represent 70% of the total amount of resources consumed. The next five only add up to 13%, so the "top five" seems like a good target group. Here are the results, in order of total consumption: Project % of total Node Hours Nodes ------------------------------------------ 1. TripleO 38% 31 hours 20 2. Neutron 13% 38 hours 32 3. Nova 9% 21 hours 25 4. Kolla 5% 12 hours 18 5. OSA 5% 22 hours 17 What that means is that a single computer (of the size of a CI worker) couldn't even process the jobs required to run on a single patch for Neutron or TripleO in a 24-hour period. Now, we have lots of workers in the gate, of course, but there is also other potential overhead involved in that parallelism, like waiting for nodes to be available for dependent jobs. And of course, we'd like to be able to check more than patch per day. Most projects have smaller gate job sets than check, but assuming they are equivalent, a Neutron patch from submission to commit would undergo 76 hours of testing, not including revisions and not including rechecks. That's an enormous amount of time and resource for a single patch! Now, obviously nobody wants to run fewer tests on patches before they land, and I'm not really suggesting that we take that approach necessarily. However, I think there are probably a lot of places that we can cut down the amount of *work* we do. Some ways to do this are: 1. Evaluate whether or not you need to run all of tempest on two configurations of a devstack on each patch. Maybe having a stripped-down tempest (like just smoke) to run on unique configs, or even specific tests. 2. Revisit your "irrelevant_files" lists to see where you might be able to avoid running heavy jobs on patches that only touch something small. 3. Consider moving some jobs to the experimental queue and run them on-demand for patches that touch particular subsystems or affect particular configurations. 4. Consider some periodic testing for things that maybe don't need to run on every single patch. 5. Re-examine tests that take a long time to run to see if something can be done to make them more efficient. 6. Consider performance improvements in the actual server projects, which also benefits the users. If you're a project that is not in the top ten then your job configuration probably doesn't matter that much, since your usage is dwarfed by the heavy projects. If the heavy projects would consider making changes to decrease their workload, even small gains have the ability to multiply into noticeable improvement. The higher you are on the above list, the more impact a small change will have on the overall picture. Also, thanks to Neutron and TripleO, both of which have already addressed this in some respect, and have other changes on the horizon. Thanks for listening! --Dan 1: https://gist.github.com/kk7ds/5edbfacb2a341bb18df8f8f32d01b37c 2; http://paste.openstack.org/show/C4pwUpdgwUDrpW6V6vnC/

12 36

[kolla][glance] Few questions about images.
by Eddie Yen 06 Apr '21

06 Apr '21

Hi everyone, I want to ask about the image operating permission & Windows images issue we met since we can't find any answers on the internet. 1. Now we're still using Rocky with Ceph as storage. Sometimes we need to re-pack the image on the Openstack. We used to save as snapshot (re-pack by Nova ephemeral VM) or upload to image (re-pack by volume), then set snapshot/image's visibility to the public. But since Rocky, we can't do this anymore because when we try to set public, Horizon always shows "not enough permission" error. The workaround we're using for now is creating a nova snapshot after re-pack, download the snapshot, then upload the snapshot again as public. But it's utterly wasting time if the images are huge. So we want to know how to unleash this limitation can let us just change snapshot to public at least. 2. Openstack uses virtio as a network device by default, so we always install a virtio driver when packing Windows images. As the network performance issue in GSO/TSO enablement, we also need to disable them in device properties. But since Windows 10 2004 build (my thought) device properties always reset these settings after Sysprep. We found there's a workaround [1] to solve this issue, but may not work sometimes. Is there a better way to solve this issue? Many thanks, Eddie. [1] PersistAllDeviceInstalls | Microsoft Docs <https://docs.microsoft.com/en-us/windows-hardware/customize/desktop/unatten…>

1 1

launch VM on volume vs. image
by Tony Liu 03 Apr '21

03 Apr '21

Hi, With Ceph as the backend storage, launching a VM on volume takes much longer than launching on image. Why is that? Could anyone elaborate the high level workflow for those two cases? Thanks! Tony

3 9

[ironic] Review Jams
by Julia Kreger 01 Apr '21

01 Apr '21

In the Ironic team's recent mid-cycle call, we discussed the need to return to occasionally having review jams in order to help streamline the review process. In other words, get eyes on a change in parallel and be able to discuss the change. The goal is to help get people on the same page in terms of what and why. Be on hand to answer questions or back-fill context. This is to hopefully avoid the more iterative back and forth nature of code review, which can draw out a long chain of patches. As always, the goal is not perfection, but forward movement especially for complex changes. We've established two time windows that will hopefully not to be too hard for some contributors to make it to. It doesn't need to be everyone, but it would help for at least some people whom actively review or want to actively participate in reviewing, or whom are even interested in a feature to join us for our meeting. I've added an entry on to our wiki page to cover this, with the current agenda and anticipated review jam topic schedule. The tl;dr is we will use meetpad[1] and meet on Mondays at 2 PM UTC and Tuesdays at 6 PM UTC. The hope is to to enable some overlap of reviewers. If people are interested in other times, please bring this up in the weekly meeting or on the mailing list. I'm not sending out calendar invites for this. Yet. :) See everyone next week! -Julia [0]: https://wiki.openstack.org/wiki/Meetings/Ironic#Review_Jams [1]: https://meetpad.opendev.org/ironic

3 2