September 2024 - openstack-discuss

[unmaintained][cinder] question about zuul jobs for unmaintained branches
by Brian Rosmaita 20 Sep '24

20 Sep '24

Hello Unmaintained maintenance team, I was reviewing a cinder-tempest-plugin patch (which is in an un-branched repository), and noticed that we have a bunch of Zuul cinder jobs defined for stable branches that no longer exist, for example, stable/wallaby [0]. Where this impacts the unmaintained branches is that, for example, unmaintained/wallaby's .zuul.yaml is configured to run (for example) cinder-tempest-plugin-lvm-lio-barbican in the check [1], but you can see on this review, it's not actually running: https://review.opendev.org/c/openstack/cinder/+/926279?tab=change-view-tab-… So, my question is what to do about these unused jobs: (a) delete them (b) only delete stable/x for x <= ussuri, and update the jobs for stable/x for victoria <= x <= zed to unmaintained/x (c) delete them all, but propose a patch to the appropriate unmaintained branches zuul.yaml files that contain the job definition so that the Unmaintained team can make any changes needed to get them running without having to go through the cinder-tempest-plugin-core team's review/approval process (d) something I haven't thought of Thanks! brian [0] https://opendev.org/openstack/cinder-tempest-plugin/src/branch/master/.zuul… [1] https://opendev.org/openstack/cinder/src/branch/unmaintained/wallaby/.zuul.…

2 1

New to OpenStack. Test environment with 1 control node and 2 compute nodes. Unable to migrate or resize instances
by collinl＠churchofjesuschrist.org 20 Sep '24

20 Sep '24

As the subject says, I am new to OpenStack, and have spun up a test cluster with one control node, and two compute nodes. It seems to work for several tasks (i.e. I can spin up a cirros instance on both of the compute nodes, which seems to work just fine) However, when I attempt to migrate an instance from one compute node to the other, the instance goes in to an error state. (In order to get it running again, I have to set the state to active, then reboot the instance) I am not sure which logs will have the required information to help debug this issue, but I have looked through nova and cinder and keystone logs. Nothing I am seeing is obvious to me as to the cause of the failures. Also taking a snapshot seems to fail as well, with this error: Error: Unable to create snapshot. Details Invalid input received: Invalid volume: Volume e12b3ff8-9870-45f9-b59a-c6fb194ed517 status must be available, in-use, but current status is: attaching. (HTTP 400) (Request-ID: req-499d3bf9-97a0-48b2-8565-254a87287cff) (HTTP 400) (Request-ID: req-d0cd5aed-894e-40b2-8a2f-4b22188ec372) Looking at the volumes in my environment, both volumes show a status of "attaching" (both are for a test cirros instance, both are type nfs, and both instances seem to work just fine (meaning I can connect to them either on the console or via ssh and run commands without errors) Is the reason that each of those tasks is failing because of the "attaching" status of the volumes? I spun up the cluster using packstack and all the nodes are installed on Rocky Linux 9.4. I added an nfs backend after packstack was all done, and that does seem to be working, as the ephemeral volumes used to spin up the test instances does seem to be on the nfs storage, rather than the default lvm storage backend. I have installed openstack using the bobcat release of openstack (from the centos-openstack-bobcat repository). I have been crawling through the standalone install documentation to see if I can find something that explains why the instance migration isn't working, but, again, nothing is jumping out at me. Any guidance on where I should be looking, or advice on what I may be missing would be greatly appreciated.

4 10

upper-constraints (< $latest) considered harmful
by Stephen Finucane 20 Sep '24

20 Sep '24

o/ I'll jump straight into it. For those of you not in the loop we're currently going through a situation where it looks like OSC will be held back due to a bug that the Nova gates are highlighted. The actual details aren't hugely important, but what is important is the fact that this bug was first included in the OSC 7.0.0 release, which was released on August 6th. For those of you who are better at mental calendar maths than I am (or already looked at your calendar), you'll know this was over 6 weeks ago, well before the various feature freeze deadlines . and you might ask why did we only spot this last week? That would be because the upper-constraint (henceforth referred to as u-c) bump failed [1] due a different issue, meaning u-c kept us pinned at <= 6.6.1 which in turn meant every CI job *except* the ones running in the OSC gate itself kept testing the old release instead of the thing we were supposed to be released as part of Dalmatian. That only changed last week when 7.1.0 was released and the u-c bump for *that* merged [2], at which point people started shouting and hollering. Now, there is one other important consideration to factor into this, namely that, due to a flaw in our tooling, we didn't end up cutting a release branch for OSC alongside the rest of the client-library deliverables. Had we done so, we'd have seen 7.1.0 much earlier than we actually did [3]. That 7.1.0 release fixed the initial issue that prevented us releasing 7.0.0. However, this doesn't change the fact that 7.0.0 came out 6 weeks ago and not once did we get to test it outside of OSC's gate, and what's worse, we (the SDK team) were totally oblivious to the fact). Had it been known, we'd have cut a new release as soon as the fix [4] for that issue had merged (August 12th, which for the record is over 5 weeks ago). This is not the first time this has happened to us with one of our own dependencies. We had a very similar issue in Bobcat (2023.2) where we cut an oslo.db 13.0.0 release very early in that cycle, followed by a number of subsequent releases, only to have none of them end up making their way into u-c due to issues with some services. We then got to the end of the cycle, realized this, and had to do a frenzied mega revert to get us back to a pre-13.x state and allow us to release an oslo.db deliverable for Bobcat. It also happened with Castellan, whose 4.2.0 release sat waiting to merge for 4 months [4] before undergoing a similar mass revert. Adding more fuel to this fire, it's not just a problem with our own dependencies either. In fact, the problem is probably worse there. Need I remind people that it took 5 releases or 2 years, 8 months and change to uncap SQLAlchemy 2.x (a contributing factor in the oslo.db issue, fwiw). We had to struggle to get Sphinx uncapped in Bobcat [5] and fight to not revert that cap. You currently can't build docs for many projects on Fedora hosts since we're capping, among many other things, the Pillow library. The list goes on. Which brings me to my main points, of which there are three. Firstly: * There needs to be a clear time limit for how long a u-c bump can remain unmerged and a plan for what happens if affected projects have not resolved their issues within that time. There will always be exceptions - no one could rightly have asked services to switch to SQLAlchemy 2.x in a fortnight, and libraries can have legit bugs that justify a block - but those exceptions should be well-understood and, as with the bumps themselves, time-limited. A project that needs 4 releases to adapt to a crucial new library version is not a healthy project. secondly: * Caps or u-c bump reverts should similarly be clearly time-limited. We should not be defaulting to e.g re-capping Sphinx because your docs build has started to fail and then leaving that cap in place for months. Cap, fix the issue, and remove the cap asap. and, finally: * A failing u-c bump for an openstack deliverable should be treated with the highest priority 🚨🚨🚨 and should be something that the corresponding team should be made aware of immediately so they can start coordinating a resolution. We should not have u-c patches languishing for weeks, let alone months, only for there to be a last minute panic like this. I have ideas on the above (which basically warrant to more stick/[7]) but, perhaps fortunately, this isn't something I can decide on my own. Nor, should I add, is it something I expect the understaffed and oversubscribed release team to be able to do themselves. Instead, I think it's something that the TC and community as a whole need to settle on. So as soon as we get Dalmatian out the door, let's do that. Cheers, Stephen [1] https://review.opendev.org/c/openstack/requirements/+/925763 [2] https://review.opendev.org/c/openstack/requirements/+/928948 [3] https://review.opendev.org/c/openstack/releases/+/928838 [4] https://review.opendev.org/c/openstack/requirements/+/883141 [5] https://review.opendev.org/c/openstack/requirements/+/891694 [6] https://review.opendev.org/c/openstack/requirements/+/927102 [7] https://y.yarn.co/dbdc0d5b-04e6-4c39-9cd3-44c3be3729ab.mp4

8 9

[release] Release countdown for week R-1, Sep 23-27
by Elõd Illés 20 Sep '24

20 Sep '24

Development Focus ----------------- We are on the final mile of the 2024.2 Dalmatian development cycle! Remember that the 2024.2 Dalmatian final release will include the latest release candidate (for cycle-with-rc deliverables) or the latest intermediary release (for cycle-with-intermediary deliverables) available. September 26th, 2024 is the deadline for final 2024.2 Dalmatian release candidates as well as any last cycle-with-intermediary deliverables. We will then enter a quiet period until we tag the final release on October 2nd, 2024. Teams should be prioritizing fixing release-critical bugs, before that deadline. Otherwise it's time to start planning the 2025.1 Epoxy development cycle, including discussing PTG sessions content, in preparation of Project Teams Gathering. Actions ------- Watch for any translation patches coming through on the stable/2024.2 branch and merge them quickly. If you discover a release-critical issue, please make sure to fix it on the master branch first, then backport the bugfix to the stable/2024.2 branch before triggering a new release. Please drop by #openstack-release with any questions or concerns about the upcoming release ! Upcoming Deadlines & Dates -------------------------- Final 2024.2 Dalmatian release: October 2nd, 2024 Project Teams Gathering: October 21-25, 2024 Előd Illés irc: elodilles @ #openstack-release

1 0

[cinder] Retype encrypted cinder volume to non-encrypted volume
by Jan Wasilewski 20 Sep '24

20 Sep '24

Hi, I would like to ask about the procedure to retype a Cinder volume that is in-use (bootable volume, connected to a VM) from an encrypted (LUKS) type to a non-encrypted type (1000 IOPS). I tried to do this via the CLI[1], and while it seemed to work (at least according to the CLI output), the process didn’t complete successfully. Cinder logs[2] indicate that everything looks more or less fine: the volume is created, the volume migration is OK, and the retype is marked as completed. However, in the end, it doesn’t work (final output of [1]). I reviewed the blueprint at https://blueprints.launchpad.net/cinder/+spec/retype-encrypted-volume, which appears to be implemented. Am I doing something wrong, or does this functionality not work as expected? Thanks in advance for your help and guidance. /JW *[1] Retype from CLI: https://paste.openstack.org/show/bVpFT82oURnanXkixreT/ <https://paste.openstack.org/show/bVpFT82oURnanXkixreT/>[2] cinder-logs: https://paste.openstack.org/show/b5rZpWNMqdi6qdpjaDfg/ <https://paste.openstack.org/show/b5rZpWNMqdi6qdpjaDfg/>*

4 6

[glance] Still not working over uwsgi
by Thomas Goirand 20 Sep '24

20 Sep '24

Hi, I've been told in the past in this very list, that Glance would now work over uwsgi. Though that's not the case. Normal image upload works, but doing a "openstack server backup" fails, as well as shelving (it's the same operation...). We get: "Failed to upload image data due to HTTP error: webob.exc.HTTPRequestEntityTooLarge: Image storage media is full: Object PUT failed" even though the Swift user Glance is configured for isn't over quota. Is there any configuration options that we missed? Or should we still consider that Glance over uwsgi is broken? Cheers, Thomas Goirand (zigo)

3 2

[neutron] Drivers meeting canceled today and next Friday, September 27th
by Brian Haley 20 Sep '24

20 Sep '24

Hello Neutrinos: I'm going to cancel today's and next Friday's drivers meeting as there is no agenda and I have conflicts both days/times. We can spend the time making sure 2024.2 RC is in a good place :) Have a nice weekend! -Brian

1 0

[debian] Dalmatian release validated from the Debian point of view
by Thomas Goirand 20 Sep '24

20 Sep '24

Hi, This is probably a breaking record from my side: I've finished packaging even with all project not being tagged for RC1 ! :) Indeed, I have just finished running tempest on a new Dalmatian release today. Result is: no new tempest failure, so hopefully, no regression. Feel free to start consuming the Debian package of OpenStack for Dalmatian the usual way (for example with extrepo on Bookworm): extrepo enable openstack_dalmatian apt-get update I'll be uploading Dalmatian to Unstable in the next following days. This OpenStack release will be the one in Debian 13, aka Trixie. This will run under Python 3.13, so I'm getting ready to fix Python 3.13 bugs as soon as I can, both upstream and in Debian. Cheers, Thomas Goirand (zigo)

3 2

Farewell o/
by Mark Goddard 19 Sep '24

19 Sep '24

Dear OpenStack, Tomorrow will be my final day working at StackHPC before I move to a new role outside of the OpenStack community. My journey with OpenStack began at Cray 10 years ago, trying to coerce a very fresh Ironic project into managing a rack of PCs and resulting in the SNMP power driver [1]. Since then I moved to StackHPC where I've deployed and maintained many OpenStack systems of different sizes, generally with an element of High Performance Computing (HPC). I've been lucky enough to have had the opportunity to get involved in upstream OpenStack development. Over time I grew from contributor to Kolla & Ironic core to Kolla PTL for 5 cycles. I attended several summits and PTGs, and this really helped in working out how to navigate and get more involved in the community. More recently I have been more focussed on downstream than upstream work, but still marvel at the way so many people around the world come together to volunteer their time and effort to build complex software ecosystems. I'd like to thank everyone in this community who has helped me over the years. It really is a special place. To anyone standing on the sidelines wondering how to get more involved - speak up on IRC, join the PTG, triage bugs, review some code. It doesn't need to take up too much of your time. If you're like me you'll feel self-conscious at first, and unsure about the value you're providing. But keep at it, and over time you'll become an important part of the community. All the best, Mark [1] https://review.opendev.org/c/openstack/ironic/+/118404

8 7

[ironic] [neutron] [ptg] DHCP cross-project session?
by Jay Faulkner 19 Sep '24

19 Sep '24

HI all, Myself and CID from the GR-OSS team are planning on, community willing, adding a new, more reliable DHCP backend to both Ironic and Neutron. Right now, that looks like kea, but we are less concerned about the specific project as much as we are about it not being dnsmasq ( https://bugs.launchpad.net/ironic/+bug/2026757 ). Given neither of us are incredibly familiar with the Neutron codebase, we wanted to get buy in and some general implementation advice. As a note; if desired this can happen outside PTG, but this DHCP conversation is a PTG topic for Ironic so I thought this would be a good place to do it. -- Jay Faulkner

5 7