[magnum][murano][release][sahara][tc][trove][zaqar] Emergency bypass reviewer/PTL/liaison approval for Yoga release fixes
A number of teams seem not to have noticed that their tests (and possibly the services themselves) were broken by the removal[*] of the "tenant" argument from RequestContext objects in oslo.context 4.0.0 released on February 3, 2022. As we're at a little over a day left before we need to finalize the Yoga release, I'm seeking permission from the TC on behalf of the OpenStack Release Managers to bypass normal core reviewer, PTL and release liaison approval processes in order to approve any remaining changes for official OpenStack projects necessary for this transition, and to produce new release candidates with those fixes included as necessary. [*] https://review.opendev.org/815938 -- Jeremy Stanley
---- On Mon, 28 Mar 2022 13:28:45 -0500 Jeremy Stanley <fungi@yuggoth.org> wrote ----
A number of teams seem not to have noticed that their tests (and possibly the services themselves) were broken by the removal[*] of the "tenant" argument from RequestContext objects in oslo.context 4.0.0 released on February 3, 2022.
As we're at a little over a day left before we need to finalize the Yoga release, I'm seeking permission from the TC on behalf of the OpenStack Release Managers to bypass normal core reviewer, PTL and release liaison approval processes in order to approve any remaining changes for official OpenStack projects necessary for this transition, and to produce new release candidates with those fixes included as necessary.
Thanks, Fungi and the release team for bringing it and taking care of those projects release. We called the TC emergency meeting to discuss it and agreed on the above proposal to allow fungi to approve the fixes and proceed on their release. - https://meetings.opendev.org/meetings/tc/2022/tc.2022-03-28-18.07.log.html This is something those project's PTLs/Release liaisons (who were recently appointed based on their request) should give serious thought to this situation and take care of it at least during the release time that their code/tests are in working state. -gmann
[*] https://review.opendev.org/815938 -- Jeremy Stanley
On Monday, 28 March 2022 20:28:45 CEST Jeremy Stanley wrote:
A number of teams seem not to have noticed that their tests (and possibly the services themselves) were broken by the removal[*] of the "tenant" argument from RequestContext objects in oslo.context 4.0.0 released on February 3, 2022.
As we're at a little over a day left before we need to finalize the Yoga release, I'm seeking permission from the TC on behalf of the OpenStack Release Managers to bypass normal core reviewer, PTL and release liaison approval processes in order to approve any remaining changes for official OpenStack projects necessary for this transition, and to produce new release candidates with those fixes included as necessary.
I think the relevant patches were merged last week in sahara, both master and yoga. https://review.opendev.org/q/topic:story%252F2009921 -- Luigi
Same for magnum, 10 days ago. https://review.opendev.org/q/I10daa22b614aab456b837c03278eb04da0355ff7 Merged in stable/yoga and tagged 14.0.0.0rc2. Am I missing something? Spyros On Mon, Mar 28, 2022 at 9:27 PM Luigi Toscano <ltoscano@redhat.com> wrote:
On Monday, 28 March 2022 20:28:45 CEST Jeremy Stanley wrote:
A number of teams seem not to have noticed that their tests (and possibly the services themselves) were broken by the removal[*] of the "tenant" argument from RequestContext objects in oslo.context 4.0.0 released on February 3, 2022.
As we're at a little over a day left before we need to finalize the Yoga release, I'm seeking permission from the TC on behalf of the OpenStack Release Managers to bypass normal core reviewer, PTL and release liaison approval processes in order to approve any remaining changes for official OpenStack projects necessary for this transition, and to produce new release candidates with those fixes included as necessary.
I think the relevant patches were merged last week in sahara, both master and yoga. https://review.opendev.org/q/topic:story%252F2009921
-- Luigi
On 2022-03-28 21:32:54 +0200 (+0200), Spyros Trigazis wrote:
Same for magnum, 10 days ago. https://review.opendev.org/q/I10daa22b614aab456b837c03278eb04da0355ff7
Merged in stable/yoga and tagged 14.0.0.0rc2.
Am I missing something? [...]
Nope, it was crossed off the list shortly after I sent my initial message, since it looks like the fix did end up in RC2 on Friday (the error reports the Release Managers were working from were from earlier in the week). -- Jeremy Stanley
Sorry, it seems there were some false alarms. Magnum seems to be OK regarding the mentioned change. Előd On 2022. 03. 28. 21:32, Spyros Trigazis wrote:
Same for magnum, 10 days ago. https://review.opendev.org/q/I10daa22b614aab456b837c03278eb04da0355ff7
Merged in stable/yoga and tagged 14.0.0.0rc2.
Am I missing something?
Spyros
On Mon, Mar 28, 2022 at 9:27 PM Luigi Toscano <ltoscano@redhat.com> wrote:
On Monday, 28 March 2022 20:28:45 CEST Jeremy Stanley wrote: > A number of teams seem not to have noticed that their tests (and > possibly the services themselves) were broken by the removal[*] of > the "tenant" argument from RequestContext objects in oslo.context > 4.0.0 released on February 3, 2022. > > As we're at a little over a day left before we need to finalize the > Yoga release, I'm seeking permission from the TC on behalf of the > OpenStack Release Managers to bypass normal core reviewer, PTL and > release liaison approval processes in order to approve any remaining > changes for official OpenStack projects necessary for this > transition, and to produce new release candidates with those fixes > included as necessary. > > [*] https://review.opendev.org/815938
I think the relevant patches were merged last week in sahara, both master and yoga. https://review.opendev.org/q/topic:story%252F2009921
-- Luigi
On 3/28/22 21:32, Spyros Trigazis wrote:
Same for magnum, 10 days ago. https://review.opendev.org/q/I10daa22b614aab456b837c03278eb04da0355ff7 <https://review.opendev.org/q/I10daa22b614aab456b837c03278eb04da0355ff7>
Merged in stable/yoga and tagged 14.0.0.0rc2.
Am I missing something?
Spyros Hi,
I'm still getting these though: https://buildd.debian.org/status/fetch.php?pkg=magnum&arch=all&ver=14.0.0%7Erc1-3&stamp=1648488744&raw=0 Any idea what's going on? It doesn't do that on my laptop... :/ Cheers, Thomas Goirand (zigo)
On 2022-03-29 10:22:18 +0200 (+0200), Thomas Goirand wrote: [...]
I'm still getting these though:
Any idea what's going on? It doesn't do that on my laptop... :/
"BlockingIOError: [Errno 11] write could not complete without blocking" is what you should expect if one side of a socket or other file descriptor is opened in noblock mode and too much data is written too quickly, so it's not surprising that it would crop up at random and you'd see it on some systems but not others. I got as far as looking through the TestApp class in the webtest library Pecan uses, but am not clear where the socket setup is occurring. I suppose it's also possible that the test runner or even sbuild itself is setting stdout/stderr to blocking mode. -- Jeremy Stanley
On 2022-03-28 21:23:22 +0200 (+0200), Luigi Toscano wrote: [...]
I think the relevant patches were merged last week in sahara, both master and yoga. https://review.opendev.org/q/topic:story%252F2009921
Yes, in Sahara's case, it looks like the release request for a new RC was waiting for PTL or release liaison approval: https://review.opendev.org/835037 It seems to have received no feedback by the final RC deadline and was therefore abandoned initially, but has been restored since it was pointed out by downstream distro package maintainers that the test for Sahara were still failing on the existing release candidate. -- Jeremy Stanley
Indeed, the patches are merged, unfortunately the generated release patch was abandoned as no replies came from release liaisons to continue with it. The release patch is now restored: https://review.opendev.org/c/openstack/releases/+/835037 Előd On 2022. 03. 28. 21:23, Luigi Toscano wrote:
On Monday, 28 March 2022 20:28:45 CEST Jeremy Stanley wrote:
A number of teams seem not to have noticed that their tests (and possibly the services themselves) were broken by the removal[*] of the "tenant" argument from RequestContext objects in oslo.context 4.0.0 released on February 3, 2022.
As we're at a little over a day left before we need to finalize the Yoga release, I'm seeking permission from the TC on behalf of the OpenStack Release Managers to bypass normal core reviewer, PTL and release liaison approval processes in order to approve any remaining changes for official OpenStack projects necessary for this transition, and to produce new release candidates with those fixes included as necessary.
[*] https://review.opendev.org/815938 I think the relevant patches were merged last week in sahara, both master and yoga. https://review.opendev.org/q/topic:story%252F2009921
On 2022-03-28 18:28:45 +0000 (+0000), Jeremy Stanley wrote:
A number of teams seem not to have noticed that their tests (and possibly the services themselves) were broken by the removal[*] of the "tenant" argument from RequestContext objects in oslo.context 4.0.0 released on February 3, 2022. [...]
Since this initial list was assembled, we've been able to cross Magnum and Zaqar off the list as already fixed or affected by unrelated issues. Note that some projects were included because they did not acknowledge release requests with the fixes, so while they may have patches already merged in stable/yoga, the tags slated for inclusion in the release did not cover those fixes. In such cases, the Release Managers are seeking to approve pending release requests in order to make sure those fixes end up in the coordinated release on Wednesday. -- Jeremy Stanley
On 2022-03-28 18:28:45 +0000 (+0000), Jeremy Stanley wrote:
A number of teams seem not to have noticed that their tests (and possibly the services themselves) were broken by the removal[*] of the "tenant" argument from RequestContext objects in oslo.context 4.0.0 released on February 3, 2022.
As we're at a little over a day left before we need to finalize the Yoga release, I'm seeking permission from the TC on behalf of the OpenStack Release Managers to bypass normal core reviewer, PTL and release liaison approval processes in order to approve any remaining changes for official OpenStack projects necessary for this transition, and to produce new release candidates with those fixes included as necessary.
Apparently there were not yet fixes proposed for all projects. I tried to put one together for Murano, but am a bit out of my depth. My naive attempt to find places where tenant_id gets passed into the context manager still misses something, or is having a ripple effect in other parts of the code. It's getting rather late in my part of the World, so if someone else (perhaps someone with more context, pun intended) has ideas on how to get it working, please feel free to push up new revisions of this change: https://review.opendev.org/835488 Any help is most appreciated. If we can get it working in master, then merging that and backporting to stable/yoga will be the next step for Murano. -- Jeremy Stanley
Hi Jeremy,
Apparently there were not yet fixes proposed for all projects. I tried to put one together for Murano, but am a bit out of my depth. My naive attempt to find places where tenant_id gets passed into the context manager still misses something, or is having a ripple effect in other parts of the code. It's getting rather late in my part of the World, so if someone else (perhaps someone with more context, pun intended) has ideas on how to get it working, please feel free to push up new revisions of this change:
https://review.opendev.org/835488
Any help is most appreciated. If we can get it working in master, then merging that and backporting to stable/yoga will be the next step for Murano.
I'm starting to have a look now. I'll let you know how I get on. cheers, Andy
On 2022-03-29 16:13:12 +1100 (+1100), Andy Botting wrote: [...]
I'm starting to have a look now. I'll let you know how I get on.
Thanks! Looks like it's passing all the voting jobs now except lower-constraints, with an array of errors where tenant_id is coming back as None in session objects and DB inserts, but also some failures for API authorization problems which I guess could also drive back to wrong tenant_id values as well. I can't tell if the patch for the tenant->project_id migration is the cause, or if this is a separate latent regression. I suppose one response might be to drop the lower-constraints job or make it non-voting, or simply backport this to stable/yoga even though it hasn't merged to master. -- Jeremy Stanley
Thanks, Jeremy and Andy, I dropped the l-c job and passed all the jobs, I approved the patch and also backport to stable/yoga. On Tue, Mar 29, 2022 at 8:17 PM Jeremy Stanley <fungi@yuggoth.org> wrote:
On 2022-03-29 16:13:12 +1100 (+1100), Andy Botting wrote: [...]
I'm starting to have a look now. I'll let you know how I get on.
Thanks! Looks like it's passing all the voting jobs now except lower-constraints, with an array of errors where tenant_id is coming back as None in session objects and DB inserts, but also some failures for API authorization problems which I guess could also drive back to wrong tenant_id values as well.
I can't tell if the patch for the tenant->project_id migration is the cause, or if this is a separate latent regression. I suppose one response might be to drop the lower-constraints job or make it non-voting, or simply backport this to stable/yoga even though it hasn't merged to master. -- Jeremy Stanley
-- Thanks, Rong Zhu
On 2022-03-29 21:51:44 +0800 (+0800), Rong Zhu wrote:
Thanks, Jeremy and Andy, I dropped the l-c job and passed all the jobs, I approved the patch and also backport to stable/yoga. [...]
Thanks for picking this up! Once the backport merges, the Release Managers can tag a new RC for Murano and then we'll be in a good place for the final Yoga releases tomorrow. -- Jeremy Stanley
Thanks everyone for all the efforts! The necessary releases are done [1] thanks to you! Now we are looking forward to the Yoga final release in a couple of hours! Thanks, Előd [1] https://etherpad.opendev.org/p/tenant-projectid-last-minute-fixes On 2022. 03. 29. 16:39, Jeremy Stanley wrote:
On 2022-03-29 21:51:44 +0800 (+0800), Rong Zhu wrote:
Thanks, Jeremy and Andy, I dropped the l-c job and passed all the jobs, I approved the patch and also backport to stable/yoga. [...]
Thanks for picking this up! Once the backport merges, the Release Managers can tag a new RC for Murano and then we'll be in a good place for the final Yoga releases tomorrow.
participants (8)
-
Andy Botting
-
Előd Illés
-
Ghanshyam Mann
-
Jeremy Stanley
-
Luigi Toscano
-
Rong Zhu
-
Spyros Trigazis
-
Thomas Goirand