[release][infra] SSH host key issue - RedFlag stop approving
Hi, tag-release jobs started to fail today and what i see is mostly that ECDSA host key for IP address X not in list of known hosts [1][2][3] Please don't approve release patches until we haven't figure out what is the problem. [1] https://zuul.opendev.org/t/openstack/build/a490703c83514eb7ba9d0ff6307f7371/... [2] https://zuul.opendev.org/t/openstack/build/00d3f765ea6847a3bf7ea53294cf2a88/... [3] https://zuul.opendev.org/t/openstack/build/d8aaeff52dac442e89fd3e253596e0c2/... Thanks, Előd irc:elodilles @ #openstack-release
On 2022-01-25 11:51:20 +0100 (+0100), Előd Illés wrote:
tag-release jobs started to fail today and what i see is mostly that ECDSA host key for IP address X not in list of known hosts [1][2][3] [...]
Just a quick update, that message about an unrecognized host key was a red herring. The actual error appears after that, indicating the signed tag push was rejected for insufficient permissions. We suspect this may be a regression in Gerrit 3.4 (to which we upgraded in yesterday's maintenance) and are testing a couple of potential workarounds while preparing to raise it with the Gerrit developers. We'll keep everyone posted. -- Jeremy Stanley
On 2022-01-25 15:40:21 +0000 (+0000), Jeremy Stanley wrote: [...]
The actual error appears after that, indicating the signed tag push was rejected for insufficient permissions. We suspect this may be a regression in Gerrit 3.4 (to which we upgraded in yesterday's maintenance) and are testing a couple of potential workarounds while preparing to raise it with the Gerrit developers.
We'll keep everyone posted.
We've merged a workaround so people won't be blocked: http://lists.opendev.org/pipermail/service-announce/2022-January/000031.html It's okay to resume approving automated release requests. If anyone is pushing tags directly to Gerrit for some reason, please be very careful to make sure they're signed until we can get the situation properly solved. -- Jeremy Stanley
Thanks for the workaround! The first tag-release jobs started to pass, so it works. The Red Flag for approving patches is now officially removed :) The release management team started to approve release patches. Thanks again, Előd irc:elodilles @ #openstack-release On 2022. 01. 25. 22:02, Jeremy Stanley wrote:
On 2022-01-25 15:40:21 +0000 (+0000), Jeremy Stanley wrote: [...]
The actual error appears after that, indicating the signed tag push was rejected for insufficient permissions. We suspect this may be a regression in Gerrit 3.4 (to which we upgraded in yesterday's maintenance) and are testing a couple of potential workarounds while preparing to raise it with the Gerrit developers.
We'll keep everyone posted. We've merged a workaround so people won't be blocked:
http://lists.opendev.org/pipermail/service-announce/2022-January/000031.html
It's okay to resume approving automated release requests. If anyone is pushing tags directly to Gerrit for some reason, please be very careful to make sure they're signed until we can get the situation properly solved.
On 2022-01-26 12:49:11 +0100 (+0100), Előd Illés wrote:
Thanks for the workaround! The first tag-release jobs started to pass, so it works.
The Red Flag for approving patches is now officially removed :) The release management team started to approve release patches. [...]
The three failed releases from yesterday have been rerun now and seem to have worked as well. -- Jeremy Stanley
participants (2)
-
Előd Illés
-
Jeremy Stanley