Both of the two issues have been resolved and c9s integration jobs are voting again. As a side note, It seems the second issue within tempest tests was caused by recent change in openssl in CentOS9 Stream repo and rsa key is no longer allowed for ssh. We worked around the issue by the feature in tempest to use a different format but I've submitted a feedback to know about current usage of rsa key[1]. [1] https://bugs.launchpad.net/nova/+bug/1962726 On Tue, Mar 1, 2022 at 10:47 PM Takashi Kajinami <tkajinam@redhat.com> wrote:

Unfortunately I've not yet managed to implement a workaround and it's likely I need some more time to fix it, I'm merging the patch to make c9s jobs non-voting.

https://review.opendev.org/c/openstack/puppet-openstack-integration/+/831183

I'll revert that change once we can implement workaround/solution.

On Tue, Mar 1, 2022 at 11:11 AM Takashi Kajinami <tkajinam@redhat.com> wrote:

...

Hi,

Please be aware that currently CentOS 9 integration jobs are broken because of the two problems caused by recent updates in c9s repos. https://bugs.launchpad.net/puppet-openstack-integration/+bug/1962506 https://bugs.launchpad.net/puppet-openstack-integration/+bug/1962507

I'm currently working on implementing workaround[1], but in case it would take some time, I'll make c9s integration jobs non-voting again to unblock our gate. [1] https://review.opendev.org/c/openstack/puppet-openstack-integration/+/831197

Please avoid approving changes which require integration jobs.

Thank you, Takashi

Thanks Clark for follow-up. My explanation was not correct, and I should have said RSA + SHA1 no longer works. Our problem was that the key generated by create keypair api in nova uses RSA + SHA1 thus ssh by tempest with that key no longer works since SHA1 was disabled in a recent update in CentOS 9 Stream. On Thu, Mar 10, 2022 at 12:31 AM Clark Boylan <cboylan@sapwetik.org> wrote:

On Tue, Mar 8, 2022, at 10:01 PM, Takashi Kajinami wrote:

...

Both of the two issues have been resolved and c9s integration jobs are voting again.

As a side note, It seems the second issue within tempest tests was caused by recent change in openssl in CentOS9 Stream repo and rsa key is no longer allowed for ssh.

To clarify this is RSA no longer allowed with SSH or is it just RSA + SHA1? The RSA + SHA1 problem has been known for a bit due to Fedora making that update a while back. But RSA + SHA2 does work on Fedora. The issue there is some servers like the dropbear server in Cirros and the MINA SSHD used by Gerrit either don't support RSA + SHA2 or lack the required negotation bits to allow RSA + SHA2.

Gerrit 3.6 should fix this, and I believe there is some effort to update Cirros to a newer version of dropbear which will support RSA + SHA2.

Separately, it might be a good idea to try and push back on these systems to stop defaulting to RSA + SHA1 if that combination is not allowed. They should default to RSA + SHA2 if that is the only version of RSA that will function on their platform. Then if the server supports it but cannot negotiate it properly (this is the case with Gerrit) it should continue to function.

...

We worked around the issue by the feature in tempest to use a different format but I've submitted a feedback to know about current usage of rsa key[1]. [1] https://bugs.launchpad.net/nova/+bug/1962726