Thanks Clark for follow-up. My explanation was not correct, and I should have said RSA + SHA1 no longer works.
Our problem was that the key generated by create keypair api in nova uses RSA + SHA1 thus ssh by tempest with that key no longer works since SHA1 was disabled in a recent update in CentOS 9 Stream.
On Thu, Mar 10, 2022 at 12:31 AM Clark Boylan cboylan@sapwetik.org wrote:
On Tue, Mar 8, 2022, at 10:01 PM, Takashi Kajinami wrote:
Both of the two issues have been resolved and c9s integration jobs are voting again.
As a side note, It seems the second issue within tempest tests was caused by recent change in openssl in CentOS9 Stream repo and rsa key is no longer allowed for ssh.
To clarify this is RSA no longer allowed with SSH or is it just RSA + SHA1? The RSA + SHA1 problem has been known for a bit due to Fedora making that update a while back. But RSA + SHA2 does work on Fedora. The issue there is some servers like the dropbear server in Cirros and the MINA SSHD used by Gerrit either don't support RSA + SHA2 or lack the required negotation bits to allow RSA + SHA2.
Gerrit 3.6 should fix this, and I believe there is some effort to update Cirros to a newer version of dropbear which will support RSA + SHA2.
Separately, it might be a good idea to try and push back on these systems to stop defaulting to RSA + SHA1 if that combination is not allowed. They should default to RSA + SHA2 if that is the only version of RSA that will function on their platform. Then if the server supports it but cannot negotiate it properly (this is the case with Gerrit) it should continue to function.
We worked around the issue by the feature in tempest to use a different format but I've submitted a feedback to know about current usage of rsa key[1]. [1] https://bugs.launchpad.net/nova/+bug/1962726