[release][infra[PTL] SSH key error - RedFlag stop approving

Herve Beraud hberaud at redhat.com
Wed Nov 25 09:32:41 UTC 2020


Hello,

After a series of tests this morning and yesterday evening, we can consider
our previous issue is now solved, and we can restart approving the releases
as usual.

Indeed, all patches needed to fix our problem are now merged.

Also the following refs who failed previously have been successfully
reenqueued and stein-em tags are now available for the corresponding
projects' repos:
- 2f6e3e30f67df4f60e0e2b386755118bb1338059 [1]
- f2b5d964aa5fd3290ca82c0cd12baa29e679b687 [2]
- 3f2136e225577bf80cc90481db8e8aced937e5ec [3]
- 7bf3e98f71afeeafdd703390390677728f1749e6 [4]
- ec68594ba6faeb92eb9ad82410e3ab91db023b19 [5]

To ensure that everything works as expected we also tested the following
refs, tags and deliverables have been successfully created, and no error
happened during jobs executions:
- 1ddbb150446de8ab12676b6c896eadb8ed12d7c3 [6]
- cb115f82fd52605690bdf9bc7c44c0f7d65ff1a4 [7]

We can close this topic and return to normal activity.

Thanks all for your patience.

[1]
https://opendev.org/openstack/releases/commit/2f6e3e30f67df4f60e0e2b386755118bb1338059
[2]
https://opendev.org/openstack/releases/commit/f2b5d964aa5fd3290ca82c0cd12baa29e679b687
[3]
https://opendev.org/openstack/releases/commit/3f2136e225577bf80cc90481db8e8aced937e5ec
[4]
https://opendev.org/openstack/releases/commit/7bf3e98f71afeeafdd703390390677728f1749e6
[5]
https://opendev.org/openstack/releases/commit/ec68594ba6faeb92eb9ad82410e3ab91db023b19
[6]
https://opendev.org/openstack/releases/commit/1ddbb150446de8ab12676b6c896eadb8ed12d7c3
[7]
https://opendev.org/openstack/releases/commit/cb115f82fd52605690bdf9bc7c44c0f7d65ff1a4


Le mar. 24 nov. 2020 à 18:37, Herve Beraud <hberaud at redhat.com> a écrit :

> Just a quick reminder of the current state of the openstack release
> management.
>
> We are stuck by the SSH issue and we stopped approving release by waiting
> to fix all elements of the problem.
>
> Three patches are on the rails to solve the issue, however one isn't yet
> merged:
> - https://review.opendev.org/c/openstack/project-config/+/763797/ (merged)
> - https://review.opendev.org/c/openstack/project-config/+/763830/ (merged)
> - https://review.opendev.org/c/zuul/zuul-jobs/+/763834/ (not merged)
>
> We are stopping approving releases until this issue isn't resolved.
>
> When all these patches will be merged we will try to re-enqueue our
> failing ref to observe if it solves our issue and we will decide if we can
> restart approving releases.
>
> I added the PTLs to the object of this thread to inform them about that and
> also so that they are not surprised that we do not approve their release
> patches.
>
> We will communicate as soon as we will restart approving release patches.
>
> Thanks for your comprehension.
>
> Le lun. 23 nov. 2020 à 12:49, Herve Beraud <hberaud at redhat.com> a écrit :
>
>> Hello,
>>
>> We are facing SSH issues on our release jobs [1][2].
>> Indeed, our release account failed to connect to gerrit through SSH [2].
>> The ID seems to be recognized when we try it.
>> So that would probably mean we have a stale host key on the machine the script  is running on .
>>
>> We suppose that the issue could be a side effect of the gerrit upgrade, we updated the etherpad to inform  the infra team [3]
>>
>> @release managers: We  should stop approving, so, please hold all our unmerged patches while this incident isn't solved.
>>
>> Thanks for your reading,
>>
>> [1]
>> http://lists.openstack.org/pipermail/release-job-failures/2020-November/001487.html[2]
>> Example of faced error:
>> ```
>> ssh://release@review.opendev.org:29418/openstack/python-senlinclient.git
>> did not work. Description: Host key verification failed.
>> Could not read from remote repository
>> ```
>> [3] https://etherpad.opendev.org/p/gerrit-3.2-post-upgrade-notes
>>
>> --
>> Hervé Beraud
>> Senior Software Engineer at Red Hat
>> irc: hberaud
>> https://github.com/4383/
>> https://twitter.com/4383hberaud
>> -----BEGIN PGP SIGNATURE-----
>>
>> wsFcBAABCAAQBQJb4AwCCRAHwXRBNkGNegAALSkQAHrotwCiL3VMwDR0vcja10Q+
>> Kf31yCutl5bAlS7tOKpPQ9XN4oC0ZSThyNNFVrg8ail0SczHXsC4rOrsPblgGRN+
>> RQLoCm2eO1AkB0ubCYLaq0XqSaO+Uk81QxAPkyPCEGT6SRxXr2lhADK0T86kBnMP
>> F8RvGolu3EFjlqCVgeOZaR51PqwUlEhZXZuuNKrWZXg/oRiY4811GmnvzmUhgK5G
>> 5+f8mUg74hfjDbR2VhjTeaLKp0PhskjOIKY3vqHXofLuaqFDD+WrAy/NgDGvN22g
>> glGfj472T3xyHnUzM8ILgAGSghfzZF5Skj2qEeci9cB6K3Hm3osj+PbvfsXE/7Kw
>> m/xtm+FjnaywZEv54uCmVIzQsRIm1qJscu20Qw6Q0UiPpDFqD7O6tWSRKdX11UTZ
>> hwVQTMh9AKQDBEh2W9nnFi9kzSSNu4OQ1dRMcYHWfd9BEkccezxHwUM4Xyov5Fe0
>> qnbfzTB1tYkjU78loMWFaLa00ftSxP/DtQ//iYVyfVNfcCwfDszXLOqlkvGmY1/Y
>> F1ON0ONekDZkGJsDoS6QdiUSn8RZ2mHArGEWMV00EV5DCIbCXRvywXV43ckx8Z+3
>> B8qUJhBqJ8RS2F+vTs3DTaXqcktgJ4UkhYC2c1gImcPRyGrK9VY0sCT+1iA+wp/O
>> v6rDpkeNksZ9fFSyoY2o
>> =ECSj
>> -----END PGP SIGNATURE-----
>>
>>
>
> --
> Hervé Beraud
> Senior Software Engineer at Red Hat
> irc: hberaud
> https://github.com/4383/
> https://twitter.com/4383hberaud
> -----BEGIN PGP SIGNATURE-----
>
> wsFcBAABCAAQBQJb4AwCCRAHwXRBNkGNegAALSkQAHrotwCiL3VMwDR0vcja10Q+
> Kf31yCutl5bAlS7tOKpPQ9XN4oC0ZSThyNNFVrg8ail0SczHXsC4rOrsPblgGRN+
> RQLoCm2eO1AkB0ubCYLaq0XqSaO+Uk81QxAPkyPCEGT6SRxXr2lhADK0T86kBnMP
> F8RvGolu3EFjlqCVgeOZaR51PqwUlEhZXZuuNKrWZXg/oRiY4811GmnvzmUhgK5G
> 5+f8mUg74hfjDbR2VhjTeaLKp0PhskjOIKY3vqHXofLuaqFDD+WrAy/NgDGvN22g
> glGfj472T3xyHnUzM8ILgAGSghfzZF5Skj2qEeci9cB6K3Hm3osj+PbvfsXE/7Kw
> m/xtm+FjnaywZEv54uCmVIzQsRIm1qJscu20Qw6Q0UiPpDFqD7O6tWSRKdX11UTZ
> hwVQTMh9AKQDBEh2W9nnFi9kzSSNu4OQ1dRMcYHWfd9BEkccezxHwUM4Xyov5Fe0
> qnbfzTB1tYkjU78loMWFaLa00ftSxP/DtQ//iYVyfVNfcCwfDszXLOqlkvGmY1/Y
> F1ON0ONekDZkGJsDoS6QdiUSn8RZ2mHArGEWMV00EV5DCIbCXRvywXV43ckx8Z+3
> B8qUJhBqJ8RS2F+vTs3DTaXqcktgJ4UkhYC2c1gImcPRyGrK9VY0sCT+1iA+wp/O
> v6rDpkeNksZ9fFSyoY2o
> =ECSj
> -----END PGP SIGNATURE-----
>
>

-- 
Hervé Beraud
Senior Software Engineer at Red Hat
irc: hberaud
https://github.com/4383/
https://twitter.com/4383hberaud
-----BEGIN PGP SIGNATURE-----

wsFcBAABCAAQBQJb4AwCCRAHwXRBNkGNegAALSkQAHrotwCiL3VMwDR0vcja10Q+
Kf31yCutl5bAlS7tOKpPQ9XN4oC0ZSThyNNFVrg8ail0SczHXsC4rOrsPblgGRN+
RQLoCm2eO1AkB0ubCYLaq0XqSaO+Uk81QxAPkyPCEGT6SRxXr2lhADK0T86kBnMP
F8RvGolu3EFjlqCVgeOZaR51PqwUlEhZXZuuNKrWZXg/oRiY4811GmnvzmUhgK5G
5+f8mUg74hfjDbR2VhjTeaLKp0PhskjOIKY3vqHXofLuaqFDD+WrAy/NgDGvN22g
glGfj472T3xyHnUzM8ILgAGSghfzZF5Skj2qEeci9cB6K3Hm3osj+PbvfsXE/7Kw
m/xtm+FjnaywZEv54uCmVIzQsRIm1qJscu20Qw6Q0UiPpDFqD7O6tWSRKdX11UTZ
hwVQTMh9AKQDBEh2W9nnFi9kzSSNu4OQ1dRMcYHWfd9BEkccezxHwUM4Xyov5Fe0
qnbfzTB1tYkjU78loMWFaLa00ftSxP/DtQ//iYVyfVNfcCwfDszXLOqlkvGmY1/Y
F1ON0ONekDZkGJsDoS6QdiUSn8RZ2mHArGEWMV00EV5DCIbCXRvywXV43ckx8Z+3
B8qUJhBqJ8RS2F+vTs3DTaXqcktgJ4UkhYC2c1gImcPRyGrK9VY0sCT+1iA+wp/O
v6rDpkeNksZ9fFSyoY2o
=ECSj
-----END PGP SIGNATURE-----
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-discuss/attachments/20201125/1fa9efe8/attachment-0001.html>


More information about the openstack-discuss mailing list