<div dir="ltr"><div>Hello,</div><div><br></div><div>After a series of tests this morning and yesterday evening, we can consider our previous issue is now solved, and we can restart approving the releases as usual.</div><div><br></div><div>Indeed, all patches needed to fix our problem are now merged.</div><div><br></div><div>Also the following refs who failed previously have been successfully reenqueued and stein-em tags are now available for the corresponding projects' repos:<br></div><div>- 2f6e3e30f67df4f60e0e2b386755118bb1338059 [1]<br></div><div>- f2b5d964aa5fd3290ca82c0cd12baa29e679b687 [2]<br></div><div>- 3f2136e225577bf80cc90481db8e8aced937e5ec [3]<br></div><div>- 7bf3e98f71afeeafdd703390390677728f1749e6 [4]<br></div><div>- ec68594ba6faeb92eb9ad82410e3ab91db023b19 [5]<br></div><div><br></div><div>To ensure that everything works as expected we also tested the following refs, tags and deliverables have been successfully  created, and no error happened during jobs executions:</div><div>- 1ddbb150446de8ab12676b6c896eadb8ed12d7c3 [6]<br></div><div>- cb115f82fd52605690bdf9bc7c44c0f7d65ff1a4 [7]<br></div><div><br></div><div><span lang="en"><span title="">We can close this topic and return to normal activity.</span></span></div><div><br></div><div>Thanks all for your patience.</div><div><br></div><div>[1] <a href="https://opendev.org/openstack/releases/commit/2f6e3e30f67df4f60e0e2b386755118bb1338059" target="_blank">https://opendev.org/openstack/releases/commit/2f6e3e30f67df4f60e0e2b386755118bb1338059</a></div><div>[2] <a href="https://opendev.org/openstack/releases/commit/f2b5d964aa5fd3290ca82c0cd12baa29e679b687" target="_blank">https://opendev.org/openstack/releases/commit/f2b5d964aa5fd3290ca82c0cd12baa29e679b687</a></div><div>[3] <a href="https://opendev.org/openstack/releases/commit/3f2136e225577bf80cc90481db8e8aced937e5ec" target="_blank">https://opendev.org/openstack/releases/commit/3f2136e225577bf80cc90481db8e8aced937e5ec</a> </div><div>[4] <a href="https://opendev.org/openstack/releases/commit/7bf3e98f71afeeafdd703390390677728f1749e6" target="_blank">https://opendev.org/openstack/releases/commit/7bf3e98f71afeeafdd703390390677728f1749e6</a></div><div>[5] <a href="https://opendev.org/openstack/releases/commit/ec68594ba6faeb92eb9ad82410e3ab91db023b19" target="_blank">https://opendev.org/openstack/releases/commit/ec68594ba6faeb92eb9ad82410e3ab91db023b19</a> </div><div>[6] <a href="https://opendev.org/openstack/releases/commit/1ddbb150446de8ab12676b6c896eadb8ed12d7c3" target="_blank">https://opendev.org/openstack/releases/commit/1ddbb150446de8ab12676b6c896eadb8ed12d7c3</a></div><div>[7] <a href="https://opendev.org/openstack/releases/commit/cb115f82fd52605690bdf9bc7c44c0f7d65ff1a4" target="_blank">https://opendev.org/openstack/releases/commit/cb115f82fd52605690bdf9bc7c44c0f7d65ff1a4</a></div><div><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">Le mar. 24 nov. 2020 à 18:37, Herve Beraud <<a href="mailto:hberaud@redhat.com" target="_blank">hberaud@redhat.com</a>> a écrit :<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div dir="ltr"><div>Just a quick reminder of the current state of the openstack release management.</div><div><br></div><div>We are stuck by the SSH issue and we stopped approving release by waiting to fix all elements of the problem.</div><div><br></div><div>Three patches are on the rails to solve the issue, however one isn't yet merged:</div><div>- <a href="https://review.opendev.org/c/openstack/project-config/+/763797/" target="_blank">https://review.opendev.org/c/openstack/project-config/+/763797/</a> (merged)</div><div>- <a href="https://review.opendev.org/c/openstack/project-config/+/763830/" target="_blank">https://review.opendev.org/c/openstack/project-config/+/763830/</a> (merged)</div><div>- <a href="https://review.opendev.org/c/zuul/zuul-jobs/+/763834/" target="_blank">https://review.opendev.org/c/zuul/zuul-jobs/+/763834/</a> (not merged)</div><div><br></div><div>We are stopping approving releases until this issue isn't resolved.</div><div><br></div><div>When all these patches will be merged we will try to re-enqueue our failing ref to observe if it solves our issue and we will decide if we can restart approving releases.</div><div><br></div><div>I added the PTLs to the object of this thread to inform them about that <span lang="en"><span title="">and also so that they are not surprised that we do not approve their release patches.</span></span></div><div><br></div><div>We will communicate as soon as we will restart approving release patches.</div><div><br></div><div>Thanks for your comprehension.<br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">Le lun. 23 nov. 2020 à 12:49, Herve Beraud <<a href="mailto:hberaud@redhat.com" target="_blank">hberaud@redhat.com</a>> a écrit :<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><pre><table><tbody><tr><td><span>Hello,<br><br></span><span>We are facing SSH issues on our release jobs [1][2].</span><br><span>Indeed, our release account failed to connect to gerrit through SSH [2].  </span><br><span>The ID seems to be recognized when we try it.</span><br><span>So that would probably mean we have a stale host key on the machine the script  is running on .</span><br><span></span><br><span>We suppose that the issue could be a side effect of the gerrit upgrade, we updated the etherpad to inform  the infra team [3]</span><br><span></span><br><span>@release managers: We  should stop approving, so, please hold all our unmerged patches while this incident isn't solved.<br><br>Thanks for your reading,<br></span><span></span></td></tr><tr></tr></tbody></table></pre>[1] <a href="http://lists.openstack.org/pipermail/release-job-failures/2020-November/001487.html%5B2%5D" target="_blank">http://lists.openstack.org/pipermail/release-job-failures/2020-November/001487.html[2]</a> <br>Example of faced error:<br>```                                        <br>ssh://<a href="http://release@review.opendev.org:29418/openstack/python-senlinclient.git" target="_blank">release@review.opendev.org:29418/openstack/python-senlinclient.git</a> did not work. Description: Host key verification failed.<br>Could not read from remote repository <br>```<div>[3] <a href="https://etherpad.opendev.org/p/gerrit-3.2-post-upgrade-notes" target="_blank">https://etherpad.opendev.org/p/gerrit-3.2-post-upgrade-notes</a></div><br>-- <br><div dir="ltr"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div>Hervé Beraud</div><div>Senior Software Engineer at Red Hat</div><div>irc: hberaud</div><div><a href="https://github.com/4383/" target="_blank">https://github.com/4383/</a></div><div><a href="https://twitter.com/4383hberaud" target="_blank">https://twitter.com/4383hberaud</a><br></div><div>-----BEGIN PGP SIGNATURE-----<br><br>wsFcBAABCAAQBQJb4AwCCRAHwXRBNkGNegAALSkQAHrotwCiL3VMwDR0vcja10Q+<br>Kf31yCutl5bAlS7tOKpPQ9XN4oC0ZSThyNNFVrg8ail0SczHXsC4rOrsPblgGRN+<br>RQLoCm2eO1AkB0ubCYLaq0XqSaO+Uk81QxAPkyPCEGT6SRxXr2lhADK0T86kBnMP<br>F8RvGolu3EFjlqCVgeOZaR51PqwUlEhZXZuuNKrWZXg/oRiY4811GmnvzmUhgK5G<br>5+f8mUg74hfjDbR2VhjTeaLKp0PhskjOIKY3vqHXofLuaqFDD+WrAy/NgDGvN22g<br>glGfj472T3xyHnUzM8ILgAGSghfzZF5Skj2qEeci9cB6K3Hm3osj+PbvfsXE/7Kw<br>m/xtm+FjnaywZEv54uCmVIzQsRIm1qJscu20Qw6Q0UiPpDFqD7O6tWSRKdX11UTZ<br>hwVQTMh9AKQDBEh2W9nnFi9kzSSNu4OQ1dRMcYHWfd9BEkccezxHwUM4Xyov5Fe0<br>qnbfzTB1tYkjU78loMWFaLa00ftSxP/DtQ//iYVyfVNfcCwfDszXLOqlkvGmY1/Y<br>F1ON0ONekDZkGJsDoS6QdiUSn8RZ2mHArGEWMV00EV5DCIbCXRvywXV43ckx8Z+3<br>B8qUJhBqJ8RS2F+vTs3DTaXqcktgJ4UkhYC2c1gImcPRyGrK9VY0sCT+1iA+wp/O<br>v6rDpkeNksZ9fFSyoY2o<br>=ECSj<br>-----END PGP SIGNATURE-----<br><br></div></div></div></div></div></div></div></div></div></div></div></div></div></div>
</blockquote></div><br clear="all"><br>-- <br><div dir="ltr"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div>Hervé Beraud</div><div>Senior Software Engineer at Red Hat</div><div>irc: hberaud</div><div><a href="https://github.com/4383/" target="_blank">https://github.com/4383/</a></div><div><a href="https://twitter.com/4383hberaud" target="_blank">https://twitter.com/4383hberaud</a><br></div><div>-----BEGIN PGP SIGNATURE-----<br><br>wsFcBAABCAAQBQJb4AwCCRAHwXRBNkGNegAALSkQAHrotwCiL3VMwDR0vcja10Q+<br>Kf31yCutl5bAlS7tOKpPQ9XN4oC0ZSThyNNFVrg8ail0SczHXsC4rOrsPblgGRN+<br>RQLoCm2eO1AkB0ubCYLaq0XqSaO+Uk81QxAPkyPCEGT6SRxXr2lhADK0T86kBnMP<br>F8RvGolu3EFjlqCVgeOZaR51PqwUlEhZXZuuNKrWZXg/oRiY4811GmnvzmUhgK5G<br>5+f8mUg74hfjDbR2VhjTeaLKp0PhskjOIKY3vqHXofLuaqFDD+WrAy/NgDGvN22g<br>glGfj472T3xyHnUzM8ILgAGSghfzZF5Skj2qEeci9cB6K3Hm3osj+PbvfsXE/7Kw<br>m/xtm+FjnaywZEv54uCmVIzQsRIm1qJscu20Qw6Q0UiPpDFqD7O6tWSRKdX11UTZ<br>hwVQTMh9AKQDBEh2W9nnFi9kzSSNu4OQ1dRMcYHWfd9BEkccezxHwUM4Xyov5Fe0<br>qnbfzTB1tYkjU78loMWFaLa00ftSxP/DtQ//iYVyfVNfcCwfDszXLOqlkvGmY1/Y<br>F1ON0ONekDZkGJsDoS6QdiUSn8RZ2mHArGEWMV00EV5DCIbCXRvywXV43ckx8Z+3<br>B8qUJhBqJ8RS2F+vTs3DTaXqcktgJ4UkhYC2c1gImcPRyGrK9VY0sCT+1iA+wp/O<br>v6rDpkeNksZ9fFSyoY2o<br>=ECSj<br>-----END PGP SIGNATURE-----<br><br></div></div></div></div></div></div></div></div></div></div></div></div></div></div>
</blockquote></div><br clear="all"><br>-- <br><div dir="ltr"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div>Hervé Beraud</div><div>Senior Software Engineer at Red Hat</div><div>irc: hberaud</div><div><a href="https://github.com/4383/" target="_blank">https://github.com/4383/</a></div><div><a href="https://twitter.com/4383hberaud" target="_blank">https://twitter.com/4383hberaud</a><br></div><div>-----BEGIN PGP SIGNATURE-----<br><br>wsFcBAABCAAQBQJb4AwCCRAHwXRBNkGNegAALSkQAHrotwCiL3VMwDR0vcja10Q+<br>Kf31yCutl5bAlS7tOKpPQ9XN4oC0ZSThyNNFVrg8ail0SczHXsC4rOrsPblgGRN+<br>RQLoCm2eO1AkB0ubCYLaq0XqSaO+Uk81QxAPkyPCEGT6SRxXr2lhADK0T86kBnMP<br>F8RvGolu3EFjlqCVgeOZaR51PqwUlEhZXZuuNKrWZXg/oRiY4811GmnvzmUhgK5G<br>5+f8mUg74hfjDbR2VhjTeaLKp0PhskjOIKY3vqHXofLuaqFDD+WrAy/NgDGvN22g<br>glGfj472T3xyHnUzM8ILgAGSghfzZF5Skj2qEeci9cB6K3Hm3osj+PbvfsXE/7Kw<br>m/xtm+FjnaywZEv54uCmVIzQsRIm1qJscu20Qw6Q0UiPpDFqD7O6tWSRKdX11UTZ<br>hwVQTMh9AKQDBEh2W9nnFi9kzSSNu4OQ1dRMcYHWfd9BEkccezxHwUM4Xyov5Fe0<br>qnbfzTB1tYkjU78loMWFaLa00ftSxP/DtQ//iYVyfVNfcCwfDszXLOqlkvGmY1/Y<br>F1ON0ONekDZkGJsDoS6QdiUSn8RZ2mHArGEWMV00EV5DCIbCXRvywXV43ckx8Z+3<br>B8qUJhBqJ8RS2F+vTs3DTaXqcktgJ4UkhYC2c1gImcPRyGrK9VY0sCT+1iA+wp/O<br>v6rDpkeNksZ9fFSyoY2o<br>=ECSj<br>-----END PGP SIGNATURE-----<br><br></div></div></div></div></div></div></div></div></div></div></div></div></div>