Hello, We are facing SSH issues on our release jobs [1][2]. Indeed, our release account failed to connect to gerrit through SSH [2]. The ID seems to be recognized when we try it. So that would probably mean we have a stale host key on the machine the script is running on . We suppose that the issue could be a side effect of the gerrit upgrade, we updated the etherpad to inform the infra team [3] @release managers: We should stop approving, so, please hold all our unmerged patches while this incident isn't solved. Thanks for your reading, [1] http://lists.openstack.org/pipermail/release-job-failures/2020-November/001487.html[2] Example of faced error: ``` ssh://release@review.opendev.org:29418/openstack/python-senlinclient.git did not work. Description: Host key verification failed. Could not read from remote repository ``` [3] https://etherpad.opendev.org/p/gerrit-3.2-post-upgrade-notes -- Hervé Beraud Senior Software Engineer at Red Hat irc: hberaud https://github.com/4383/ https://twitter.com/4383hberaud -----BEGIN PGP SIGNATURE----- wsFcBAABCAAQBQJb4AwCCRAHwXRBNkGNegAALSkQAHrotwCiL3VMwDR0vcja10Q+ Kf31yCutl5bAlS7tOKpPQ9XN4oC0ZSThyNNFVrg8ail0SczHXsC4rOrsPblgGRN+ RQLoCm2eO1AkB0ubCYLaq0XqSaO+Uk81QxAPkyPCEGT6SRxXr2lhADK0T86kBnMP F8RvGolu3EFjlqCVgeOZaR51PqwUlEhZXZuuNKrWZXg/oRiY4811GmnvzmUhgK5G 5+f8mUg74hfjDbR2VhjTeaLKp0PhskjOIKY3vqHXofLuaqFDD+WrAy/NgDGvN22g glGfj472T3xyHnUzM8ILgAGSghfzZF5Skj2qEeci9cB6K3Hm3osj+PbvfsXE/7Kw m/xtm+FjnaywZEv54uCmVIzQsRIm1qJscu20Qw6Q0UiPpDFqD7O6tWSRKdX11UTZ hwVQTMh9AKQDBEh2W9nnFi9kzSSNu4OQ1dRMcYHWfd9BEkccezxHwUM4Xyov5Fe0 qnbfzTB1tYkjU78loMWFaLa00ftSxP/DtQ//iYVyfVNfcCwfDszXLOqlkvGmY1/Y F1ON0ONekDZkGJsDoS6QdiUSn8RZ2mHArGEWMV00EV5DCIbCXRvywXV43ckx8Z+3 B8qUJhBqJ8RS2F+vTs3DTaXqcktgJ4UkhYC2c1gImcPRyGrK9VY0sCT+1iA+wp/O v6rDpkeNksZ9fFSyoY2o =ECSj -----END PGP SIGNATURE-----
Just a quick reminder of the current state of the openstack release management. We are stuck by the SSH issue and we stopped approving release by waiting to fix all elements of the problem. Three patches are on the rails to solve the issue, however one isn't yet merged: - https://review.opendev.org/c/openstack/project-config/+/763797/ (merged) - https://review.opendev.org/c/openstack/project-config/+/763830/ (merged) - https://review.opendev.org/c/zuul/zuul-jobs/+/763834/ (not merged) We are stopping approving releases until this issue isn't resolved. When all these patches will be merged we will try to re-enqueue our failing ref to observe if it solves our issue and we will decide if we can restart approving releases. I added the PTLs to the object of this thread to inform them about that and also so that they are not surprised that we do not approve their release patches. We will communicate as soon as we will restart approving release patches. Thanks for your comprehension. Le lun. 23 nov. 2020 à 12:49, Herve Beraud <hberaud@redhat.com> a écrit :
Hello,
We are facing SSH issues on our release jobs [1][2]. Indeed, our release account failed to connect to gerrit through SSH [2]. The ID seems to be recognized when we try it. So that would probably mean we have a stale host key on the machine the script is running on .
We suppose that the issue could be a side effect of the gerrit upgrade, we updated the etherpad to inform the infra team [3]
@release managers: We should stop approving, so, please hold all our unmerged patches while this incident isn't solved.
Thanks for your reading,
[1] http://lists.openstack.org/pipermail/release-job-failures/2020-November/001487.html[2] Example of faced error: ``` ssh://release@review.opendev.org:29418/openstack/python-senlinclient.git did not work. Description: Host key verification failed. Could not read from remote repository ``` [3] https://etherpad.opendev.org/p/gerrit-3.2-post-upgrade-notes
-- Hervé Beraud Senior Software Engineer at Red Hat irc: hberaud https://github.com/4383/ https://twitter.com/4383hberaud -----BEGIN PGP SIGNATURE-----
wsFcBAABCAAQBQJb4AwCCRAHwXRBNkGNegAALSkQAHrotwCiL3VMwDR0vcja10Q+ Kf31yCutl5bAlS7tOKpPQ9XN4oC0ZSThyNNFVrg8ail0SczHXsC4rOrsPblgGRN+ RQLoCm2eO1AkB0ubCYLaq0XqSaO+Uk81QxAPkyPCEGT6SRxXr2lhADK0T86kBnMP F8RvGolu3EFjlqCVgeOZaR51PqwUlEhZXZuuNKrWZXg/oRiY4811GmnvzmUhgK5G 5+f8mUg74hfjDbR2VhjTeaLKp0PhskjOIKY3vqHXofLuaqFDD+WrAy/NgDGvN22g glGfj472T3xyHnUzM8ILgAGSghfzZF5Skj2qEeci9cB6K3Hm3osj+PbvfsXE/7Kw m/xtm+FjnaywZEv54uCmVIzQsRIm1qJscu20Qw6Q0UiPpDFqD7O6tWSRKdX11UTZ hwVQTMh9AKQDBEh2W9nnFi9kzSSNu4OQ1dRMcYHWfd9BEkccezxHwUM4Xyov5Fe0 qnbfzTB1tYkjU78loMWFaLa00ftSxP/DtQ//iYVyfVNfcCwfDszXLOqlkvGmY1/Y F1ON0ONekDZkGJsDoS6QdiUSn8RZ2mHArGEWMV00EV5DCIbCXRvywXV43ckx8Z+3 B8qUJhBqJ8RS2F+vTs3DTaXqcktgJ4UkhYC2c1gImcPRyGrK9VY0sCT+1iA+wp/O v6rDpkeNksZ9fFSyoY2o =ECSj -----END PGP SIGNATURE-----
-- Hervé Beraud Senior Software Engineer at Red Hat irc: hberaud https://github.com/4383/ https://twitter.com/4383hberaud -----BEGIN PGP SIGNATURE----- wsFcBAABCAAQBQJb4AwCCRAHwXRBNkGNegAALSkQAHrotwCiL3VMwDR0vcja10Q+ Kf31yCutl5bAlS7tOKpPQ9XN4oC0ZSThyNNFVrg8ail0SczHXsC4rOrsPblgGRN+ RQLoCm2eO1AkB0ubCYLaq0XqSaO+Uk81QxAPkyPCEGT6SRxXr2lhADK0T86kBnMP F8RvGolu3EFjlqCVgeOZaR51PqwUlEhZXZuuNKrWZXg/oRiY4811GmnvzmUhgK5G 5+f8mUg74hfjDbR2VhjTeaLKp0PhskjOIKY3vqHXofLuaqFDD+WrAy/NgDGvN22g glGfj472T3xyHnUzM8ILgAGSghfzZF5Skj2qEeci9cB6K3Hm3osj+PbvfsXE/7Kw m/xtm+FjnaywZEv54uCmVIzQsRIm1qJscu20Qw6Q0UiPpDFqD7O6tWSRKdX11UTZ hwVQTMh9AKQDBEh2W9nnFi9kzSSNu4OQ1dRMcYHWfd9BEkccezxHwUM4Xyov5Fe0 qnbfzTB1tYkjU78loMWFaLa00ftSxP/DtQ//iYVyfVNfcCwfDszXLOqlkvGmY1/Y F1ON0ONekDZkGJsDoS6QdiUSn8RZ2mHArGEWMV00EV5DCIbCXRvywXV43ckx8Z+3 B8qUJhBqJ8RS2F+vTs3DTaXqcktgJ4UkhYC2c1gImcPRyGrK9VY0sCT+1iA+wp/O v6rDpkeNksZ9fFSyoY2o =ECSj -----END PGP SIGNATURE-----
Hello, After a series of tests this morning and yesterday evening, we can consider our previous issue is now solved, and we can restart approving the releases as usual. Indeed, all patches needed to fix our problem are now merged. Also the following refs who failed previously have been successfully reenqueued and stein-em tags are now available for the corresponding projects' repos: - 2f6e3e30f67df4f60e0e2b386755118bb1338059 [1] - f2b5d964aa5fd3290ca82c0cd12baa29e679b687 [2] - 3f2136e225577bf80cc90481db8e8aced937e5ec [3] - 7bf3e98f71afeeafdd703390390677728f1749e6 [4] - ec68594ba6faeb92eb9ad82410e3ab91db023b19 [5] To ensure that everything works as expected we also tested the following refs, tags and deliverables have been successfully created, and no error happened during jobs executions: - 1ddbb150446de8ab12676b6c896eadb8ed12d7c3 [6] - cb115f82fd52605690bdf9bc7c44c0f7d65ff1a4 [7] We can close this topic and return to normal activity. Thanks all for your patience. [1] https://opendev.org/openstack/releases/commit/2f6e3e30f67df4f60e0e2b38675511... [2] https://opendev.org/openstack/releases/commit/f2b5d964aa5fd3290ca82c0cd12baa... [3] https://opendev.org/openstack/releases/commit/3f2136e225577bf80cc90481db8e8a... [4] https://opendev.org/openstack/releases/commit/7bf3e98f71afeeafdd703390390677... [5] https://opendev.org/openstack/releases/commit/ec68594ba6faeb92eb9ad82410e3ab... [6] https://opendev.org/openstack/releases/commit/1ddbb150446de8ab12676b6c896ead... [7] https://opendev.org/openstack/releases/commit/cb115f82fd52605690bdf9bc7c44c0... Le mar. 24 nov. 2020 à 18:37, Herve Beraud <hberaud@redhat.com> a écrit :
Just a quick reminder of the current state of the openstack release management.
We are stuck by the SSH issue and we stopped approving release by waiting to fix all elements of the problem.
Three patches are on the rails to solve the issue, however one isn't yet merged: - https://review.opendev.org/c/openstack/project-config/+/763797/ (merged) - https://review.opendev.org/c/openstack/project-config/+/763830/ (merged) - https://review.opendev.org/c/zuul/zuul-jobs/+/763834/ (not merged)
We are stopping approving releases until this issue isn't resolved.
When all these patches will be merged we will try to re-enqueue our failing ref to observe if it solves our issue and we will decide if we can restart approving releases.
I added the PTLs to the object of this thread to inform them about that and also so that they are not surprised that we do not approve their release patches.
We will communicate as soon as we will restart approving release patches.
Thanks for your comprehension.
Le lun. 23 nov. 2020 à 12:49, Herve Beraud <hberaud@redhat.com> a écrit :
Hello,
We are facing SSH issues on our release jobs [1][2]. Indeed, our release account failed to connect to gerrit through SSH [2]. The ID seems to be recognized when we try it. So that would probably mean we have a stale host key on the machine the script is running on .
We suppose that the issue could be a side effect of the gerrit upgrade, we updated the etherpad to inform the infra team [3]
@release managers: We should stop approving, so, please hold all our unmerged patches while this incident isn't solved.
Thanks for your reading,
[1] http://lists.openstack.org/pipermail/release-job-failures/2020-November/001487.html[2] Example of faced error: ``` ssh://release@review.opendev.org:29418/openstack/python-senlinclient.git did not work. Description: Host key verification failed. Could not read from remote repository ``` [3] https://etherpad.opendev.org/p/gerrit-3.2-post-upgrade-notes
-- Hervé Beraud Senior Software Engineer at Red Hat irc: hberaud https://github.com/4383/ https://twitter.com/4383hberaud -----BEGIN PGP SIGNATURE-----
wsFcBAABCAAQBQJb4AwCCRAHwXRBNkGNegAALSkQAHrotwCiL3VMwDR0vcja10Q+ Kf31yCutl5bAlS7tOKpPQ9XN4oC0ZSThyNNFVrg8ail0SczHXsC4rOrsPblgGRN+ RQLoCm2eO1AkB0ubCYLaq0XqSaO+Uk81QxAPkyPCEGT6SRxXr2lhADK0T86kBnMP F8RvGolu3EFjlqCVgeOZaR51PqwUlEhZXZuuNKrWZXg/oRiY4811GmnvzmUhgK5G 5+f8mUg74hfjDbR2VhjTeaLKp0PhskjOIKY3vqHXofLuaqFDD+WrAy/NgDGvN22g glGfj472T3xyHnUzM8ILgAGSghfzZF5Skj2qEeci9cB6K3Hm3osj+PbvfsXE/7Kw m/xtm+FjnaywZEv54uCmVIzQsRIm1qJscu20Qw6Q0UiPpDFqD7O6tWSRKdX11UTZ hwVQTMh9AKQDBEh2W9nnFi9kzSSNu4OQ1dRMcYHWfd9BEkccezxHwUM4Xyov5Fe0 qnbfzTB1tYkjU78loMWFaLa00ftSxP/DtQ//iYVyfVNfcCwfDszXLOqlkvGmY1/Y F1ON0ONekDZkGJsDoS6QdiUSn8RZ2mHArGEWMV00EV5DCIbCXRvywXV43ckx8Z+3 B8qUJhBqJ8RS2F+vTs3DTaXqcktgJ4UkhYC2c1gImcPRyGrK9VY0sCT+1iA+wp/O v6rDpkeNksZ9fFSyoY2o =ECSj -----END PGP SIGNATURE-----
-- Hervé Beraud Senior Software Engineer at Red Hat irc: hberaud https://github.com/4383/ https://twitter.com/4383hberaud -----BEGIN PGP SIGNATURE-----
wsFcBAABCAAQBQJb4AwCCRAHwXRBNkGNegAALSkQAHrotwCiL3VMwDR0vcja10Q+ Kf31yCutl5bAlS7tOKpPQ9XN4oC0ZSThyNNFVrg8ail0SczHXsC4rOrsPblgGRN+ RQLoCm2eO1AkB0ubCYLaq0XqSaO+Uk81QxAPkyPCEGT6SRxXr2lhADK0T86kBnMP F8RvGolu3EFjlqCVgeOZaR51PqwUlEhZXZuuNKrWZXg/oRiY4811GmnvzmUhgK5G 5+f8mUg74hfjDbR2VhjTeaLKp0PhskjOIKY3vqHXofLuaqFDD+WrAy/NgDGvN22g glGfj472T3xyHnUzM8ILgAGSghfzZF5Skj2qEeci9cB6K3Hm3osj+PbvfsXE/7Kw m/xtm+FjnaywZEv54uCmVIzQsRIm1qJscu20Qw6Q0UiPpDFqD7O6tWSRKdX11UTZ hwVQTMh9AKQDBEh2W9nnFi9kzSSNu4OQ1dRMcYHWfd9BEkccezxHwUM4Xyov5Fe0 qnbfzTB1tYkjU78loMWFaLa00ftSxP/DtQ//iYVyfVNfcCwfDszXLOqlkvGmY1/Y F1ON0ONekDZkGJsDoS6QdiUSn8RZ2mHArGEWMV00EV5DCIbCXRvywXV43ckx8Z+3 B8qUJhBqJ8RS2F+vTs3DTaXqcktgJ4UkhYC2c1gImcPRyGrK9VY0sCT+1iA+wp/O v6rDpkeNksZ9fFSyoY2o =ECSj -----END PGP SIGNATURE-----
-- Hervé Beraud Senior Software Engineer at Red Hat irc: hberaud https://github.com/4383/ https://twitter.com/4383hberaud -----BEGIN PGP SIGNATURE----- wsFcBAABCAAQBQJb4AwCCRAHwXRBNkGNegAALSkQAHrotwCiL3VMwDR0vcja10Q+ Kf31yCutl5bAlS7tOKpPQ9XN4oC0ZSThyNNFVrg8ail0SczHXsC4rOrsPblgGRN+ RQLoCm2eO1AkB0ubCYLaq0XqSaO+Uk81QxAPkyPCEGT6SRxXr2lhADK0T86kBnMP F8RvGolu3EFjlqCVgeOZaR51PqwUlEhZXZuuNKrWZXg/oRiY4811GmnvzmUhgK5G 5+f8mUg74hfjDbR2VhjTeaLKp0PhskjOIKY3vqHXofLuaqFDD+WrAy/NgDGvN22g glGfj472T3xyHnUzM8ILgAGSghfzZF5Skj2qEeci9cB6K3Hm3osj+PbvfsXE/7Kw m/xtm+FjnaywZEv54uCmVIzQsRIm1qJscu20Qw6Q0UiPpDFqD7O6tWSRKdX11UTZ hwVQTMh9AKQDBEh2W9nnFi9kzSSNu4OQ1dRMcYHWfd9BEkccezxHwUM4Xyov5Fe0 qnbfzTB1tYkjU78loMWFaLa00ftSxP/DtQ//iYVyfVNfcCwfDszXLOqlkvGmY1/Y F1ON0ONekDZkGJsDoS6QdiUSn8RZ2mHArGEWMV00EV5DCIbCXRvywXV43ckx8Z+3 B8qUJhBqJ8RS2F+vTs3DTaXqcktgJ4UkhYC2c1gImcPRyGrK9VY0sCT+1iA+wp/O v6rDpkeNksZ9fFSyoY2o =ECSj -----END PGP SIGNATURE-----
participants (1)
-
Herve Beraud