Re: [Release-job-failures] release-post job for openstack/releases for ref refs/heads/master failed
We got two similar release processing failures:
- tag-releases https://zuul.opendev.org/t/openstack/build/707bb50b4bf645f8903d8ca59b43abbf : FAILURE in 2m 31s
This one was an error running the tag-releases post-merge job after https://review.opendev.org/c/openstack/releases/+/779217 merged. It failed with: Running: git push --dry-run ssh://release@review.openstack.org:29418/openstack/tripleo-ipsec.git --all ssh://release@review.openstack.org:29418/openstack/tripleo-ipsec.git did not work. Description: Host key verification failed.
- tag-releases https://zuul.opendev.org/t/openstack/build/655e8980a53c4e829de25f0b8507b5f6 : FAILURE in 2m 57s Similar failure while running the tag-releases post-merge job after https://review.opendev.org/c/openstack/releases/+/779218 merged.
Since this "host key verification failed" error hit tripleo-ipsec three times in a row, I suspect we have something stuck (or we always hit the same cache/worker). -- Thierry Carrez (ttx)
On 2021-03-09 12:39:57 +0100 (+0100), Thierry Carrez wrote:
We got two similar release processing failures: [...] Since this "host key verification failed" error hit tripleo-ipsec three times in a row, I suspect we have something stuck (or we always hit the same cache/worker).
The errors happened when run on nodes in different providers. Looking at the build logs, I also notice that they fail for the stable/rocky branch but succeeded for other branches of the same repository. They're specifically erroring when trying to reach the Gerrit server over SSH, the connection details for which are encoded on the .gitreview file in each branch. This leads me to wonder whether there's something about the fact that the stable/rocky branch of tripleo-ipsec is still using the old review.openstack.org hostname to push, and maybe we're not pre-seeding an appropriate hostkey entry for that in the known_hosts file? -- Jeremy Stanley
On Tue, Mar 9, 2021 at 7:20 PM Jeremy Stanley <fungi@yuggoth.org> wrote:
On 2021-03-09 12:39:57 +0100 (+0100), Thierry Carrez wrote:
We got two similar release processing failures: [...] Since this "host key verification failed" error hit tripleo-ipsec three times in a row, I suspect we have something stuck (or we always hit the same cache/worker).
The errors happened when run on nodes in different providers. Looking at the build logs, I also notice that they fail for the stable/rocky branch but succeeded for other branches of the same repository. They're specifically erroring when trying to reach the Gerrit server over SSH, the connection details for which are encoded on the .gitreview file in each branch. This leads me to wonder whether there's something about the fact that the stable/rocky branch of tripleo-ipsec is still using the old review.openstack.org hostname to push, and maybe we're not pre-seeding an appropriate hostkey entry for that in the known_hosts file?
Hi Thierry, Jeremy is there something tripleo can do to help here? Should I update that https://opendev.org/openstack/tripleo-ipsec/src/commit/ec8aec1d42b9085ed9152... to use opendev.org? Though I know Elod would have words with me if I did ;) as we declared it EOL @ https://review.opendev.org/c/openstack/releases/+/779218 regards, marios
-- Jeremy Stanley
Marios Andreou wrote:
On Tue, Mar 9, 2021 at 7:20 PM Jeremy Stanley <fungi@yuggoth.org <mailto:fungi@yuggoth.org>> wrote:
On 2021-03-09 12:39:57 +0100 (+0100), Thierry Carrez wrote: > We got two similar release processing failures: [...] > Since this "host key verification failed" error hit tripleo-ipsec > three times in a row, I suspect we have something stuck (or we > always hit the same cache/worker).
The errors happened when run on nodes in different providers. Looking at the build logs, I also notice that they fail for the stable/rocky branch but succeeded for other branches of the same repository. They're specifically erroring when trying to reach the Gerrit server over SSH, the connection details for which are encoded on the .gitreview file in each branch. This leads me to wonder whether there's something about the fact that the stable/rocky branch of tripleo-ipsec is still using the old review.openstack.org <http://review.openstack.org> hostname to push, and maybe we're not pre-seeding an appropriate hostkey entry for that in the known_hosts file?
Hi Thierry, Jeremy
is there something tripleo can do to help here?
Should I update that https://opendev.org/openstack/tripleo-ipsec/src/commit/ec8aec1d42b9085ed9152... <https://opendev.org/openstack/tripleo-ipsec/src/commit/ec8aec1d42b9085ed9152ff767eb6744095d9e16/.gitreview#L2> to use opendev.org <http://opendev.org>? Though I know Elod would have words with me if I did ;) as we declared it EOL @ https://review.opendev.org/c/openstack/releases/+/779218 <https://review.opendev.org/c/openstack/releases/+/779218>
I'm not sure... We'll discuss it during the release meeting today. -- Thierry
For the meeting, in advance, my opinion: What Jeremy wrote looks promising. I mean it might be that the root cause is the wrong hostname/hostkey. I think we have two options: 1. fix the hostkey in the known_hosts file (though I don't know how we could do this o:)) 2. technically the tagging has not happened yet (as far as I see), so we ask Marios (:)) to update the .gitreview and then we give another round for the 'remove' and 'readd' rocky-eol tag in tripleo-ipsec.yaml. We can discuss this further @ the meeting today. Cheers, Előd On 2021. 03. 11. 12:42, Thierry Carrez wrote:
Marios Andreou wrote:
On Tue, Mar 9, 2021 at 7:20 PM Jeremy Stanley <fungi@yuggoth.org <mailto:fungi@yuggoth.org>> wrote:
On 2021-03-09 12:39:57 +0100 (+0100), Thierry Carrez wrote: > We got two similar release processing failures: [...] > Since this "host key verification failed" error hit tripleo-ipsec > three times in a row, I suspect we have something stuck (or we > always hit the same cache/worker).
The errors happened when run on nodes in different providers. Looking at the build logs, I also notice that they fail for the stable/rocky branch but succeeded for other branches of the same repository. They're specifically erroring when trying to reach the Gerrit server over SSH, the connection details for which are encoded on the .gitreview file in each branch. This leads me to wonder whether there's something about the fact that the stable/rocky branch of tripleo-ipsec is still using the old review.openstack.org <http://review.openstack.org> hostname to push, and maybe we're not pre-seeding an appropriate hostkey entry for that in the known_hosts file?
Hi Thierry, Jeremy
is there something tripleo can do to help here?
Should I update that https://opendev.org/openstack/tripleo-ipsec/src/commit/ec8aec1d42b9085ed9152... <https://opendev.org/openstack/tripleo-ipsec/src/commit/ec8aec1d42b9085ed9152ff767eb6744095d9e16/.gitreview#L2> to use opendev.org <http://opendev.org>? Though I know Elod would have words with me if I did ;) as we declared it EOL @ https://review.opendev.org/c/openstack/releases/+/779218 <https://review.opendev.org/c/openstack/releases/+/779218>
I'm not sure... We'll discuss it during the release meeting today.
2) can be done easily. My 2 cents on it for now Le jeu. 11 mars 2021 à 14:24, Előd Illés <elod.illes@est.tech> a écrit :
For the meeting, in advance, my opinion:
What Jeremy wrote looks promising. I mean it might be that the root cause is the wrong hostname/hostkey.
I think we have two options:
1. fix the hostkey in the known_hosts file (though I don't know how we could do this o:)) 2. technically the tagging has not happened yet (as far as I see), so we ask Marios (:)) to update the .gitreview and then we give another round for the 'remove' and 'readd' rocky-eol tag in tripleo-ipsec.yaml.
We can discuss this further @ the meeting today.
Cheers,
Előd
On 2021. 03. 11. 12:42, Thierry Carrez wrote:
Marios Andreou wrote:
On Tue, Mar 9, 2021 at 7:20 PM Jeremy Stanley <fungi@yuggoth.org <mailto:fungi@yuggoth.org>> wrote:
On 2021-03-09 12:39:57 +0100 (+0100), Thierry Carrez wrote: > We got two similar release processing failures: [...] > Since this "host key verification failed" error hit tripleo-ipsec > three times in a row, I suspect we have something stuck (or we > always hit the same cache/worker).
The errors happened when run on nodes in different providers. Looking at the build logs, I also notice that they fail for the stable/rocky branch but succeeded for other branches of the same repository. They're specifically erroring when trying to reach the Gerrit server over SSH, the connection details for which are encoded on the .gitreview file in each branch. This leads me to wonder whether there's something about the fact that the stable/rocky branch of tripleo-ipsec is still using the old review.openstack.org <http://review.openstack.org> hostname to push, and maybe we're not pre-seeding an appropriate hostkey entry for that in the known_hosts file?
Hi Thierry, Jeremy
is there something tripleo can do to help here?
Should I update that
https://opendev.org/openstack/tripleo-ipsec/src/commit/ec8aec1d42b9085ed9152...
to use opendev.org <http://opendev.org>? Though I know Elod would have words with me if I did ;) as we declared it EOL @ https://review.opendev.org/c/openstack/releases/+/779218 <https://review.opendev.org/c/openstack/releases/+/779218>
I'm not sure... We'll discuss it during the release meeting today.
-- Hervé Beraud Senior Software Engineer at Red Hat irc: hberaud https://github.com/4383/ https://twitter.com/4383hberaud -----BEGIN PGP SIGNATURE----- wsFcBAABCAAQBQJb4AwCCRAHwXRBNkGNegAALSkQAHrotwCiL3VMwDR0vcja10Q+ Kf31yCutl5bAlS7tOKpPQ9XN4oC0ZSThyNNFVrg8ail0SczHXsC4rOrsPblgGRN+ RQLoCm2eO1AkB0ubCYLaq0XqSaO+Uk81QxAPkyPCEGT6SRxXr2lhADK0T86kBnMP F8RvGolu3EFjlqCVgeOZaR51PqwUlEhZXZuuNKrWZXg/oRiY4811GmnvzmUhgK5G 5+f8mUg74hfjDbR2VhjTeaLKp0PhskjOIKY3vqHXofLuaqFDD+WrAy/NgDGvN22g glGfj472T3xyHnUzM8ILgAGSghfzZF5Skj2qEeci9cB6K3Hm3osj+PbvfsXE/7Kw m/xtm+FjnaywZEv54uCmVIzQsRIm1qJscu20Qw6Q0UiPpDFqD7O6tWSRKdX11UTZ hwVQTMh9AKQDBEh2W9nnFi9kzSSNu4OQ1dRMcYHWfd9BEkccezxHwUM4Xyov5Fe0 qnbfzTB1tYkjU78loMWFaLa00ftSxP/DtQ//iYVyfVNfcCwfDszXLOqlkvGmY1/Y F1ON0ONekDZkGJsDoS6QdiUSn8RZ2mHArGEWMV00EV5DCIbCXRvywXV43ckx8Z+3 B8qUJhBqJ8RS2F+vTs3DTaXqcktgJ4UkhYC2c1gImcPRyGrK9VY0sCT+1iA+wp/O v6rDpkeNksZ9fFSyoY2o =ECSj -----END PGP SIGNATURE-----
On 2021-03-11 12:12:52 +0200 (+0200), Marios Andreou wrote: [...]
is there something tripleo can do to help here?
Should I update that https://opendev.org/openstack/tripleo-ipsec/src/commit/ec8aec1d42b9085ed9152... to use opendev.org? Though I know Elod would have words with me if I did ;) as we declared it EOL @ https://review.opendev.org/c/openstack/releases/+/779218
Yeah, it's interesting that the job wants to push anything for that branch to start with. But when we finish the work to delete those EOL branches I suppose the problem will disappear anyway. -- Jeremy Stanley
On 2021-03-11 16:23:34 +0000 (+0000), Jeremy Stanley wrote: [...]
Yeah, it's interesting that the job wants to push anything for that branch to start with. But when we finish the work to delete those EOL branches I suppose the problem will disappear anyway.
Oh, right, the branch is not *yet* EOL, and it's breaking when trying to push the rocky-eol tag I guess? -- Jeremy Stanley
participants (5)
-
Előd Illés
-
Herve Beraud
-
Jeremy Stanley
-
Marios Andreou
-
Thierry Carrez