Re: [Release-job-failures] Pre-release of openstack/mistral-dashboard for ref refs/tags/10.0.0.0rc1 failed
zuul@openstack.org wrote:
Build failed.
- release-openstack-python https://zuul.opendev.org/t/openstack/build/dacb9f52eaa345b397a28e6dcdf54187 : SUCCESS in 3m 49s - announce-release https://zuul.opendev.org/t/openstack/build/254ac00652174e60869fd1caa11ab261 : FAILURE in 6m 34s - propose-update-constraints https://zuul.opendev.org/t/openstack/build/1a94db3e1ad5473ab4814555ac9b2df4 : SUCCESS in 3m 27s
Transient failure ("Temporary local problem") sending email: + send-mail -v /home/zuul/relnotes/mistral-dashboard-10.0.0.0rc1 send: 'ehlo [127.0.0.1]\r\n' reply: b'250-lists.openstack.org Hello [127.0.0.1] [108.44.198.35]\r\n' reply: b'250-SIZE 52428800\r\n' reply: b'250-8BITMIME\r\n' reply: b'250-PIPELINING\r\n' reply: b'250 HELP\r\n' reply: retcode (250); Msg: b'lists.openstack.org Hello [127.0.0.1] [108.44.198.35]\nSIZE 52428800\n8BITMIME\nPIPELINING\nHELP' send: 'mail FROM:<no-reply@openstack.org> size=1098\r\n' reply: b'250 OK\r\n' reply: retcode (250); Msg: b'OK' send: 'rcpt TO:<release-announce@lists.openstack.org>\r\n' reply: b'451 Temporary local problem - please try later\r\n' reply: retcode (451); Msg: b'Temporary local problem - please try later' This was for mistral-dashboard rc1, I think we can survive that announce not being sent. -- Thierry Carrez (ttx)
Thierry Carrez wrote:
zuul@openstack.org wrote:
Build failed.
- release-openstack-python https://zuul.opendev.org/t/openstack/build/dacb9f52eaa345b397a28e6dcdf54187 : SUCCESS in 3m 49s - announce-release https://zuul.opendev.org/t/openstack/build/254ac00652174e60869fd1caa11ab261 : FAILURE in 6m 34s - propose-update-constraints https://zuul.opendev.org/t/openstack/build/1a94db3e1ad5473ab4814555ac9b2df4 : SUCCESS in 3m 27s
Transient failure ("Temporary local problem") sending email:
+ send-mail -v /home/zuul/relnotes/mistral-dashboard-10.0.0.0rc1 send: 'ehlo [127.0.0.1]\r\n' reply: b'250-lists.openstack.org Hello [127.0.0.1] [108.44.198.35]\r\n' reply: b'250-SIZE 52428800\r\n' reply: b'250-8BITMIME\r\n' reply: b'250-PIPELINING\r\n' reply: b'250 HELP\r\n' reply: retcode (250); Msg: b'lists.openstack.org Hello [127.0.0.1] [108.44.198.35]\nSIZE 52428800\n8BITMIME\nPIPELINING\nHELP' send: 'mail FROM:<no-reply@openstack.org> size=1098\r\n' reply: b'250 OK\r\n' reply: retcode (250); Msg: b'OK' send: 'rcpt TO:<release-announce@lists.openstack.org>\r\n' reply: b'451 Temporary local problem - please try later\r\n' reply: retcode (451); Msg: b'Temporary local problem - please try later'
This was for mistral-dashboard rc1, I think we can survive that announce not being sent.
Same for mistral and watcher-tempest-plugin. This might not be as "temporary" as it appears. -- Thierry Carrez (ttx)
On 2020-04-23 14:53:45 +0200 (+0200), Thierry Carrez wrote: [...]
reply: retcode (451); Msg: b'Temporary local problem - please try later' [...]
We've been witnessing occasional flurries of OOM events on the lists.openstack.org server, and this temporary delivery failure coincided with when one was happening so is almost certainly related. Whether it's a cause or an effect is unknown as we've been so far unable to track down what's actually triggering these sudden spikes in memory utilization, though given the volume of messages the server is handling on a constant basis I have serious doubts release activity is the problem. I suspect these release announcements just had unlucky timing. Normally this wouldn't cause a permanent delivery failure, but the release announcements job doesn't retry delivery on temporary failures. This is something which might be worth looking into adding if a better delivery guarantee is desired. -- Jeremy Stanley
participants (2)
-
Jeremy Stanley
-
Thierry Carrez