On 12/12/2018 2:00 PM, Matt Riedemann wrote:
I wanted to send this separate from the latest gate status update [1] since it's primarily about latent cinder bugs causing failures in the gate for which no one is really investigating.
Matt, thank you for putting together this information. I am sorry that these issues with Cinder are impacting Nova's ability to merge code. I don't think we knew that this was having an impact on Nova.
Running down our tracked gate bugs [2] there are several related to cinder-backup testing:
* http://status.openstack.org/elastic-recheck/#1483434 * http://status.openstack.org/elastic-recheck/#1745168 * http://status.openstack.org/elastic-recheck/#1739482 * http://status.openstack.org/elastic-recheck/#1635643
All of those bugs were reported a long time ago. I've done some investigation into them (at least at the time of reporting) and some are simply due to cinder-api using synchronous RPC calls to cinder-volume (or cinder-backup) and that doesn't scale. This bug isn't a backup issue, but it's definitely related to using RPC call rather than cast:
Thanks to bringing this up Dan Smith has proposed a patch that may help with the timeouts. https://review.openstack.org/#/c/624809/ The thought is that cocurrent LVM processes might be the source of the timeout. We will continue to work with Dan on that patch.
Regarding the backup tests specifically, I don't see a reason why they need to be run in the integrated gate jobs, e.g. tempest-full(-py3). They don't involve other services, so in my opinion we should move the backup tests to a separate job which only runs on cinder changes to alleviate these latent bugs failing jobs for unrelated changes and resetting the entire gate.
I would need someone from the cinder team that is more involved in knowing what their job setup looks like to identify a candidate job for these tests if this is something everyone can agree on doing.
We have a member of the team that might have some bandwidth to start working on check/gate issues. I have added this issue to our meeting agenda for next week. We should be able to get attention from the team members can help at that point in time.
[1] http://lists.openstack.org/pipermail/openstack-discuss/2018-December/000867.... [2] http://status.openstack.org/elastic-recheck/