[infra] Update on test throughput and Zuul backlogs

Matt Riedemann mriedemos at gmail.com
Sat Dec 8 18:28:47 UTC 2018


On 12/6/2018 5:16 PM, Clark Boylan wrote:
> All that said flaky tests are still an issue. One set of problems seems related to slower than expected/before test nodes in the BHS1 region. We've been debugging these with OVH (thank you amorin!) and think we've managed to make some improvements though so far the problems persist. Current theory is that we are acting as our own noisy neighbors starving the hypervisors of disk IO throughput. In order to test that we've halved the total number of resources we'll use there. More details athttps://etherpad.openstack.org/p/bhs1-test-node-slowness  including a list of e-r bugs that may be tied to this issue.
> 
> One thing to keep in mind is that while the test nodes are slower than we'd like, they have also exposed some situations where our software is less efficient than we'd like. At least one bug,https://bugs.launchpad.net/nova/+bug/1807219, has been identified through this. I would encourage people debugging these slow tests to look to see if this exposes a deficiency in our software that can be fixed.

Here are a couple of fixes for recently fingerprinted gate bugs:

https://review.openstack.org/#/c/623669/

https://review.openstack.org/#/c/623597/

Those are in grenade and devstack respectively so we'll need some QA cores.

-- 

Thanks,

Matt



More information about the openstack-discuss mailing list