[openstack-dev] [QA][gate][all] dsvm gate stability and scenario tests
jordan.pittier at scality.com
Fri Mar 17 13:24:55 UTC 2017
On Fri, Mar 17, 2017 at 1:58 PM, Sean Dague <sean at dague.net> wrote:
> On 03/17/2017 08:27 AM, Jordan Pittier wrote:
> > The patch that reduced the number of Tempest Scenarios we run in every
> > job and also reduce the test run concurrency  was merged 13 days ago.
> > Since, the situation (i.e the high number of false negative job results)
> > has not improved significantly. We need to keep looking collectively at
> > this.
> While the situation hasn't completely cleared out -
> http://tinyurl.com/mdmdxlk - since we've merged this we've not seen that
> job go over 25% failure rate in the gate, which it was regularly
> crossing in the prior 2 week period. That does feel like progress. In
> spot checking I we are also rarely failing in scenario tests now, but
> the fails tend to end up inside heavy API tests running in parallel.
> > There seems to be an agreement that we are hitting some memory limit.
> > Several of our most frequent failures are memory related . So we
> > should either reduce our memory usage or ask for bigger VMs, with more
> > than 8GB of RAM.
> > There was/is several attempts to reduce our memory usage, by reducing
> > the Mysql memory consumption ( but quickly reverted ), reducing
> > the number of Apache workers (, ), more apache2 tuning . If you
> > have any crazy idea to help in this regard, please help. This is high
> > priority for the whole openstack project, because it's plaguing many
> > projects.
> Interesting, I hadn't seen the revert. It is also curious that it was
> largely limitted to the neutron-api test job. It's also notable that the
> sort buffers seem to have been set to the minimum allowed limit of mysql
> - and is over an order of magnitude decrease from the existing default.
> I wonder about redoing the change with everything except it and seeing
> how that impacts the neutron-api job.
Yes, that would be great because mysql is by far our biggest memory
consumer so we should target this first.
> Sean Dague
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
-------------- next part --------------
An HTML attachment was scrubbed...
More information about the OpenStack-dev