Open Stack

On 03/17/2017 08:27 AM, Jordan Pittier wrote:
> The patch that reduced the number of Tempest Scenarios we run in every
> job and also reduce the test run concurrency [0] was merged 13 days ago.
> Since, the situation (i.e the high number of false negative job results)
> has not improved significantly. We need to keep looking collectively at
> this.

While the situation hasn't completely cleared out -
http://tinyurl.com/mdmdxlk - since we've merged this we've not seen that
job go over 25% failure rate in the gate, which it was regularly
crossing in the prior 2 week period. That does feel like progress. In
spot checking I we are also rarely failing in scenario tests now, but
the fails tend to end up inside heavy API tests running in parallel.

> There seems to be an agreement that we are hitting some memory limit.
> Several of our most frequent failures are memory related [1]. So we
> should either reduce our memory usage or ask for bigger VMs, with more
> than 8GB of RAM.
> 
> There was/is several attempts to reduce our memory usage, by reducing
> the Mysql memory consumption ([2] but quickly reverted [3]), reducing
> the number of Apache workers ([4], [5]), more apache2 tuning [6]. If you
> have any crazy idea to help in this regard, please help. This is high
> priority for the whole openstack project, because it's plaguing many
> projects.

Interesting, I hadn't seen the revert. It is also curious that it was
largely limitted to the neutron-api test job. It's also notable that the
sort buffers seem to have been set to the minimum allowed limit of mysql
-
https://dev.mysql.com/doc/refman/5.6/en/innodb-parameters.html#sysvar_innodb_sort_buffer_size
- and is over an order of magnitude decrease from the existing default.

I wonder about redoing the change with everything except it and seeing
how that impacts the neutron-api job.

	-Sean

-- 
Sean Dague
http://dague.net