Open Stack

On 03/17/2017 09:24 AM, Jordan Pittier wrote:
> 
> 
> On Fri, Mar 17, 2017 at 1:58 PM, Sean Dague <sean at dague.net
> <mailto:sean at dague.net>> wrote:
> 
>     On 03/17/2017 08:27 AM, Jordan Pittier wrote:
>     > The patch that reduced the number of Tempest Scenarios we run in every
>     > job and also reduce the test run concurrency [0] was merged 13 days ago.
>     > Since, the situation (i.e the high number of false negative job results)
>     > has not improved significantly. We need to keep looking collectively at
>     > this.
> 
>     While the situation hasn't completely cleared out -
>     http://tinyurl.com/mdmdxlk - since we've merged this we've not seen that
>     job go over 25% failure rate in the gate, which it was regularly
>     crossing in the prior 2 week period. That does feel like progress. In
>     spot checking I we are also rarely failing in scenario tests now, but
>     the fails tend to end up inside heavy API tests running in parallel.
> 
> 
>     > There seems to be an agreement that we are hitting some memory limit.
>     > Several of our most frequent failures are memory related [1]. So we
>     > should either reduce our memory usage or ask for bigger VMs, with more
>     > than 8GB of RAM.
>     >
>     > There was/is several attempts to reduce our memory usage, by reducing
>     > the Mysql memory consumption ([2] but quickly reverted [3]), reducing
>     > the number of Apache workers ([4], [5]), more apache2 tuning [6]. If you
>     > have any crazy idea to help in this regard, please help. This is high
>     > priority for the whole openstack project, because it's plaguing many
>     > projects.
> 
>     Interesting, I hadn't seen the revert. It is also curious that it was
>     largely limitted to the neutron-api test job. It's also notable that the
>     sort buffers seem to have been set to the minimum allowed limit of mysql
>     -
>     https://dev.mysql.com/doc/refman/5.6/en/innodb-parameters.html#sysvar_innodb_sort_buffer_size
>     <https://dev.mysql.com/doc/refman/5.6/en/innodb-parameters.html#sysvar_innodb_sort_buffer_size>
>     - and is over an order of magnitude decrease from the existing default.
> 
>     I wonder about redoing the change with everything except it and seeing
>     how that impacts the neutron-api job.
> 
> Yes, that would be great because mysql is by far our biggest memory
> consumer so we should target this first.

While it is the single biggest process, weighing in at 500 MB, the
python services are really our biggest memory consumers. They are
collectively far outweighing either mysql or rabbit, and are the reason
that even with 64MB guests we're running out of memory. So we want to
keep that under perspective.

	-Sean

-- 
Sean Dague
http://dague.net