Open Stack

On Fri, Mar 17, 2017 at 3:11 PM, Sean Dague <sean at dague.net> wrote:

> On 03/17/2017 09:24 AM, Jordan Pittier wrote:
> >
> >
> > On Fri, Mar 17, 2017 at 1:58 PM, Sean Dague <sean at dague.net
> > <mailto:sean at dague.net>> wrote:
> >
> >     On 03/17/2017 08:27 AM, Jordan Pittier wrote:
> >     > The patch that reduced the number of Tempest Scenarios we run in
> every
> >     > job and also reduce the test run concurrency [0] was merged 13
> days ago.
> >     > Since, the situation (i.e the high number of false negative job
> results)
> >     > has not improved significantly. We need to keep looking
> collectively at
> >     > this.
> >
> >     While the situation hasn't completely cleared out -
> >     http://tinyurl.com/mdmdxlk - since we've merged this we've not seen
> that
> >     job go over 25% failure rate in the gate, which it was regularly
> >     crossing in the prior 2 week period. That does feel like progress. In
> >     spot checking I we are also rarely failing in scenario tests now, but
> >     the fails tend to end up inside heavy API tests running in parallel.
> >
> >
> >     > There seems to be an agreement that we are hitting some memory
> limit.
> >     > Several of our most frequent failures are memory related [1]. So we
> >     > should either reduce our memory usage or ask for bigger VMs, with
> more
> >     > than 8GB of RAM.
> >     >
> >     > There was/is several attempts to reduce our memory usage, by
> reducing
> >     > the Mysql memory consumption ([2] but quickly reverted [3]),
> reducing
> >     > the number of Apache workers ([4], [5]), more apache2 tuning [6].
> If you
> >     > have any crazy idea to help in this regard, please help. This is
> high
> >     > priority for the whole openstack project, because it's plaguing
> many
> >     > projects.
> >
> >     Interesting, I hadn't seen the revert. It is also curious that it was
> >     largely limitted to the neutron-api test job. It's also notable that
> the
> >     sort buffers seem to have been set to the minimum allowed limit of
> mysql
> >     -
> >     https://dev.mysql.com/doc/refman/5.6/en/innodb-parameters.
> html#sysvar_innodb_sort_buffer_size
> >     <https://dev.mysql.com/doc/refman/5.6/en/innodb-parameters.
> html#sysvar_innodb_sort_buffer_size>
> >     - and is over an order of magnitude decrease from the existing
> default.
> >
> >     I wonder about redoing the change with everything except it and
> seeing
> >     how that impacts the neutron-api job.
> >
> > Yes, that would be great because mysql is by far our biggest memory
> > consumer so we should target this first.
>
> While it is the single biggest process, weighing in at 500 MB, the
> python services are really our biggest memory consumers. They are
> collectively far outweighing either mysql or rabbit, and are the reason
> that even with 64MB guests we're running out of memory. So we want to
> keep that under perspective.
>
Absolutely. I have https://review.openstack.org/#/c/446986/ in that vain.
And if someone wants to start the work of not running the several Swift
*auditor*, *updater*, *reaper*, *replicator* services, in case the Swift
Replication factor is set to 1, that's also a good memory saving.

>
>         -Sean
>
> --
> Sean Dague
> http://dague.net
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20170317/abff9aec/attachment.html>