[openstack-dev] [infra][Neutron] Running out of memory on gate for linux bridge job

Matt Riedemann mriedem at linux.vnet.ibm.com
Wed Jan 18 23:24:15 UTC 2017


On 1/13/2017 9:48 AM, Jakub Libosvar wrote:
> Hi,
>
> recently I noticed we got oom-killer in action in one of our jobs [1]. I
> saw it several times, so far only with linux bridge job. The consequence
> is that usually mysqld gets killed as a processes that consumes most of
> the memory, sometimes even nova-api gets killed.
>
> Does anybody know whether we can bump memory on nodes in the gate
> without losing resources for running other jobs?
> Has anybody experience with memory consumption being higher when using
> linux bridge agents?
>
> Any other ideas?
>
> Thanks,
> Jakub
>
> [1]
> http://logs.openstack.org/73/373973/13/check/gate-tempest-dsvm-neutron-linuxbridge-ubuntu-xenial/295d92f/logs/syslog.txt.gz#_Jan_11_13_56_32
>
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>

I don't think it's just the linuxbridge job, see:

http://status.openstack.org//elastic-recheck/index.html#1656850

And the linked logstash query, then expand by build_name.

I also tracked that in logstash to have started around 1/10 which was 
under our 10-days of logs, so something happened around then to start 
tipping us over. I had some leads in the bug report but I think the 
keystone team took over from there.

-- 

Thanks,

Matt Riedemann




More information about the OpenStack-dev mailing list