Open Stack

Hey all,

So this took me a lot longer than I would have liked (the fact that I
pushed this up in the middle of a conference should be evidence), but I
believe I have managed to track this down:
https://review.openstack.org/#/c/275483/

We also have only ever been clearing our internal cache when reloading. We
should perhaps look at doing that periodically but that is a different
issue to the newly introduced leak. Although I haven't checked, I suspect
our cache is a lot larger recently (looking at cacti and other metrics)
probably due to cross-repo-dependencies so there is more need to keep it
clear. This may have contributed to the recent larger memory use.

Cheers,
Josh

On Sat, Jan 9, 2016 at 7:20 AM, James E. Blair <corvus at inaugust.com> wrote:

> Hi,
>
> Some time before December 17th we merged a change to Zuul that causes a
> memory leak.  Prior to then, the amount of used memory on OpenStack's
> Zuul server consumed a very steady 5GB.  Now it continually increases.
> We can see the behavior start very clearly with our December 17 restart
> of Zuul.  But it had been running for quite some time (perhaps a month
> or more?) before then, so that is not much use in narrowing down the
> change, and several changes were merged in that time period that could
> have an effect.
>
> See this graph:
>
>
> http://cacti.openstack.org/cacti/graph.php?action=zoom&local_graph_id=392&rra_id=4&view_type=&graph_start=1449499887&graph_end=1452197778&graph_height=120&graph_width=500&title_font_size=10
>
> If anyone else running a Zuul from git master has observed increased
> memory usage, would you please let us know any dates where you restarted
> Zuul (with the latest commit) and either did or did not observe
> increased memory usage?  That may help us quickly narrow down what we
> need to look at.
>
> Thanks,
>
> Jim
>
> _______________________________________________
> OpenStack-Infra mailing list
> OpenStack-Infra at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-infra/attachments/20160203/84605421/attachment.html>