Open Stack

Awesome detective work, Joshua, thank you!

-jay

On 02/03/2016 03:56 AM, Joshua Hesketh wrote:
> Hey all,
>
> So this took me a lot longer than I would have liked (the fact that I
> pushed this up in the middle of a conference should be evidence), but I
> believe I have managed to track this down:
> https://review.openstack.org/#/c/275483/
>
> We also have only ever been clearing our internal cache when reloading.
> We should perhaps look at doing that periodically but that is a
> different issue to the newly introduced leak. Although I haven't
> checked, I suspect our cache is a lot larger recently (looking at cacti
> and other metrics) probably due to cross-repo-dependencies so there is
> more need to keep it clear. This may have contributed to the recent
> larger memory use.
>
> Cheers,
> Josh
>
> On Sat, Jan 9, 2016 at 7:20 AM, James E. Blair <corvus at inaugust.com
> <mailto:corvus at inaugust.com>> wrote:
>
>     Hi,
>
>     Some time before December 17th we merged a change to Zuul that causes a
>     memory leak.  Prior to then, the amount of used memory on OpenStack's
>     Zuul server consumed a very steady 5GB.  Now it continually increases.
>     We can see the behavior start very clearly with our December 17 restart
>     of Zuul.  But it had been running for quite some time (perhaps a month
>     or more?) before then, so that is not much use in narrowing down the
>     change, and several changes were merged in that time period that could
>     have an effect.
>
>     See this graph:
>
>     http://cacti.openstack.org/cacti/graph.php?action=zoom&local_graph_id=392&rra_id=4&view_type=&graph_start=1449499887&graph_end=1452197778&graph_height=120&graph_width=500&title_font_size=10
>
>     If anyone else running a Zuul from git master has observed increased
>     memory usage, would you please let us know any dates where you restarted
>     Zuul (with the latest commit) and either did or did not observe
>     increased memory usage?  That may help us quickly narrow down what we
>     need to look at.
>
>     Thanks,
>
>     Jim
>
>     _______________________________________________
>     OpenStack-Infra mailing list
>     OpenStack-Infra at lists.openstack.org
>     <mailto:OpenStack-Infra at lists.openstack.org>
>     http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
>
>
>
>
> _______________________________________________
> OpenStack-Infra mailing list
> OpenStack-Infra at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
>