Open Stack

On Thu, Feb 4, 2016 at 2:44 AM, James E. Blair <corvus at inaugust.com> wrote:

> Joshua Hesketh <joshua.hesketh at gmail.com> writes:
>
> > Hey all,
> >
> > So this took me a lot longer than I would have liked (the fact that I
> > pushed this up in the middle of a conference should be evidence), but I
> > believe I have managed to track this down:
> > https://review.openstack.org/#/c/275483/
> >
> > We also have only ever been clearing our internal cache when reloading.
> We
> > should perhaps look at doing that periodically but that is a different
> > issue to the newly introduced leak. Although I haven't checked, I suspect
> > our cache is a lot larger recently (looking at cacti and other metrics)
> > probably due to cross-repo-dependencies so there is more need to keep it
> > clear. This may have contributed to the recent larger memory use.
>
> Ah, that explanation makes sense, thanks!
>
> On the subject of clearing the cache more often, I think we may not want
> to wipe out the cache more often than we do now -- in fact, I think we
> may want to look into ways to keep from doing even that, because
> whenever we reload now, Zuul slows down considerably as it has to query
> Gerrit again for all of the data previously in its cache.
>


I can see a lot of 3rd parties or simpler CI's not needing to reload zuul
very often so this cache would never get cleared. Perhaps cached objects
should have an expiry time (of a day or so) and can be cleaned up
periodically? Additionally if clearing the cache on a reload is causing
pain maybe we should move the cache into the scheduler and keep it between
reloads?

Cheers,
Josh



>
> But we certainly don't want multiple copies of the entire cache.  Thanks
> again!
>
> -Jim
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-infra/attachments/20160209/04d89047/attachment.html>