Open Stack

hmm, seems like we have hit the issue again, however in a different job now:
Latest logs:
https://zuul.opendev.org/t/openstack/build/0565c3d252194f9ba67f4af20e8be65d
Link to the review where it occurred:
https://review.opendev.org/c/osf/refstack-client/+/788743


On Tue, 6 Apr 2021 at 18:47, Clark Boylan <cboylan at sapwetik.org> wrote:

> On Tue, Apr 6, 2021, at 9:11 AM, Radosław Piliszek wrote:
> > On Tue, Apr 6, 2021 at 6:02 PM Jeremy Stanley <fungi at yuggoth.org> wrote:
> > > Looking at the error, I strongly suspect memory exhaustion. We could
> > > try tuning xz to use less memory when compressing.
>
> Worth noting that we continue to suspect memory pressure, and in
> particular diving into swap, for random failures that appear timing or
> performance related. I still think it would be a helpful exercise for
> OpenStack to look at its memory consumption (remember end users will
> experience this too) and see if there are any unexpected areas of memory
> use. I think the last time i skimmed logs the privsep daemon was a large
> consumer because we separate instance is run for each service and they all
> add up.
>
> >
> > That was my hunch as well, hence why I test using gzip.
> >
> > On Tue, Apr 6, 2021 at 5:51 PM Clark Boylan <cboylan at sapwetik.org>
> wrote:
> > >
> > > On Tue, Apr 6, 2021, at 8:14 AM, Radosław Piliszek wrote:
> > > > I am testing whether replacing xz with gzip would solve the problem
> [1] [2].
> > >
> > > The reason we used xz is that the files are very large and gz
> compression is very poor compared to xz for these files and these files are
> not really human readable as is (you need to load them into journald
> first). Let's test it and see what the gz file sizes look like but if they
> are still quite large then this is unlikely to be an appropriate fix.
> >
> > Let's see how bad the file sizes are.
> > If they are acceptable, we can keep gzip and be happy.
> > Otherwise we try to tune the params to make xz a better citizen as
> > fungi suggested.
> >
> > -yoctozepto
> >
> >
>
>

-- 
Martin
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-discuss/attachments/20210507/66251edd/attachment.html>