Open Stack

On Mon, Sep 18, 2017, at 04:58 PM, Clark Boylan wrote:
> On Mon, Sep 18, 2017, at 06:43 AM, Andreas Jaeger wrote:
> > Just a friendly reminder that the upgrade will happen TODAY, Monday
> > 18th, starting at 15:00 UTC. The infra team expects that it takes 8
> > hours, so until 2359 UTC.
> 
> This work was functionally completed at 23:43 UTC. We are now running
> Gerrit 2.13.9. There are some cleanup steps that need to be performed in
> Infra land, mostly to get puppet running properly again.
> 
> You will also notice that newer Gerrit behaves in some new and exciting
> ways. Most of these should be improvements like not needing to reapprove
> changes that already have a +1 Workflow but also have a +1 Verified;
> recheck should now work for these cases. If you find a new behavior that
> looks like a bug please let us know, but we should also work to file
> them upstream so that newer Gerrit can address them.
> 
> Feel free to ask us questions if anything else comes up.
> 
> Thank you to everyone that helped with the upgrade. Seems like these get
> more and more difficult with each Gerrit release so all the help is
> greatly appreciated.

As a followup we have been tracking new fun issues/behaviors in Gerrit
and fixing them over the last couple days. Here is an update on where we
are currently at.

Gerrit emails are slow. You may have noticed that you aren't getting
quite as much Gerrit email as before. This is because Gerrit is only
sending about one email a minute. Upstream bug is at
https://bugs.chromium.org/p/gerrit/issues/detail?id=7261 and we have
just got https://review.openstack.org/#/c/505677 merged based on the
info in that upstream bug. This won't be applied until we get puppet
running on review.openstack.org again (more on that later) and will
require another Gerrit service restart.

The Gerrit web UI's file editor behaves oddly resulting in what appear
to be API timeouts. This also seems to affect gertty. I don't think
anyone has dug in far enough to understand what is going on yet.

Now for known issues that should be fixed.

The Gerrit dashboard creator was using queries that didn't work with new
Gerrit query behavior. Sdague got this sorted out quick.

The Gerrit event stream changed its ref-updated data and now includes
refs/heads/$branchname instead of just $branchname under refName when
changes merge. This confused Zuul and meant no post jobs were running.
Zuul has been updated to handle this new behavior and post jobs are
running.

There were no gitweb links. This wasn't caught in testing because we
used a test cgit setup on review-dev. Fix here was just to switch to
using cgit on review.openstack.org (though the link is still called
"gitweb" in the Gerrit UI for reasons).

Memory consumption has gone up which initially led to frequent garbage
collection which led to 500 errors. We bumped heap memory available to
Gerrit up to 48GB (from 30GB) and that seems to have stabilized things.
Thankfully while needing more memory it doesn't seem to continuously
grow like it did on the old version (which forced us to do semi frequent
service restarts). We will have to monitor Gerrit to ensure it is
properly stable over time.

We could not create new projects in Gerrit. This is because Gerrit 2.12
dropped the --name argument from the create-project command which
Gerritlib was using. We have updated Gerritlib to check the Gerrit
version and pass the correct arguments to create-project.

Unfortunately, we still can't create new projects just yet, this is
related to puppet not running on review.openstack.org right now. The
gerrit server itself is fine and would puppet except that we force
puppet to run on our git mirror farm first to ensure proper mirroring of
repos and those have been failing since the CentOS 7.4 release. Once
we've got puppet happy we can get back to creating new projects in
Gerrit.

All the details can be found at
https://etherpad.openstack.org/p/gerrit-2.13-issues.

Thank you for your patience,
Clark