Open Stack

Sean Dague <sean at dague.net> writes:

> So as the review queues grow, and our gerrit becomes longer and longer
> in the tooth, we are now getting to the point where people are spending
> real time building local work arrounds for features lacking in gerrit
> 2.4 that have since been implemented upstream.
>
> For instance, Gerrit 2.5 added the ability to put UnifiedDiffs in email
> templates, which would let you know if the review in question touched
> files that you like to keep an eye on (and trigger local filtering based
> on that).
>
> Gerrit 2.8 implements a secondary index mechanism that means we could
> get file regex in the web ui for reviews.
>
> We also apparently got custom dashboards somewhere along the way.
>
> What's actually currently blocking the upgrade? Is it just time? or are
> we still waiting on feature(s)? If so, do those feature(s) outweigh what
> we are missing by being so far behind master?

We don't need to be convinced to upgrade, we decided it quite a while
ago and several people have been working on it for months.  First, we
use features in Gerrit 2.4 that were never accepted in upstream gerrit.
Notably the "Important Changes" dashboard and "Work In Progress".  We
had to figure out how to deal with that, and after some analysis, came
to the conclusion that we could get functionality similar to "Important
Changes" by using custom dashboards.  However, nothing similar to "Work
In Progress" is provided in upstream Gerrit.

Because the initial attempt to upstream that patch failed, we embarked
on a new approach that was much simpler (it involves a small change to
the ACL system to allow us to achieve something similar to WIP with a
review vote).  Despite this radically simplified approach, working with
the upstream Gerrit community on this patch still turned out to be
extremely time consuming.  Khai Do has exhibited an amazing amount of
patience in shepherding it in.  David Ostrovsky has also started working
on a Gerrit plugin that implements WIP, which we may be able to use in
the future.

It's worth mentioning at this point that we have adopted an
upstream-first policy with Gerrit -- after our previous experience, we
want to avoid backing ourselves into a corner where we have to run a
fork and can not upgrade.  Our previous policy of running patches
locally first has delayed this upgrade, but by changing that policy, we
will be able to upgrade faster in the future.

At this point, the necessary patches have landed in upstream Gerrit, and
Khai has begun testing Gerrit 2.8 on review-dev.  Just recently (over
the holiday), Khai and others managed to get it running on review-dev.
That means we've learned quite a bit about file and configuration
changes that need to happen during the upgrade, and particularly about
database schema changes, not all of which apply cleanly, and some of
which will require manual changes with limited documentation.  It seems
the CLA mechanism in particular has changed and will require some work.

We're currently at the phase where having performed such a test upgrade
allows us to finally identify the problems we can expect during the
upgrade.  Soon we should have them identified, and if any require
substantial work to fix, we should be able to estimate how long they
will take.  After that, we still need to perform user testing,
integration testing with our tools (e.g. Zuul), and likely apply some
amount of polish.

There is a blueprint which mostly exists to link to this etherpad where
a number of production issues are identified and configuration decisions
are being made:

  https://blueprints.launchpad.net/openstack-ci/+spec/gerrit-2.8-upgrade

  https://etherpad.openstack.org/p/gerrit-2.8-upgrade

This is a big change that affects a lot of users, and I would like us to
make it very deliberately and carefully to reduce the impact to the
project.  We're all extremely excited about upgrading and being able to
use the new features, and I thank everyone for your patience and
assistance while we work to complete it.

-Jim