On 3/3/20 5:42 PM, Clark Boylan wrote:
On Tue, Mar 3, 2020, at 3:05 PM, Ben Nemec wrote:
Found a weird thing today. The OpenDev oslo.policy repo[0] is missing [1]. Even stranger, I see it on the Github mirror[2]. Any idea what happened here?
Some other readers may notice that the commit actually does show up for them. The reason for this is the commit is only missing from one of eight backend gitea servers. You can observe this by visiting https://gitea0X.opendev.org:3000/openstack/oslo.policy/commits/branch/master and replacing the X with 1 through 8. Number 5 is the lucky server.
My hunch is that this commit merging and subsequently being replicated coincided with a restart of gitea (or related service) on gitea05. And the replication event was missed. We've tried to ensure we replicate to catch up after explicit upgrades, which implies to me that maybe the db container updated. Note that https://review.opendev.org/#/c/705804/ merged on the same day but after the missing commit.
In any case I've triggered a full rereplication to gitea05 to make sure we are caught up and will work through the others as well to ensure none are missed. You should be able to confirm that the commit is present in about 20 minutes.
Great, thanks! I see it now.
Longer term the plan here is to run a single Gitea cluster which will allow us to do rolling restarts of services without impacting replication. Unfortunately, this requires updates to Gitea to support that.
-Ben
0: https://opendev.org/openstack/oslo.policy/commits/branch/master 1: https://review.opendev.org/#/c/708212/ 2: https://github.com/openstack/oslo.policy/commits/master