review.opendev.org Downtime March 20, 2020 at 1400UTC
Hello all, The OpenDev team is planning to take a short downtime March 20, 2020 from 14:00UTC to 15:00UTC. This outage will enable us to rename a few projects and redeploy Gerrit using containers. The container based deployment is an important first step in our longer term plan for upgrading Gerrit to a modern version. We recognize that this will be a disruption during an already disrupted week for many. We feel this is worthwhile as it fits into project release schedules and sets us up well for the future. Thank you for your patience. Feel free to ask us any questions you may have on this thread or in #opendev on Freenode. Clark
A friendly reminder: In a few minutes review.opendev.org will be completely down as explained below, Andreas On 3/17/20 10:08 PM, Clark Boylan wrote:
Hello all,
The OpenDev team is planning to take a short downtime March 20, 2020 from 14:00UTC to 15:00UTC. This outage will enable us to rename a few projects and redeploy Gerrit using containers. The container based deployment is an important first step in our longer term plan for upgrading Gerrit to a modern version.
We recognize that this will be a disruption during an already disrupted week for many. We feel this is worthwhile as it fits into project release schedules and sets us up well for the future. Thank you for your patience.
Feel free to ask us any questions you may have on this thread or in #opendev on Freenode.
Clark
-- Andreas Jaeger aj@suse.com Twitter: jaegerandi SUSE Software Solutions Germany GmbH, Maxfeldstr. 5, D 90409 Nürnberg (HRB 36809, AG Nürnberg) GF: Felix Imendörffer GPG fingerprint = EF18 1673 38C4 A372 86B1 E699 5294 24A3 FF91 2ACB
On Fri, 2020-03-20 at 14:43 +0100, Andreas Jaeger wrote:
A friendly reminder: In a few minutes review.opendev.org will be completely down as explained below,
on a related note i noticed i was getting different contenent from https://review.openstack.org/p/openstack/nova and ssh://sean-k-mooney@review.openstack.org:29418/openstack/nova.git last week. these were my remotes in my nove repo gerrit ssh://sean-k-mooney@review.openstack.org:29418/openstack/nova.git (fetch) gerrit ssh://sean-k-mooney@review.openstack.org:29418/openstack/nova.git (push) origin https://review.openstack.org/p/openstack/nova (fetch) origin https://review.openstack.org/p/openstack/nova (push) the gerrit remote when i did a fetch was up to date with https://opendev.org/openstack/nova and with https://github.com/openstack/nova but https://review.openstack.org/p/openstack/nova was behind both of them by a few hours maybe a day i dont rememeber but i just remember tinking it was odd. is this somethign ye were aware of that could happen? it was if the redirect from review.openstack.org was pointing an out of sync gerrit backend. i know i should use the updated urls but this is an old copy of nova form before the switch so just said i would mention this incase anyone else hits this issue.
Andreas
On 3/17/20 10:08 PM, Clark Boylan wrote:
Hello all,
The OpenDev team is planning to take a short downtime March 20, 2020 from 14:00UTC to 15:00UTC. This outage will enable us to rename a few projects and redeploy Gerrit using containers. The container based deployment is an important first step in our longer term plan for upgrading Gerrit to a modern version.
We recognize that this will be a disruption during an already disrupted week for many. We feel this is worthwhile as it fits into project release schedules and sets us up well for the future. Thank you for your patience.
Feel free to ask us any questions you may have on this thread or in #opendev on Freenode.
Clark
On 2020-03-20 14:14:57 +0000 (+0000), Sean Mooney wrote:
on a related note i noticed i was getting different contenent from https://review.openstack.org/p/openstack/nova and ssh://sean-k-mooney@review.openstack.org:29418/openstack/nova.git last week.
these were my remotes in my nove repo gerrit ssh://sean-k-mooney@review.openstack.org:29418/openstack/nova.git (fetch) gerrit ssh://sean-k-mooney@review.openstack.org:29418/openstack/nova.git (push) origin https://review.openstack.org/p/openstack/nova (fetch) origin https://review.openstack.org/p/openstack/nova (push)
I definitely don't recommend cloning from review.o.o, better to rely on https://opendev.org/openstack/nova in your origin remotes.
the gerrit remote when i did a fetch was up to date with https://opendev.org/openstack/nova and with https://github.com/openstack/nova but https://review.openstack.org/p/openstack/nova was behind both of them by a few hours maybe a day i dont rememeber but i just remember tinking it was odd.
That https://review.openstack.org/p/openstack/nova URL is "just another replica" like opendev.org and github.com (the /p is directed to a local copy Gerrit is replicating to a local copy on the server's filesystem).
is this somethign ye were aware of that could happen? it was if the redirect from review.openstack.org was pointing an out of sync gerrit backend. [...]
Gerrit can't guarantee consistency for its replication tasks, so it can certainly happen. I think we had plans to remove that /p replica anyway (it's going to potentially conflict with some URLs for newer Gerrit versions), but generally if you notice a discrepancy between ssh://sean-k-mooney@review.openstack.org:29418/openstack/nova.git and https://opendev.org/openstack/nova do let us know, we've been trying to get on top of a race condition in our Gitea updates where Gerrit will silently fail to replicate refs to some of the servers while they're in the middle of restarting. -- Jeremy Stanley
On 2020-03-20 14:14:57 +0000 (+0000), Sean Mooney wrote:
on a related note i noticed i was getting different contenent from https://review.openstack.org/p/openstack/nova and ssh://sean-k-mooney@review.openstack.org:29418/openstack/nova.git last week.
these were my remotes in my nove repo gerrit ssh://sean-k-mooney@review.openstack.org:29418/openstack/nova.git (fetch) gerrit ssh://sean-k-mooney@review.openstack.org:29418/openstack/nova.git (push) origin https://review.openstack.org/p/openstack/nova (fetch) origin https://review.openstack.org/p/openstack/nova (push)
I definitely don't recommend cloning from review.o.o, better to rely on https://opendev.org/openstack/nova in your origin remotes. ya i normally set it up against either github or https://opendev.org/openstack/nova
On Fri, 2020-03-20 at 14:28 +0000, Jeremy Stanley wrote: the review.o.o is much larger and take up way more space on disk so i normally avoid it but i think i origianly create this repo usign gertty for reviews. this was on my local laptop and i normally only use that repo for doing quick fixes.
the gerrit remote when i did a fetch was up to date with https://opendev.org/openstack/nova and with https://github.com/openstack/nova but https://review.openstack.org/p/openstack/nova was behind both of them by a few hours maybe a day i dont rememeber but i just remember tinking it was odd.
That https://review.openstack.org/p/openstack/nova URL is "just another replica" like opendev.org and github.com (the /p is directed to a local copy Gerrit is replicating to a local copy on the server's filesystem).
is this somethign ye were aware of that could happen? it was if the redirect from review.openstack.org was pointing an out of sync gerrit backend.
[...]
Gerrit can't guarantee consistency for its replication tasks, so it can certainly happen. I think we had plans to remove that /p replica anyway (it's going to potentially conflict with some URLs for newer Gerrit versions), but generally if you notice a discrepancy between ssh://sean-k-mooney@review.openstack.org:29418/openstack/nova.git and https://opendev.org/openstack/nova do let us know, we've been trying to get on top of a race condition in our Gitea updates where Gerrit will silently fail to replicate refs to some of the servers while they're in the middle of restarting.
ya if i notice an issue between those too ill let you know but in generall they seam to mostly be in sync.
On Fri, Mar 20, 2020, at 7:48 AM, Sean Mooney wrote:
On 2020-03-20 14:14:57 +0000 (+0000), Sean Mooney wrote:
on a related note i noticed i was getting different contenent from https://review.openstack.org/p/openstack/nova and ssh://sean-k-mooney@review.openstack.org:29418/openstack/nova.git last week.
these were my remotes in my nove repo gerrit ssh://sean-k-mooney@review.openstack.org:29418/openstack/nova.git (fetch) gerrit ssh://sean-k-mooney@review.openstack.org:29418/openstack/nova.git (push) origin https://review.openstack.org/p/openstack/nova (fetch) origin https://review.openstack.org/p/openstack/nova (push)
I definitely don't recommend cloning from review.o.o, better to rely on https://opendev.org/openstack/nova in your origin remotes. ya i normally set it up against either github or https://opendev.org/openstack/nova
On Fri, 2020-03-20 at 14:28 +0000, Jeremy Stanley wrote: the review.o.o is much larger and take up way more space on disk so i normally avoid it but i think i origianly create this repo usign gertty for reviews.
this was on my local laptop and i normally only use that repo for doing quick fixes.
the gerrit remote when i did a fetch was up to date with https://opendev.org/openstack/nova and with https://github.com/openstack/nova but https://review.openstack.org/p/openstack/nova was behind both of them by a few hours maybe a day i dont rememeber but i just remember tinking it was odd.
That https://review.openstack.org/p/openstack/nova URL is "just another replica" like opendev.org and github.com (the /p is directed to a local copy Gerrit is replicating to a local copy on the server's filesystem).
is this somethign ye were aware of that could happen? it was if the redirect from review.openstack.org was pointing an out of sync gerrit backend.
[...]
Gerrit can't guarantee consistency for its replication tasks, so it can certainly happen. I think we had plans to remove that /p replica anyway (it's going to potentially conflict with some URLs for newer Gerrit versions), but generally if you notice a discrepancy between ssh://sean-k-mooney@review.openstack.org:29418/openstack/nova.git and https://opendev.org/openstack/nova do let us know, we've been trying to get on top of a race condition in our Gitea updates where Gerrit will silently fail to replicate refs to some of the servers while they're in the middle of restarting.
ya if i notice an issue between those too ill let you know but in generall they seam to mostly be in sync.
To followup on this we think we addressed the major issue with https://review.opendev.org/#/c/711130/. We think the important bit is ensuring that Gerrit can detect that replication is going to fail by turning off the ssh daemon before other Gitea services and starting it last. Gerrit will happily retry replication when it detects that it is in a failure state. Unfortunately, the old restart logic had things stopping together and the ssh side would happily accept data if the actual services was done and that led to problems. Again, do let us know if you notice this in the future, but we're hopeful this recent change addresses the problem. Clark
participants (4)
-
Andreas Jaeger
-
Clark Boylan
-
Jeremy Stanley
-
Sean Mooney