Stable gate status?
Hi Sean and Anita,
what is your opinion about the state of stable/havana gate?
If you think it's not stable enough to return back patches which were pulled out, I'd have to announce the delay of 2013.2.2 stable release until gate is fixed.
Thanks, Alan
2014-02-02 Alan Pevec apevec@gmail.com:
Hi Anita,
regarding few neutron stable/havana patches you pulled out of gate e.g.
New patchset patch-id matches previous patchset, but commit message has changed. To view, visit https://review.openstack.org/70232
...
Removed from the gate due to failing tests. Please do not reverify or reapprove until neutron gate tests are stable.
I'm not sure this is specific to neutron gate tests, there are few know (waiting for...to be become ACTIVE and SSH timeouts) since long time. I was trying to get last few patches in 2013.2.2 stable/havana release on Thursday Feb 06. Which specific gate issues are you fixing and what is ETA?
Cheers, Alan
On Feb 3, 2014, at 8:27 PM, Alan Pevec apevec@gmail.com wrote:
Hi Sean and Anita,
what is your opinion about the state of stable/havana gate?
If you think it's not stable enough to return back patches which were pulled out, I'd have to announce the delay of 2013.2.2 stable release until gate is fixed.
Thanks, Alan
2014-02-02 Alan Pevec apevec@gmail.com:
Hi Anita,
regarding few neutron stable/havana patches you pulled out of gate e.g.
New patchset patch-id matches previous patchset, but commit message has changed. To view, visit https://review.openstack.org/70232
...
Removed from the gate due to failing tests. Please do not reverify or reapprove until neutron gate tests are stable.
I'm not sure this is specific to neutron gate tests, there are few know (waiting for...to be become ACTIVE and SSH timeouts) since long time. I was trying to get last few patches in 2013.2.2 stable/havana release on Thursday Feb 06. Which specific gate issues are you fixing and what is ETA?
Cheers, Alan
We’re currently working around kernel problems in the master gate and we’ve seen the same bug in the stable checks too [1]. Many the current patches up for stable merges have multiple rechecks which is very concerning to me. It is nearly impossible to tell if these patches make things better or worse and approving will likely cause many gate resets annoying the entire community. We’re fairly close to having an appropriate workaround, so my vote would be to delay at least a week.
mark
[1] http://logs.openstack.org/10/65810/1/check/check-tempest-dsvm-neutron-pg/210...
2014-02-04 Mark McClain mmcclain@yahoo-inc.com:
On Feb 3, 2014, at 8:27 PM, Alan Pevec apevec@gmail.com wrote:
Hi Sean and Anita,
what is your opinion about the state of stable/havana gate?
If you think it's not stable enough to return back patches which were pulled out, I'd have to announce the delay of 2013.2.2 stable release until gate is fixed.
Thanks, Alan
2014-02-02 Alan Pevec apevec@gmail.com:
Hi Anita,
regarding few neutron stable/havana patches you pulled out of gate e.g.
New patchset patch-id matches previous patchset, but commit message has changed. To view, visit https://review.openstack.org/70232
...
Removed from the gate due to failing tests. Please do not reverify or reapprove until neutron gate tests are stable.
I'm not sure this is specific to neutron gate tests, there are few know (waiting for...to be become ACTIVE and SSH timeouts) since long time. I was trying to get last few patches in 2013.2.2 stable/havana release on Thursday Feb 06. Which specific gate issues are you fixing and what is ETA?
Cheers, Alan
We're currently working around kernel problems in the master gate and we've seen the same bug in the stable checks too [1]. Many the current patches up for stable merges have multiple rechecks which is very concerning to me. It is nearly impossible to tell if these patches make things better or worse and approving will likely cause many gate resets annoying the entire community. We're fairly close to having an appropriate workaround, so my vote would be to delay at least a week.
mark
[1] http://logs.openstack.org/10/65810/1/check/check-tempest-dsvm-neutron-pg/210...
ok, since I'm handling 2013.2.2 I'm making the call: release 2013.2.2 is now planned for Thursday Feb 13th. Freeze stays in effect, no new proposed changes should be approved, only those few which where pulled out.
Mark, Sean, do you have a list of issues which must be fixed before we can declare gate back to normal?
Cheers, Alan
On Tue, Feb 4, 2014 at 9:45 AM, Alan Pevec apevec@gmail.com wrote:
2014-02-04 Mark McClain mmcclain@yahoo-inc.com:
On Feb 3, 2014, at 8:27 PM, Alan Pevec apevec@gmail.com wrote:
Hi Sean and Anita,
what is your opinion about the state of stable/havana gate?
If you think it's not stable enough to return back patches which were pulled out, I'd have to announce the delay of 2013.2.2 stable release until gate is fixed.
Thanks, Alan
2014-02-02 Alan Pevec apevec@gmail.com:
Hi Anita,
regarding few neutron stable/havana patches you pulled out of gate e.g.
New patchset patch-id matches previous patchset, but commit message
has changed.
To view, visit https://review.openstack.org/70232
...
Removed from the gate due to failing tests. Please do not reverify or reapprove until neutron gate tests are stable.
I'm not sure this is specific to neutron gate tests, there are few know (waiting for...to be become ACTIVE and SSH timeouts) since long time. I was trying to get last few patches in 2013.2.2 stable/havana release on Thursday Feb 06. Which specific gate issues are you fixing and what is ETA?
Cheers, Alan
We're currently working around kernel problems in the master gate and
we've seen the same bug in the stable checks too [1]. Many the current patches up for stable merges have multiple rechecks which is very concerning to me. It is nearly impossible to tell if these patches make things better or worse and approving will likely cause many gate resets annoying the entire community. We're fairly close to having an appropriate workaround, so my vote would be to delay at least a week.
mark
[1]
http://logs.openstack.org/10/65810/1/check/check-tempest-dsvm-neutron-pg/210...
ok, since I'm handling 2013.2.2 I'm making the call: release 2013.2.2 is now planned for Thursday Feb 13th. Freeze stays in effect, no new proposed changes should be approved, only those few which where pulled out.
Do we have a list of those somewhere? I'm particularly interested in https://review.openstack.org/#/c/66149/ as a fix for https://bugs.launchpad.net/keystone/+bug/1251123
Doug
Mark, Sean, do you have a list of issues which must be fixed before we can declare gate back to normal?
Cheers, Alan
Openstack-stable-maint mailing list Openstack-stable-maint@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-stable-maint
2014-02-04 Doug Hellmann doug.hellmann@dreamhost.com:
Do we have a list of those somewhere?
Pulled out where following Neutron patches (IMHO all innocent for gate breaking): https://review.openstack.org/62206 https://review.openstack.org/67214 https://review.openstack.org/70232
I'm particularly interested in https://review.openstack.org/#/c/66149/ as a fix for https://bugs.launchpad.net/keystone/+bug/1251123
We can discuss it as an exception, I've opened "2013.2.2 exception requests" thread on stable-maint last week. I was only +1 on that patch because it's stable/havana only and I didn't see reports from anyone running Havana with this fix, only Kieran reported running similar patch on _Grizzly_. There's a minor inline comment from Kieran, but that's not a blocker afaict. Also reviews from Keystone Core members would help.
Cheers, Alan
On Tue, Feb 4, 2014 at 11:52 AM, Alan Pevec apevec@gmail.com wrote:
2014-02-04 Doug Hellmann doug.hellmann@dreamhost.com:
Do we have a list of those somewhere?
Pulled out where following Neutron patches (IMHO all innocent for gate breaking): https://review.openstack.org/62206 https://review.openstack.org/67214 https://review.openstack.org/70232
I'm particularly interested in https://review.openstack.org/#/c/66149/as a fix for https://bugs.launchpad.net/keystone/+bug/1251123
We can discuss it as an exception, I've opened "2013.2.2 exception requests" thread on stable-maint last week. I was only +1 on that patch because it's stable/havana only and I didn't see reports from anyone running Havana with this fix, only Kieran reported running similar patch on _Grizzly_. There's a minor inline comment from Kieran, but that's not a blocker afaict. Also reviews from Keystone Core members would help.
We just discovered the issue this week at DreamHost, and the patch seems to help for havana.
I noticed the inline comment, but as it's a backport I didn't think we wanted to make implementation changes?
Doug
Cheers, Alan
Hi Mark and Anita,
could we declare stable/havana neutron gate jobs good enough at this point? There are still random failures as this no-op change shows https://review.openstack.org/72576 but I don't think they're stable/havana specific.
Do we have a list of those somewhere?
Pulled out where following Neutron patches (IMHO all innocent for gate breaking): https://review.openstack.org/62206 https://review.openstack.org/67214 https://review.openstack.org/70232
I've resubmitted those without "Removed from the gate..." lines in the commit message, waiting for rechecks now.
I'm particularly interested in https://review.openstack.org/#/c/66149/ as a fix for https://bugs.launchpad.net/keystone/+bug/1251123
This one is last remaining exception request for 2013.2.2 and is waiting for the master change to be reviewed: https://review.openstack.org/#/q/Ida39b4699ed6c568609a5121573fc3be5c4ab2f4,n... I hope keystone core could review this one quickly so that backport can be updated and merged.
Thanks! Alan
On 02/11/2014 04:57 AM, Alan Pevec wrote:
Hi Mark and Anita,
could we declare stable/havana neutron gate jobs good enough at this point? There are still random failures as this no-op change shows https://review.openstack.org/72576 but I don't think they're stable/havana specific.
Do we have a list of those somewhere?
Pulled out where following Neutron patches (IMHO all innocent for gate breaking): https://review.openstack.org/62206 https://review.openstack.org/67214 https://review.openstack.org/70232
I've resubmitted those without "Removed from the gate..." lines in the commit message, waiting for rechecks now.
I'm particularly interested in https://review.openstack.org/#/c/66149/ as a fix for https://bugs.launchpad.net/keystone/+bug/1251123
This one is last remaining exception request for 2013.2.2 and is waiting for the master change to be reviewed: https://review.openstack.org/#/q/Ida39b4699ed6c568609a5121573fc3be5c4ab2f4,n... I hope keystone core could review this one quickly so that backport can be updated and merged.
Thanks! Alan
I will reaffirm here what I had stated in IRC.
If Mark McClain gives his assent for stable/havana patches to be approved, I will not remove Neutron stable/havana patches from the gate queue before they start running tests. If after they start running tests, they demonstrate that they are failing, I will remove them from the gate as a means to keep the gate flowing. If the stable/havana gate jobs are indeed stable, I will not be removing any patches that should be merged.
Adding commit lines is the fastest way to submit a new patchset without affecting the code in the patch (hence removing it from the gate queue), so thank you for removing those additional lines in the commit message.
Thank you, Anita.
2014-02-11 16:14 GMT+01:00 Anita Kuno:
On 02/11/2014 04:57 AM, Alan Pevec wrote:
Hi Mark and Anita,
could we declare stable/havana neutron gate jobs good enough at this point? There are still random failures as this no-op change shows https://review.openstack.org/72576 but I don't think they're stable/havana specific.
...
I will reaffirm here what I had stated in IRC.
If Mark McClain gives his assent for stable/havana patches to be approved, I will not remove Neutron stable/havana patches from the gate queue before they start running tests. If after they start running tests, they demonstrate that they are failing, I will remove them from the gate as a means to keep the gate flowing. If the stable/havana gate jobs are indeed stable, I will not be removing any patches that should be merged.
As discussed on #openstack-infra last week, stable-maint team should start looking more closely at Tempest stable/havana branch and Matthew Treinish from Tempest core joined the stable-maint team to help us there.
In the meantime, we need to do something more urgently, there are remaining failures showing up frequently in stable/havana jobs which seem to have been fixed or at least improved on master:
* bug 1254890 - "Timed out waiting for thing ... to become ACTIVE" causes tempest-dsvm-* failures resolution unclear?
* bug 1253896 - "Attempts to verify guests are running via SSH fails. SSH connection to guest does not work." based on Salvatore's comment 56, I've marked it as Won't Fix in neutron/havana and opened tempest/havana to propose what Tempest test or jobs should skip for Havana. Please chime-in in the bug if you have suggestions.
Cheers, Alan
I rebased the https://review.openstack.org/#/c/72576/ no-op change.
----- Original Message -----
From: "Alan Pevec" apevec@gmail.com To: "openstack-stable-maint" openstack-stable-maint@lists.openstack.org Cc: "OpenStack Development Mailing List" openstack-dev@lists.openstack.org Sent: Tuesday, February 18, 2014 7:52:23 PM Subject: Re: [openstack-dev] [Openstack-stable-maint] Stable gate status?
2014-02-11 16:14 GMT+01:00 Anita Kuno:
On 02/11/2014 04:57 AM, Alan Pevec wrote:
Hi Mark and Anita,
could we declare stable/havana neutron gate jobs good enough at this point? There are still random failures as this no-op change shows https://review.openstack.org/72576 but I don't think they're stable/havana specific.
...
I will reaffirm here what I had stated in IRC.
If Mark McClain gives his assent for stable/havana patches to be approved, I will not remove Neutron stable/havana patches from the gate queue before they start running tests. If after they start running tests, they demonstrate that they are failing, I will remove them from the gate as a means to keep the gate flowing. If the stable/havana gate jobs are indeed stable, I will not be removing any patches that should be merged.
As discussed on #openstack-infra last week, stable-maint team should start looking more closely at Tempest stable/havana branch and Matthew Treinish from Tempest core joined the stable-maint team to help us there.
In the meantime, we need to do something more urgently, there are remaining failures showing up frequently in stable/havana jobs which seem to have been fixed or at least improved on master:
- bug 1254890 - "Timed out waiting for thing ... to become ACTIVE"
causes tempest-dsvm-* failures resolution unclear?
- bug 1253896 - "Attempts to verify guests are running via SSH fails.
SSH connection to guest does not work." based on Salvatore's comment 56, I've marked it as Won't Fix in neutron/havana and opened tempest/havana to propose what Tempest test or jobs should skip for Havana. Please chime-in in the bug if you have suggestions.
Cheers, Alan
OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
2014-02-20 8:57 GMT+01:00 Miguel Angel Ajo Pelayo mangelajo@redhat.com:
I rebased the https://review.openstack.org/#/c/72576/ no-op change.
And it failed in check-tempest-dsvm-neutron-pg with bug 1254890 - "Timed out waiting for thing ... to become ACTIVE" while previous check on Feb 17 failed in check-tempest-dsvm-neutron-isolated with bug 1253896 - "Attempts to verify guests are running via SSH fails. SSH connection to guest does not work."
Cheers, Alan
participants (5)
-
Alan Pevec
-
Anita Kuno
-
Doug Hellmann
-
Mark McClain
-
Miguel Angel Ajo Pelayo