Preparing for 2014.1.1 and design summit session summary
Hi all,
at "Stable branches maintenance" design summit session[1] dates for stable/icehouse release were confirmed, I've put them in our wiki[2] First release is coming fast and starting today we are in freeze period, but given the current patch queue[3] I propose we start with soft-freeze until end of day Friday. Soft-freeze is to allow patches currently under review to get merged but no new patches should be accepted. As always, feel free to raise exception requests on stable-maint list.
Other points which were discussed: * scripted freeze enforcement - infra team said this could be automated now using Gerrit 2.8 REST API, details TBD * no changes to the patch acceptance policy[4] but exception requests can be raised any time on stable-maint list and will be decided on case-by-case basis + database migrations might be allowed as an exception (it was done recently for a serious Keystone performance issue[5]) in projects using sqlalchemy-migrate and reserving[6] a range of migrations for backports. Mark McClain will document how to do it with Alembic. * starting with Icehouse support period is extended to 15 months + made possible with branchless Tempest + we will establish "stablesec" a subset of stable-maint which will work with VMT on backports of security patches. Anyone interested please contact me. * Chuck Short will be joining stable release managers
Cheers, Alan
[1] http://junodesignsummit.sched.org/event/a936d7718d1c4b1dd8dcb9dc6a671e9c https://etherpad.openstack.org/p/StableIcehouse [2] https://wiki.openstack.org/wiki/StableBranchRelease#Planned_stable.2Ficehous... [3] https://review.openstack.org/#/q/status:open+AND+branch:stable/icehouse+AND+... [4] https://wiki.openstack.org/wiki/StableBranch#Appropriate_Fixes [5] https://wiki.openstack.org/wiki/ReleaseNotes/2013.2.2#Keystone [6] https://review.openstack.org/70153
First release is coming fast and starting today we are in freeze period, but given the current patch queue[3] I propose we start with soft-freeze until end of day Friday.
Status as of Monday morning: I was rechecking patches with +2 which were hit by random SSH timeouts bugs 1323658 and 1253896. Not all managed to pass through the gate before the whole OpenStack CI got blocked on a regression in Sunday evening setuptools release(s): https://bitbucket.org/pypa/setuptools/issue/213/regression-setuptools-37-ins... I hope setuptools will be fixed today so we can stick to the schedule and release this Thursday, to avoid conflict with Juno-1 next week.
Soft-freeze is to allow patches currently under review to get merged but no new patches should be accepted. As always, feel free to raise exception requests on stable-maint list.
Does anyone has any exception proposals?
Cheers, Alan
First release is coming fast and starting today we are in freeze period, but given the current patch queue[3] I propose we start with soft-freeze until end of day Friday.
...
I hope setuptools will be fixed today so we can stick to the schedule and release this Thursday, to avoid conflict with Juno-1 next week.
Setuptools issue has been resolved but gate queues are still recovering, Zuul event queue length was over 500 this morning! Luckily, all but two Neutron patches (see below) have merged and I'll send call for testing email when regenerated stable-icehouse tarballs hit tarballs.openstack.org. I'd appreciate CI results from distros as soon as possible, please take stable/icehouse directly from git if you can.
Soft-freeze is to allow patches currently under review to get merged but no new patches should be accepted. As always, feel free to raise exception requests on stable-maint list.
https://review.openstack.org/96919 and https://review.openstack.org/96920 were in the check queue over > 18h and I've removed them from the gate to investigate the failures. Kyle, both were actually proposed after deadline, could you justify them as exception requests?
Cheers, Alan
On Jun 3, 2014, at 4:25 AM, Alan Pevec apevec@gmail.com wrote:
First release is coming fast and starting today we are in freeze period, but given the current patch queue[3] I propose we start with soft-freeze until end of day Friday.
...
I hope setuptools will be fixed today so we can stick to the schedule and release this Thursday, to avoid conflict with Juno-1 next week.
Setuptools issue has been resolved but gate queues are still recovering, Zuul event queue length was over 500 this morning! Luckily, all but two Neutron patches (see below) have merged and I'll send call for testing email when regenerated stable-icehouse tarballs hit tarballs.openstack.org. I'd appreciate CI results from distros as soon as possible, please take stable/icehouse directly from git if you can.
Soft-freeze is to allow patches currently under review to get merged but no new patches should be accepted. As always, feel free to raise exception requests on stable-maint list.
https://review.openstack.org/96919 and https://review.openstack.org/96920 were in the check queue over > 18h and I've removed them from the gate to investigate the failures. Kyle, both were actually proposed after deadline, could you justify them as exception requests?
These actually just landed in master last week. They close a long standing “High” priority bug reported by the TripleO folks. The failures are likely due to this issue [1], which may have been uncovered due to this tempest change [2]. That’s the theory we’re working on now, hopefully we’ll have this diagnosed a bit better soon.
So, if we can get these two bugs into the next stable/icehouse release, I know the TripleO folks would at least be happy. But if they slip into a subsequent release, maybe that’s ok as well.
Thanks, Kyle
[1] https://bugs.launchpad.net/neutron/+bug/1325737 [2] https://review.openstack.org/#/c/90427/
Cheers, Alan
2014-06-03 15:04 GMT+02:00 Kyle Mestery kmestery@cisco.com:
https://review.openstack.org/96919 https://review.openstack.org/96920
These actually just landed in master last week. They close a long standing “High” priority bug reported by the TripleO folks. The failures are likely due to this issue [1], which may have been uncovered due to this tempest change [2]. That’s the theory we’re working on now, hopefully we’ll have this diagnosed a bit better soon.
So, if we can get these two bugs into the next stable/icehouse release, I know the TripleO folks would at least be happy. But if they slip into a subsequent release, maybe that’s ok as well.
Could you expand on "including obvious security reasons" from the commit message in 96919 ? I've added "security" tag to https://bugs.launchpad.net/neutron/+bug/1290486 and if that's really the issue, exception is justified to include those patches into 2014.1.1.
Cheers, Alan
Alan Pevec wrote:
2014-06-03 15:04 GMT+02:00 Kyle Mestery kmestery@cisco.com:
https://review.openstack.org/96919 https://review.openstack.org/96920
These actually just landed in master last week. They close a long standing “High” priority bug reported by the TripleO folks. The failures are likely due to this issue [1], which may have been uncovered due to this tempest change [2]. That’s the theory we’re working on now, hopefully we’ll have this diagnosed a bit better soon.
So, if we can get these two bugs into the next stable/icehouse release, I know the TripleO folks would at least be happy. But if they slip into a subsequent release, maybe that’s ok as well.
Could you expand on "including obvious security reasons" from the commit message in 96919 ? I've added "security" tag to https://bugs.launchpad.net/neutron/+bug/1290486 and if that's really the issue, exception is justified to include those patches into 2014.1.1.
I suspect Kyle means that the failure triggered by this bug has security consequences. I think that would be a good addition to the 2014.1.1 stable release mix.
On Wed, Jun 4, 2014 at 10:24 AM, Thierry Carrez thierry@openstack.org wrote:
Alan Pevec wrote:
2014-06-03 15:04 GMT+02:00 Kyle Mestery kmestery@cisco.com:
https://review.openstack.org/96919 https://review.openstack.org/96920
These actually just landed in master last week. They close a long standing “High” priority bug reported by the TripleO folks. The failures are likely due to this issue [1], which may have been uncovered due to this tempest change [2]. That’s the theory we’re working on now, hopefully we’ll have this diagnosed a bit better soon.
So, if we can get these two bugs into the next stable/icehouse release, I know the TripleO folks would at least be happy. But if they slip into a subsequent release, maybe that’s ok as well.
Could you expand on "including obvious security reasons" from the commit message in 96919 ? I've added "security" tag to https://bugs.launchpad.net/neutron/+bug/1290486 and if that's really the issue, exception is justified to include those patches into 2014.1.1.
I suspect Kyle means that the failure triggered by this bug has security consequences. I think that would be a good addition to the 2014.1.1 stable release mix.
Yes, what Thierry says is exactly correct. The failure trigged by this leaves a possibly large security hole. Getting these into 2014.1.1 would be great if possible! Like I said, we're still working the gate failures around "ssh timeouts" which are occuring now, and those have affected these patches.
Thanks, Kyle
-- Thierry Carrez (ttx)
Openstack-stable-maint mailing list Openstack-stable-maint@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-stable-maint
2014-06-04 17:48 GMT+02:00 Kyle Mestery mestery@noironetworks.com:
Yes, what Thierry says is exactly correct. The failure trigged by this leaves a possibly large security hole. Getting these into 2014.1.1 would be great if possible! Like I said, we're still working the gate failures around "ssh timeouts" which are occuring now, and those have affected these patches.
ok, I've sent them to gate, hopefully we won't hit random timeouts this time!
Cheers, Alan
On Wed, Jun 4, 2014 at 10:58 AM, Alan Pevec apevec@gmail.com wrote:
2014-06-04 17:48 GMT+02:00 Kyle Mestery mestery@noironetworks.com:
Yes, what Thierry says is exactly correct. The failure trigged by this leaves a possibly large security hole. Getting these into 2014.1.1 would be great if possible! Like I said, we're still working the gate failures around "ssh timeouts" which are occuring now, and those have affected these patches.
ok, I've sent them to gate, hopefully we won't hit random timeouts this time!
Thanks Alan!
Kyle
Cheers, Alan
ok, I've sent them to gate, hopefully we won't hit random timeouts this time!
Unfortunately, 96919 and 96920 are still queued in the gate after more than 24h, see Sean's post[1] for details. Options are: * block 2014.1.1 until those two merged (ETA unknown but at least 8h more hours) * push 2014.1.1 tags for all but Neutron and release Neutron separately whenever those two are merged * promote them in the gate queue - discussing this now in #openstack-infra please join for more support and answer questions from infra team :)
Cheers, Alan
[1] http://lists.openstack.org/pipermail/openstack-dev/2014-June/036810.html
On Thu, Jun 5, 2014 at 12:40 PM, Alan Pevec apevec@gmail.com wrote:
ok, I've sent them to gate, hopefully we won't hit random timeouts this time!
Unfortunately, 96919 and 96920 are still queued in the gate after more than 24h, see Sean's post[1] for details. Options are:
- block 2014.1.1 until those two merged (ETA unknown but at least 8h more hours)
- push 2014.1.1 tags for all but Neutron and release Neutron
separately whenever those two are merged
- promote them in the gate queue - discussing this now in
#openstack-infra please join for more support and answer questions from infra team :)
Thanks Alan, I was out at lunch, back now and helping you in #openstack-infra as I type this.
Kyle
Cheers, Alan
[1] http://lists.openstack.org/pipermail/openstack-dev/2014-June/036810.html
- promote them in the gate queue - discussing this now in
#openstack-infra please join for more support and answer questions from infra team :)
Thanks Alan, I was out at lunch, back now and helping you in #openstack-infra as I type this.
Bad news, 96919 failed in gate-tempest-dsvm-neutron and gate-tempest-dsvm-neutron-pg 96920 had all jobs succeed but since it depends on 96919 it was canceled.
Cheers, Alan
Bad news, 96919 failed in gate-tempest-dsvm-neutron and gate-tempest-dsvm-neutron-pg 96920 had all jobs succeed but since it depends on 96919 it was canceled.
Prematue send... wanted to say, both failures are those random ssh timeouts which Neutron team is investigating already. I'm going to proceed with option * push 2014.1.1 tags for all but Neutron and release Neutron separately whenever those two are merged
Cheers, Alan
On Thu, Jun 5, 2014 at 4:06 PM, Alan Pevec apevec@gmail.com wrote:
Bad news, 96919 failed in gate-tempest-dsvm-neutron and gate-tempest-dsvm-neutron-pg 96920 had all jobs succeed but since it depends on 96919 it was canceled.
Prematue send... wanted to say, both failures are those random ssh timeouts which Neutron team is investigating already. I'm going to proceed with option
- push 2014.1.1 tags for all but Neutron and release Neutron
separately whenever those two are merged
Yes, I was going to suggest the same at this point.
We're heads down trying to work the "ssh timeout" issue now as well.
Thanks, Kyle
Cheers, Alan
On Thu, Jun 5, 2014 at 4:06 PM, Alan Pevec apevec@gmail.com wrote:
Bad news, 96919 failed in gate-tempest-dsvm-neutron and gate-tempest-dsvm-neutron-pg 96920 had all jobs succeed but since it depends on 96919 it was canceled.
Prematue send... wanted to say, both failures are those random ssh timeouts which Neutron team is investigating already. I'm going to proceed with option
- push 2014.1.1 tags for all but Neutron and release Neutron
separately whenever those two are merged
Alan, now that this has landed [1], we think the "ssh timeout" failure rate will go down significantly. Should we make another run at promoting the two neutron patches to try and get the Neutron 2014.1.1 release done today?
Thanks, Kyle
[1] https://review.openstack.org/#/c/90427/
Cheers, Alan
Should we make another run at promoting the two neutron patches to try and get the Neutron 2014.1.1 release done today?
Let's first see gate is really recovering. I've pushed releases for all other projects but will wait with announcement until Monday. Hopefully by then we'll be able to merge normally without promotions.
Cheers, Alan
On Fri, Jun 6, 2014 at 11:09 AM, Alan Pevec apevec@gmail.com wrote:
Should we make another run at promoting the two neutron patches to try and get the Neutron 2014.1.1 release done today?
Let's first see gate is really recovering. I've pushed releases for all other projects but will wait with announcement until Monday. Hopefully by then we'll be able to merge normally without promotions.
Both of these have merged now, thanks for the help Alan! I think we're good to cut the 2014.1.1 release of Neutron now.
Thanks, Kyle
Cheers, Alan
Both of these have merged now, thanks for the help Alan! I think we're good to cut the 2014.1.1 release of Neutron now.
Done http://launchpad.net/neutron/icehouse/2014.1.1 Announcement email will go out on Monday.
Cheers, Alan
Was pinged about this one today: https://review.openstack.org/#/c/94406/
I +A'd it on Thursday but required a trivial rebase, and has also been at the mercy of the zuul queue in getting it tested. I'm happy with letting this one in but will let Alan +A it at his discretion.
Thanks Adam
On Mon, Jun 2, 2014 at 2:19 AM, Alan Pevec apevec@gmail.com wrote:
First release is coming fast and starting today we are in freeze period, but given the current patch queue[3] I propose we start with soft-freeze until end of day Friday.
Status as of Monday morning: I was rechecking patches with +2 which were hit by random SSH timeouts bugs 1323658 and 1253896. Not all managed to pass through the gate before the whole OpenStack CI got blocked on a regression in Sunday evening setuptools release(s):
https://bitbucket.org/pypa/setuptools/issue/213/regression-setuptools-37-ins... I hope setuptools will be fixed today so we can stick to the schedule and release this Thursday, to avoid conflict with Juno-1 next week.
Soft-freeze is to allow patches currently under review to get merged but no new patches should be accepted. As always, feel free to raise exception requests on stable-maint list.
Does anyone has any exception proposals?
Cheers, Alan
Openstack-stable-maint mailing list Openstack-stable-maint@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-stable-maint
participants (5)
-
Adam Gandelman
-
Alan Pevec
-
Kyle Mestery
-
Kyle Mestery
-
Thierry Carrez