Hey,
We're still hoping to releasing 2012.2.1 on Thursday but we need to lock things down now to allow last minute regression testing.
So, please don't approve any more fixes for the branch until after the release. If there's something you think warrants an exception, leave a +2 with a comment explaining you're thinking and I'll make the call as release manager.
We've had a huge amount of activity in the last week, so I'm a little nervous that we might have broken something. The current "bugs fixed" tallies are:
79 nova 33 quantum 11 glance 7 keystone 3 cinder 1 horizon 134 total
Things to note:
* It looks like there might be a regression on stable/folsom with ceph/rbd:
https://bugs.launchpad.net/nova/+bug/1083818
Adam Gandelman and Josh Durgin are chasing it.
* There's a very serious known issue with the image cache cleanup code when you're using shared storage:
https://bugs.launchpad.net/nova/+bug/1078594
The fix is invasive and risky, but there's a known workaround (i.e. disable the image cache manager) which Vish has documented in the release notes.
We may just disable the image cache manager by default for the release.
* The Quantum linuxbridge plugin fix which allows us to move away from using libvirt's type="ethernet" has been merged, but it isn't really useful without the corresponding change to the Nova VIF driver.
https://review.openstack.org/16223
Given the limited number of users affected, maybe we can consider merging the Nova change before the release too.
One detail here is that if we merge the Nova VIF change, we would need to warn users to update Quantum before Nova.
* Please add stuff to the release notes as it occurs to you:
http://wiki.openstack.org/ReleaseNotes/2012.2.1
Cheers, Mark.
On 11/28/2012 01:57 AM, Mark McLoughlin wrote:
Hey,
We're still hoping to releasing 2012.2.1 on Thursday but we need to lock things down now to allow last minute regression testing.
So, please don't approve any more fixes for the branch until after the release. If there's something you think warrants an exception, leave a +2 with a comment explaining you're thinking and I'll make the call as release manager.
Thanks for the detailed updated and heads up about the approvals. There is one critical patch for Quantum: https://review.openstack.org/#/c/17033/. The problem that we encountered was if namespaces are disabled and the DHCP agents device is not the first device in the routing table (for example if the L3 agents tap device is created first), then instances will not receive IP addresses from the DHCP agent.
There are a couple of other patches pending review for Quantum in the stable branch (I am just testing prior to pushing them). They are not critical for this release and can wait till the next.
We've had a huge amount of activity in the last week, so I'm a little nervous that we might have broken something. The current "bugs fixed" tallies are:
79 nova 33 quantum 11 glance 7 keystone 3 cinder 1 horizon
134 total
Things to note:
It looks like there might be a regression on stable/folsom with ceph/rbd:
https://bugs.launchpad.net/nova/+bug/1083818
Adam Gandelman and Josh Durgin are chasing it.
There's a very serious known issue with the image cache cleanup code when you're using shared storage:
https://bugs.launchpad.net/nova/+bug/1078594
The fix is invasive and risky, but there's a known workaround (i.e. disable the image cache manager) which Vish has documented in the release notes.
We may just disable the image cache manager by default for the release.
The Quantum linuxbridge plugin fix which allows us to move away from using libvirt's type="ethernet" has been merged, but it isn't really useful without the corresponding change to the Nova VIF driver.
https://review.openstack.org/16223
Given the limited number of users affected, maybe we can consider merging the Nova change before the release too.
One detail here is that if we merge the Nova VIF change, we would need to warn users to update Quantum before Nova.
I have added the patch for this - https://review.openstack.org/17036
I am going to be spending the rest of the day testing the stable branches.
Please add stuff to the release notes as it occurs to you:
Cheers, Mark.
Openstack-stable-maint mailing list Openstack-stable-maint@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-stable-maint
There's a very serious known issue with the image cache cleanup code when you're using shared storage:
https://bugs.launchpad.net/nova/+bug/1078594
The fix is invasive and risky, but there's a known workaround (i.e. disable the image cache manager) which Vish has documented in the release notes.
We may just disable the image cache manager by default for the release.
Not may, but must because release note is not enough to cover possible data loss, even if in specific configuration. If https://review.openstack.org/#/c/16726/ is not merged, then we need alternative patch setting image_cache_manager_interval default to 0
Cheers, Alan
On Tue, 2012-11-27 at 23:57 +0000, Mark McLoughlin wrote:
There's a very serious known issue with the image cache cleanup code when you're using shared storage:
https://bugs.launchpad.net/nova/+bug/1078594
The fix is invasive and risky, but there's a known workaround (i.e. disable the image cache manager) which Vish has documented in the release notes.
We may just disable the image cache manager by default for the release.
Is it sufficient to just set remove_unused_base_images to False by default?
Cheers, Mark.
On Wed, 2012-11-28 at 12:57 +0000, Mark McLoughlin wrote:
On Tue, 2012-11-27 at 23:57 +0000, Mark McLoughlin wrote:
There's a very serious known issue with the image cache cleanup code when you're using shared storage:
https://bugs.launchpad.net/nova/+bug/1078594
The fix is invasive and risky, but there's a known workaround (i.e. disable the image cache manager) which Vish has documented in the release notes.
We may just disable the image cache manager by default for the release.
Is it sufficient to just set remove_unused_base_images to False by default?
From IRC:
<vishy> yeah although I think setting the interval=0 might be better <vishy> since it will be going crazy md5summing on shared storage and still running all the logic otherwise <vishy> so it may not delete images but I suspect the performance will still be horrible
Cheers, Mark.
On Wed, 2012-11-28 at 17:46 +0000, Mark McLoughlin wrote:
On Wed, 2012-11-28 at 12:57 +0000, Mark McLoughlin wrote:
On Tue, 2012-11-27 at 23:57 +0000, Mark McLoughlin wrote:
There's a very serious known issue with the image cache cleanup code when you're using shared storage:
https://bugs.launchpad.net/nova/+bug/1078594
The fix is invasive and risky, but there's a known workaround (i.e. disable the image cache manager) which Vish has documented in the release notes.
We may just disable the image cache manager by default for the release.
Is it sufficient to just set remove_unused_base_images to False by default?
From IRC:
<vishy> yeah although I think setting the interval=0 might be better <vishy> since it will be going crazy md5summing on shared storage and still running all the logic otherwise <vishy> so it may not delete images but I suspect the performance will still be horrible
Review posted:
https://review.openstack.org/17064
Cheers, Mark.
Just found a critical bug from grizzly that didn't get backported:
https://review.openstack.org/17072
Vish
On Tue, 2012-11-27 at 23:57 +0000, Mark McLoughlin wrote:
It looks like there might be a regression on stable/folsom with ceph/rbd:
https://bugs.launchpad.net/nova/+bug/1083818
Adam Gandelman and Josh Durgin are chasing it.
Adam did an awesome job of bisecting and fixing this issue:
https://bugs.launchpad.net/cinder/+bug/1083818
It appears that it needs to be backported to cinder stable/folsom too
Cheers, Mark.
participants (4)
-
Alan Pevec
-
Gary Kotton
-
Mark McLoughlin
-
Vishvananda Ishaya