[openstack-dev] [all] icehouse failure rates are somewhat catastrophic - who is actually maintaining it?

Sean Dague sean at dague.net
Thu Oct 2 11:49:40 UTC 2014


On 10/02/2014 04:47 AM, Ihar Hrachyshka wrote:
> Hi,
> 
> I guess the following review is meant:
> https://review.openstack.org/#/c/125075/
> 
> I went thru each of the failure for the patch (no dependency failures
> checked), and here are some damned lies (c) about those failures:
> 
> - bug 1323658: 2 failures (ssh connection timeout issue, shows up in
> Neutron jobs)
> - bug 1331274: 2 failures (Grenade not starting services)
> - bug 1375108: 4 failures (bug in Nova EC2 reboot code?)
> - bug 1348204: 1 failure (Cinder volume detach failing)
> - bug 1374175: 1 failure (Heat bug)
> 
> Neither of those bugs are solved in master. Some of those bugs are
> staying opened for months. The first bug was raised as Neutron bug and
> marked for RC-1, but then was untargeted due to believe among Neutron
> developers that it's a bug in Tempest.
> 
> Nevertheless, with all the hopeless state of the gate, in the Icehouse
> release scheduled for today stable maint team was able to merge fixes
> for more than 120 bugs.
> 
> So, all that said, does anyone believe that it's fair to bitch about
> stable maintainers not doing their job? Wouldn't it be more fair to
> bitch about the overall hopeless state of the gate and projects
> feeling ok releasing Juno with major failures in gate (like in case of
> the very first bug in the list)?
> 
> /Ihar

Fwiw, this whole patch stream is part of the fix for #1331274 in Juno /
Master (we need to get off of screen in grenade to have been process
control) -
https://review.openstack.org/#/q/status:open+project:openstack-dev/devstack+branch:stable/icehouse+topic:no_screen,n,z
and in order to merge any devstack icehouse changes we needed to fix the
tox bashate targets (https://review.openstack.org/#/c/125075/ is
actually a trivial additional add, so it made a really good indication
that this was latent state).

After those fails on this patch - we correctly disabled grenade on
icehouse (it should have been turned off when havana eoled, there was a
delay) - https://review.openstack.org/#/c/125371/

I made a bunch of noise about #1323658 in the project meeting, spent a
bunch of time on chasing that after, I agree that punting on it for the
release was a very questionable call. I proposed the skip -
https://review.openstack.org/#/c/125150/ which was merged.

I marked #1374175 as critical for the heat team -
https://bugs.launchpad.net/heat/+bug/1374175 - also skipping it is
working it's way through the gate now -
https://review.openstack.org/#/c/125545/.

Joe, Matt Treinish, and I looked at #1375108 last night, and found that
the test author had recheck grinded their test in ... even when it
failed in related areas. So that was straight reverted -
https://review.openstack.org/#/c/125543/

I have not looked into 1348204 at all - I just bumped it up to critical.
Looks like Matt Riedeman has a debug patch out there.

...

But that seems to answer the question I was asking. Who's maintaining
this seems to be me (or more specifically the same people that are
always working these bugs, me, joe, matt, and matt).

I don't want it to be me. Because as soon as I manage to get the
infrastructure for fixing #1331274 I want nothing more to do with
icehouse stable.

What I'm complaining about is that until I spent time on #1331274 no one
seemed to understand that devstack patches aren't mergable (and hadn't
been for weeks). If stable was maintained I'd expect that someone would
actually know the current state, or be helping with it. Also icehouse is
about to no longer be installable with pip because of pip changes.
https://review.openstack.org/#/c/124648/ has to land to fix that, other
devstack patches have to land to get us there.

If stable branches are important to the project, then stable branches
need to be front and center in the weekly project meeting. Maintaining a
thing is actually knowing the current status and working to make it better.

Removing tests is a totally fine thing to propose, for instance.

But the point is, raised well by Alan, the stable branches are basically
only being worked on by one distro, and no other vendors. So honestly,
if that doesn't change, I'd suggest dropping all but the most recent one
(so we can test upgrade testing for current master).

	-Sean

-- 
Sean Dague
http://dague.net

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 482 bytes
Desc: OpenPGP digital signature
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20141002/96b20d77/attachment.pgp>


More information about the OpenStack-dev mailing list