[openstack-dev] [Neutron] Ironing out outstanding issues in time for RC1

Armando M. armamig at gmail.com
Mon Mar 14 02:31:02 UTC 2016

On 13 March 2016 at 15:14, Ihar Hrachyshka <ihrachys at redhat.com> wrote:

> Armando M. <armamig at gmail.com> wrote:
> Neutrinos,
>> We have about ~20 outstanding bugs marked Medium/High/Critical, and we
>> have only one or two days left to have a chance to get them in the gate
>> queue [1]. Can I trouble you to go and make sure patches are up to date and
>> well reviewed?
>> Many thanks,
>> Armando
>> [1] https://launchpad.net/neutron/+milestone/mitaka-rc1
> Hi Armando and all,
> [Full disclosure: I am really interested in getting stable/mitaka cut off
> asap due to the code sprint starting on Mon where we would like to land a
> number of N bits to master.]
Do you have a list of patches that are currently blocked by the feature
freeze? I have this:


I appreciate your concern, and we should do everything we can to enable you
guys to have a productive sprint. That said, it's unfortunate that the
sprint ended up being scheduled for this week, where attention should
really be devoted to Mitaka rather than Newton...but I guess you couldn't
predict the state of the codebase too far in advance.

> Currently I see 25 unreleased bugs targeted for RC1. I believe the list is
> too broad and does not represent actual team priorities as of right now; I
> suggest we go thru the list and postpone those bugs that either won’t land
> on Mon, or aren't really critical for the release; then cut-off
> stable/mitaka to unblock master.

Yes it is (you should have seen the list before I already had my first pass
;)). Bear in mind that most of these bugs ended up being automatically
rolled over to RC1 being targeted for earlier milestones (some of them are
as old as Liberty).

> If you think the list is fine, remember that at this point we should land
> only safe fixes, or those that make release impossible [aka ‘something base
> to the cloud operation is totally broken in Mitaka comparing to Liberty’].
> If you like, you can compare Neutron with its 25 targeted bugs to other
> projects (hint: nova - 2 bugs; glance - 2 bugs; cinder - 5 bugs; horizon -
> 12 bugs).

You should not trust the Launchpad view of other projects, because,
especially if you looked at Nova, this would mean that they fixed virtually
nothing in Mitaka, which is clearly not true. In other words, some projects
moved away from tracking issues using Launchpad. So rest assured that we're
not as lousy as we look.

> If we would start landing all cool stuff that we happen to produce in the
> last week, we would be undermining freeze and release process, also raising
> chances of releasing a pile of broken code.

Can you start telling me something I don't know? :)

> With that in mind, I went through the target to see what is really
> critical for the release.
> ===

Great, let's dig in!

Bear in mind that we can really consider RC1 bugs those for which we, as a
team, can clearly identify fixes that can safely land in the next day or
so. Any other, sadly, must be dropped no matter how bad it looks, due to
the time constraint we have.

> We have 18 bugs that are High+. Below is each of those bugs, with [*] mark
> where I think RC target is justified at this point.
> https://bugs.launchpad.net/neutron/+bug/1523031: linuxbridge + l2pop +
> l3ha broken.
> ^ while it’s unfortunate, I don’t see how it stands for a release critical
> bug since it affects setup that is not really that common. Also, looking at
> the bug state, I don’t see any work started on it. I would prefer we drop
> it out of RC1 target.


> https://bugs.launchpad.net/bugs/1551288
> ^ fullstack (non-voting) job sometimes fails for native ovsdb interface.
> No idea why it’s critical for the release. Suggesting to postpone to N.

Agreed. The only comment I have to make here is: the sooner we make
fullstack voting the better. I wish we could have made that a Mitaka
priority so that we'd have an extra tool in our stability arsenal.

> https://bugs.launchpad.net/bugs/1513765 [*]
> ^ conntrack calls block ovs agent; patch in review optimizes for some use
> cases, but does not tackle the general issue of the agent being blocked;
> patch opens some rolling upgrade scenario concerns too since it touches RPC
> version and method signatures; that said, we indeed want to try to tackle
> it in RC;

> https://bugs.launchpad.net/neutron/+bug/1556139 [*]
> ^ create/delete race when returning new resource body in ml2; the patch is
> up for review and ready for merge; good to go;
> https://bugs.launchpad.net/neutron/+bug/1453350
> ^ port ready event sent to nova before dhcp is ready; not a regression:
> was always the case; the proper fix would require a lot of work; definitely
> won’t happen in Mitaka; I believe should be dropped from RC target;
> https://bugs.launchpad.net/neutron/+bug/1456073 [*]
> ^ dvr not ready yet when live migration triggered; targeted for RC on both
> nova and neutron sides; neutron patch in review, has needed +2s; good to go;
> https://bugs.launchpad.net/neutron/+bug/1462154
> ^ dvr returns fip targeted pings with fixed ips; patch up for review
> [WIP]; not sure whether it will make it; I suggest we don’t wait for that
> one;
I know the team may have a working fix. Before we drop it, let's seek an
update from the owner.

> https://bugs.launchpad.net/bugs/1478100
> ^ physical network aware dhcp scheduler; honestly, that one is rather
> feature-ish; I would untarget it from Mitaka on that ground;

This was very close at one point. This would unlock a number of scenarios,
before we drop it, let's seek an update from the owner.

> https://bugs.launchpad.net/bugs/1491131
> ^ ipset race; this one seems abandoned right now; no patches for review;
> long standing issue; I would move it to N;

We should get someone to look into it to assess how bad this is for Mitaka,
it would be awful if we shipped like this.

> https://bugs.launchpad.net/neutron/+bug/1498790
> ^ allow to delete other’s ports from your network; while obviously a bug,
> the fix could be also considered rather feature-ish, since it change API
> behaviour; there seems to be no active patches in gerrit for that; I would
> postpone it till N when we’ll have more time to look into the proper fix;

I spoke with Kevin about this, I think we have a fix almost ready to land.

> https://bugs.launchpad.net/bugs/1501206 [*]
> ^ dhcp ports reply to requests from other subnets; potential security
> hole; patch up for review; could indeed be worth looking at for RC;

> https://bugs.launchpad.net/bugs/1514056 [*]
> ^ vlan external connectivity disrupted on ovs agent restart; patch up for
> review [tiny one, need respin]; not a regression; would be fine to see it
> fixed in RC but not a tragedy if it’s skipped;
> https://bugs.launchpad.net/neutron/+bug/1528895
> ^ bulk rpc calls time out for l2 agent with default rpc timeout; honestly,
> I don’t believe it’s valid to have it High (there is a tunable for the
> timeout provided by oslo.messaging library); no proper fix up for review; I
> would say, we should postpone to N;
> https://bugs.launchpad.net/bugs/1533034
> ^ exception error fix; breaks i18n freeze; I don’t believe it justifies RC
> target at all since nothing is broken;

Yes, this can easily be postponed, I target it to keep an eye on the
triaging, but after looking into it, it doesn't look too scary.

> https://bugs.launchpad.net/neutron/+bug/1541742
> ^ fullstack database teardown failing; workaround already merged; proper
> fix would probably require changes in db backend; should be moved to N;
> https://bugs.launchpad.net/bugs/1543094
> ^ retry exception in ipam code; no patch in gerrit; bug comments suggest
> that a fix would involve alembic, which is not allowed anymore for M;
> should be postponed to N;
> https://bugs.launchpad.net/bugs/1554332
> ^ agents are too aggressive fetching; base patch in gerrit, though agents
> are not touched yet; honestly, seems a bit late to do such a change since
> we would not have time to stabilize on it; I suggest we defer to N;

Although this can be deferred to N, it would nice to land the partial fix
for Mitaka to allow for a potential easy backport if we wanted to, so
consider nudging this one in:


> https://bugs.launchpad.net/bugs/1555356
> ^ neutron imports from tempest private code; while nice to have, I don’t
> believe it should be tracked for RC.

That bug won't be closed in time for RC1, it's there to remind us we gotta
get on with it ;)

> (Other bugs are Medium and lower.)
> ===
> So from the list of 18 High+ bugs, only 5 of them are somewhat critical
> for release success [and even that list could be trimmed with no tragical
> consequences].
> I suggest we focus on those five and cut off stable/mitaka branch when
> most of them land master [or we know that it won’t happen in a day]. For
> other stuff, I prefer we take best effort approach and backport to
> stable/mitaka before final release if fixes are in time for master, AND
> they really justify High priority, AND if they are really proved to be safe.
> Thoughts?

After this virtual chat, I currently count 19 bugs in progress (regardless
of the priority). We should look at those that have code that's close to
landing and have one final push - we as reviewers should take ownership of
the fix if needed. I intend to cut Mitaka sometime on Tuesday: what's in is
in, what's out is out. I am preparing the release patches. Once that's
done, stable will be chose from the RC1 tag, and at that point everything
rolls over N-1 or to RC2 if we end up finding the need for one.

> Ihar
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20160313/b5157873/attachment.html>

More information about the OpenStack-dev mailing list