[openstack-dev] [Neutron] Core/Vendor code decomposition

Salvatore Orlando sorlando at nicira.com
Tue Dec 9 16:41:43 UTC 2014

I would like to chime into this discussion wearing my plugin developer hat.

We (the VMware team) have looked very carefully at the current proposal for
splitting off drivers and plugins from the main source code tree. Therefore
the concerns you've heard from Gary are not just ramblings but are the
results of careful examination of this proposal.

While we agree with the final goal, the feeling is that for many plugin
maintainers this process change might be too much for what can be
accomplished in a single release cycle. As a member of the drivers team, I
am still very supportive of the split, I just want to make sure that it’s
made in a sustainable way; I also understand that “sustainability” has been
one of the requirements of the current proposal, and therefore we should
all be on the same page on this aspect.

However, we did a simple exercise trying to assess the amount of work
needed to achieve something which might be acceptable to satisfy the
process. Without going into too many details, this requires efforts for:

- refactor the code to achieve a plugin module simple and thin enough to
satisfy the requirements. Unfortunately a radical approach like the one in
[1] with a reference to an external library is not pursuable for us

- maintaining code repositories outside of the neutron scope and the
necessary infrastructure

- reinforcing our CI infrastructure, and improve our error detection and
log analysis capabilities to improve reaction times upon failures triggered
by upstream changes. As you know, even if the plugin interface is
solid-ish, the dependency on the db base class increases the chances of
upstream changes breaking 3rd party plugins.

The feedback from our engineering team is that satisfying the requirements
of this new process might not be feasible in the Kilo timeframe, both for
existing plugins and for new plugins and drivers that should be upstreamed
(there are a few proposed on neutron-specs at the moment, which are all in
-2 status considering the impending approval of the split out).

The questions I would like to bring to the wider community are therefore
the following:

1 - Is there a possibility of making a further concession on the current
proposal, where maintainers are encouraged to experiment with the plugin
split in Kilo, but will actually required to do it in the next release?

2 - What could be considered as a acceptable as a new plugin? I understand
that they would be accepted only as “thin integration modules”, which
ideally should just be a pointer to code living somewhere else. I’m not
questioning the validity of this approach, but it has been brought to my
attention that this will actually be troubling for teams which have made an
investment in the previous release cycles to upstream plugins following the
“old” process

3 - Regarding the above discussion on "ML2 or not ML2". The point on
co-gating is well taken. Eventually we'd like to remove this binding -
because I believe the ML2 subteam would also like to have more freedom on
their plugin. Do we already have an idea about how doing that without
completely moving away from the db_base class approach?

Thanks for your attention and for reading through this



On 8 December 2014 at 21:51, Maru Newby <marun at redhat.com> wrote:

> On Dec 7, 2014, at 10:51 AM, Gary Kotton <gkotton at vmware.com> wrote:
> > Hi Kyle,
> > I am not missing the point. I understand the proposal. I just think that
> it has some shortcomings (unless I misunderstand, which will certainly not
> be the first time and most definitely not the last). The thinning out is to
> have a shim in place. I understand this and this will be the entry point
> for the plugin. I do not have a concern for this. My concern is that we are
> not doing this with the ML2 off the bat. That should lead by example as it
> is our reference architecture. Lets not kid anyone, but we are going  to
> hit some problems with the decomposition. I would prefer that it be done
> with the default implementation. Why?
> The proposal is to move vendor-specific logic out of the tree to increase
> vendor control over such code while decreasing load on reviewers.  ML2
> doesn’t contain vendor-specific logic - that’s the province of ML2 drivers
> - so it is not a good target for the proposed decomposition by itself.
> >       • Cause we will fix them quicker as it is something that prevent
> Neutron from moving forwards
> >       • We will just need to fix in one place first and not in N (where
> N is the vendor plugins)
> >       • This is a community effort – so we will have a lot more eyes on
> it
> >       • It will provide a reference architecture for all new plugins
> that want to be added to the tree
> >       • It will provide a working example for plugin that are already in
> tree and are to be replaced by the shim
> > If we really want to do this, we can say freeze all development (which
> is just approvals for patches) for a few days so that we will can just
> focus on this. I stated what I think should be the process on the review.
> For those who do not feel like finding the link:
> >       • Create a stack forge project for ML2
> >       • Create the shim in Neutron
> >       • Update devstack for the to use the two repos and the shim
> > When #3 is up and running we switch for that to be the gate. Then we
> start a stopwatch on all other plugins.
> As was pointed out on the spec (see Miguel’s comment on r15), the ML2
> plugin and the OVS mechanism driver need to remain in the main Neutron repo
> for now.  Neutron gates on ML2+OVS and landing a breaking change in the
> Neutron repo along with its corresponding fix to a separate ML2 repo would
> be all but impossible under the current integrated gating scheme.
> Plugins/drivers that do not gate Neutron have no such constraint.
> Maru
> > Sure, I’ll catch you on IRC tomorrow. I guess that you guys will bash
> out the details at the meetup. Sadly I will not be able to attend – so you
> will have to delay on the tar and feathers.
> > Thanks
> > Gary
> >
> >
> > From: "mestery at mestery.com" <mestery at mestery.com>
> > Reply-To: OpenStack List <openstack-dev at lists.openstack.org>
> > Date: Sunday, December 7, 2014 at 7:19 PM
> > To: OpenStack List <openstack-dev at lists.openstack.org>
> > Cc: "openstack at lists.openstack.org" <openstack at lists.openstack.org>
> > Subject: Re: [openstack-dev] [Neutron] Core/Vendor code decomposition
> >
> > Gary, you are still miss the point of this proposal. Please see my
> comments in review. We are not forcing things out of tree, we are thinning
> them. The text you quoted in the review makes that clear. We will look at
> further decomposing ML2 post Kilo, but we have to be realistic with what we
> can accomplish during Kilo.
> >
> > Find me on IRC Monday morning and we can discuss further if you still
> have questions and concerns.
> >
> > Thanks!
> > Kyle
> >
> > On Sun, Dec 7, 2014 at 2:08 AM, Gary Kotton <gkotton at vmware.com> wrote:
> >> Hi,
> >> I have raised my concerns on the proposal. I think that all plugins
> should be treated on an equal footing. My main concern is having the ML2
> plugin in tree whilst the others will be moved out of tree will be
> problematic. I think that the model will be complete if the ML2 was also
> out of tree. This will help crystalize the idea and make sure that the
> model works correctly.
> >> Thanks
> >> Gary
> >>
> >> From: "Armando M." <armamig at gmail.com>
> >> Reply-To: OpenStack List <openstack-dev at lists.openstack.org>
> >> Date: Saturday, December 6, 2014 at 1:04 AM
> >> To: OpenStack List <openstack-dev at lists.openstack.org>, "
> openstack at lists.openstack.org" <openstack at lists.openstack.org>
> >> Subject: [openstack-dev] [Neutron] Core/Vendor code decomposition
> >>
> >> Hi folks,
> >>
> >> For a few weeks now the Neutron team has worked tirelessly on [1].
> >>
> >> This initiative stems from the fact that as the project matures,
> evolution of processes and contribution guidelines need to evolve with it.
> This is to ensure that the project can keep on thriving in order to meet
> the needs of an ever growing community.
> >>
> >> The effort of documenting intentions, and fleshing out the various
> details of the proposal is about to reach an end, and we'll soon kick the
> tires to put the proposal into practice. Since the spec has grown pretty
> big, I'll try to capture the tl;dr below.
> >>
> >> If you have any comment please do not hesitate to raise them here
> and/or reach out to us.
> >>
> >> tl;dr >>>
> >>
> >> From the Kilo release, we'll initiate a set of steps to change the
> following areas:
> >>      • Code structure: every plugin or driver that exists or wants to
> exist as part of Neutron project is decomposed in an slim vendor
> integration (which lives in the Neutron repo), plus a bulkier vendor
> library (which lives in an independent publicly available repo);
> >>      • Contribution process: this extends to the following aspects:
> >>              • Design and Development: the process is largely unchanged
> for the part that pertains the vendor integration; the maintainer team is
> fully auto governed for the design and development of the vendor library;
> >>              • Testing and Continuous Integration: maintainers will be
> required to support their vendor integration with 3rd CI testing; the
> requirements for 3rd CI testing are largely unchanged;
> >>              • Defect management: the process is largely unchanged,
> issues affecting the vendor library can be tracked with whichever
> tool/process the maintainer see fit. In cases where vendor library fixes
> need to be reflected in the vendor integration, the usual OpenStack defect
> management apply.
> >>              • Documentation: there will be some changes to the way
> plugins and drivers are documented with the intention of promoting
> discoverability of the integrated solutions.
> >>      • Adoption and transition plan: we strongly advise maintainers to
> stay abreast of the developments of this effort, as their code, their CI,
> etc will be affected. The core team will provide guidelines and support
> throughout this cycle the ensure a smooth transition.
> >> To learn more, please refer to [1].
> >>
> >> Many thanks,
> >> Armando
> >>
> >> [1] https://review.openstack.org/#/c/134680
> >>
> >> _______________________________________________
> >> OpenStack-dev mailing list
> >> OpenStack-dev at lists.openstack.org
> >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >>
> >
> > _______________________________________________
> > OpenStack-dev mailing list
> > OpenStack-dev at lists.openstack.org
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20141209/501e5608/attachment.html>

More information about the OpenStack-dev mailing list