[openstack-dev] [Neutron] Core/Vendor code decomposition

Armando M. armamig at gmail.com
Tue Dec 9 17:32:52 UTC 2014


On 9 December 2014 at 09:41, Salvatore Orlando <sorlando at nicira.com> wrote:

> I would like to chime into this discussion wearing my plugin developer hat.
>
> We (the VMware team) have looked very carefully at the current proposal
> for splitting off drivers and plugins from the main source code tree.
> Therefore the concerns you've heard from Gary are not just ramblings but
> are the results of careful examination of this proposal.
>
> While we agree with the final goal, the feeling is that for many plugin
> maintainers this process change might be too much for what can be
> accomplished in a single release cycle.
>
We actually gave a lot more than a cycle:

https://review.openstack.org/#/c/134680/16/specs/kilo/core-vendor-decomposition.rst
LINE 416

And in all honestly, I can only tell that getting this done by such an
experienced team like the Neutron team @VMware shouldn't take that long.

By the way, if Kyle can do it in his teeny tiny time that he has left after
his PTL duties, then anyone can do it! :)

https://review.openstack.org/#/c/140191/

> As a member of the drivers team, I am still very supportive of the split,
> I just want to make sure that it’s made in a sustainable way; I also
> understand that “sustainability” has been one of the requirements of the
> current proposal, and therefore we should all be on the same page on this
> aspect.
>
> However, we did a simple exercise trying to assess the amount of work
> needed to achieve something which might be acceptable to satisfy the
> process. Without going into too many details, this requires efforts for:
>
> - refactor the code to achieve a plugin module simple and thin enough to
> satisfy the requirements. Unfortunately a radical approach like the one in
> [1] with a reference to an external library is not pursuable for us
>
> - maintaining code repositories outside of the neutron scope and the
> necessary infrastructure
>
> - reinforcing our CI infrastructure, and improve our error detection and
> log analysis capabilities to improve reaction times upon failures triggered
> by upstream changes. As you know, even if the plugin interface is
> solid-ish, the dependency on the db base class increases the chances of
> upstream changes breaking 3rd party plugins.
>

No-one is advocating for approach laid out in [1], but a lot of code can be
moved elsewhere (like the nsxlib) without too much effort. Don't forget
that not so long ago I was the maintainer of this plugin and the one who
built the VMware NSX CI; I know very well what it takes to scope this
effort, and I can support you in the process.

> The feedback from our engineering team is that satisfying the requirements
> of this new process might not be feasible in the Kilo timeframe, both for
> existing plugins and for new plugins and drivers that should be upstreamed
> (there are a few proposed on neutron-specs at the moment, which are all in
> -2 status considering the impending approval of the split out).
>
No new plugins can and will be accepted if they do not adopt the proposed
model, let's be very clear about this.

> The questions I would like to bring to the wider community are therefore
> the following:
>
> 1 - Is there a possibility of making a further concession on the current
> proposal, where maintainers are encouraged to experiment with the plugin
> split in Kilo, but will actually required to do it in the next release?
>
This is exactly what the spec is proposing: get started now, and it does
not matter if you don't finish in time.

> 2 - What could be considered as a acceptable as a new plugin? I understand
> that they would be accepted only as “thin integration modules”, which
> ideally should just be a pointer to code living somewhere else. I’m not
> questioning the validity of this approach, but it has been brought to my
> attention that this will actually be troubling for teams which have made an
> investment in the previous release cycles to upstream plugins following the
> “old” process
>
You are not alone. Other efforts went through the same process [1, 2, 3].
Adjusting is a way of life. No-one is advocating for throwing away existing
investment. This proposal actually promotes new and pre-existing investment.

[1] https://review.openstack.org/#/c/104452/
[2] https://review.openstack.org/#/c/103728/
[3] https://review.openstack.org/#/c/136091/

> 3 - Regarding the above discussion on "ML2 or not ML2". The point on
> co-gating is well taken. Eventually we'd like to remove this binding -
> because I believe the ML2 subteam would also like to have more freedom on
> their plugin. Do we already have an idea about how doing that without
> completely moving away from the db_base class approach?
>
Sure, if you like to participate in the process, we can only welcome you!

> Thanks for your attention and for reading through this
>
> Salvatore
>
> [1]
> http://git.openstack.org/cgit/openstack/neutron/tree/neutron/plugins/vmware/plugin.py#n22
>
> On 8 December 2014 at 21:51, Maru Newby <marun at redhat.com> wrote:
>
>>
>> On Dec 7, 2014, at 10:51 AM, Gary Kotton <gkotton at vmware.com> wrote:
>>
>> > Hi Kyle,
>> > I am not missing the point. I understand the proposal. I just think
>> that it has some shortcomings (unless I misunderstand, which will certainly
>> not be the first time and most definitely not the last). The thinning out
>> is to have a shim in place. I understand this and this will be the entry
>> point for the plugin. I do not have a concern for this. My concern is that
>> we are not doing this with the ML2 off the bat. That should lead by example
>> as it is our reference architecture. Lets not kid anyone, but we are going
>> to hit some problems with the decomposition. I would prefer that it be done
>> with the default implementation. Why?
>>
>> The proposal is to move vendor-specific logic out of the tree to increase
>> vendor control over such code while decreasing load on reviewers.  ML2
>> doesn’t contain vendor-specific logic - that’s the province of ML2 drivers
>> - so it is not a good target for the proposed decomposition by itself.
>>
>>
>> >       • Cause we will fix them quicker as it is something that prevent
>> Neutron from moving forwards
>> >       • We will just need to fix in one place first and not in N (where
>> N is the vendor plugins)
>> >       • This is a community effort – so we will have a lot more eyes on
>> it
>> >       • It will provide a reference architecture for all new plugins
>> that want to be added to the tree
>> >       • It will provide a working example for plugin that are already
>> in tree and are to be replaced by the shim
>> > If we really want to do this, we can say freeze all development (which
>> is just approvals for patches) for a few days so that we will can just
>> focus on this. I stated what I think should be the process on the review.
>> For those who do not feel like finding the link:
>> >       • Create a stack forge project for ML2
>> >       • Create the shim in Neutron
>> >       • Update devstack for the to use the two repos and the shim
>> > When #3 is up and running we switch for that to be the gate. Then we
>> start a stopwatch on all other plugins.
>>
>> As was pointed out on the spec (see Miguel’s comment on r15), the ML2
>> plugin and the OVS mechanism driver need to remain in the main Neutron repo
>> for now.  Neutron gates on ML2+OVS and landing a breaking change in the
>> Neutron repo along with its corresponding fix to a separate ML2 repo would
>> be all but impossible under the current integrated gating scheme.
>> Plugins/drivers that do not gate Neutron have no such constraint.
>>
>>
>> Maru
>>
>>
>> > Sure, I’ll catch you on IRC tomorrow. I guess that you guys will bash
>> out the details at the meetup. Sadly I will not be able to attend – so you
>> will have to delay on the tar and feathers.
>> > Thanks
>> > Gary
>> >
>> >
>> > From: "mestery at mestery.com" <mestery at mestery.com>
>> > Reply-To: OpenStack List <openstack-dev at lists.openstack.org>
>> > Date: Sunday, December 7, 2014 at 7:19 PM
>> > To: OpenStack List <openstack-dev at lists.openstack.org>
>> > Cc: "openstack at lists.openstack.org" <openstack at lists.openstack.org>
>> > Subject: Re: [openstack-dev] [Neutron] Core/Vendor code decomposition
>> >
>> > Gary, you are still miss the point of this proposal. Please see my
>> comments in review. We are not forcing things out of tree, we are thinning
>> them. The text you quoted in the review makes that clear. We will look at
>> further decomposing ML2 post Kilo, but we have to be realistic with what we
>> can accomplish during Kilo.
>> >
>> > Find me on IRC Monday morning and we can discuss further if you still
>> have questions and concerns.
>> >
>> > Thanks!
>> > Kyle
>> >
>> > On Sun, Dec 7, 2014 at 2:08 AM, Gary Kotton <gkotton at vmware.com> wrote:
>> >> Hi,
>> >> I have raised my concerns on the proposal. I think that all plugins
>> should be treated on an equal footing. My main concern is having the ML2
>> plugin in tree whilst the others will be moved out of tree will be
>> problematic. I think that the model will be complete if the ML2 was also
>> out of tree. This will help crystalize the idea and make sure that the
>> model works correctly.
>> >> Thanks
>> >> Gary
>> >>
>> >> From: "Armando M." <armamig at gmail.com>
>> >> Reply-To: OpenStack List <openstack-dev at lists.openstack.org>
>> >> Date: Saturday, December 6, 2014 at 1:04 AM
>> >> To: OpenStack List <openstack-dev at lists.openstack.org>, "
>> openstack at lists.openstack.org" <openstack at lists.openstack.org>
>> >> Subject: [openstack-dev] [Neutron] Core/Vendor code decomposition
>> >>
>> >> Hi folks,
>> >>
>> >> For a few weeks now the Neutron team has worked tirelessly on [1].
>> >>
>> >> This initiative stems from the fact that as the project matures,
>> evolution of processes and contribution guidelines need to evolve with it.
>> This is to ensure that the project can keep on thriving in order to meet
>> the needs of an ever growing community.
>> >>
>> >> The effort of documenting intentions, and fleshing out the various
>> details of the proposal is about to reach an end, and we'll soon kick the
>> tires to put the proposal into practice. Since the spec has grown pretty
>> big, I'll try to capture the tl;dr below.
>> >>
>> >> If you have any comment please do not hesitate to raise them here
>> and/or reach out to us.
>> >>
>> >> tl;dr >>>
>> >>
>> >> From the Kilo release, we'll initiate a set of steps to change the
>> following areas:
>> >>      • Code structure: every plugin or driver that exists or wants to
>> exist as part of Neutron project is decomposed in an slim vendor
>> integration (which lives in the Neutron repo), plus a bulkier vendor
>> library (which lives in an independent publicly available repo);
>> >>      • Contribution process: this extends to the following aspects:
>> >>              • Design and Development: the process is largely
>> unchanged for the part that pertains the vendor integration; the maintainer
>> team is fully auto governed for the design and development of the vendor
>> library;
>> >>              • Testing and Continuous Integration: maintainers will be
>> required to support their vendor integration with 3rd CI testing; the
>> requirements for 3rd CI testing are largely unchanged;
>> >>              • Defect management: the process is largely unchanged,
>> issues affecting the vendor library can be tracked with whichever
>> tool/process the maintainer see fit. In cases where vendor library fixes
>> need to be reflected in the vendor integration, the usual OpenStack defect
>> management apply.
>> >>              • Documentation: there will be some changes to the way
>> plugins and drivers are documented with the intention of promoting
>> discoverability of the integrated solutions.
>> >>      • Adoption and transition plan: we strongly advise maintainers to
>> stay abreast of the developments of this effort, as their code, their CI,
>> etc will be affected. The core team will provide guidelines and support
>> throughout this cycle the ensure a smooth transition.
>> >> To learn more, please refer to [1].
>> >>
>> >> Many thanks,
>> >> Armando
>> >>
>> >> [1] https://review.openstack.org/#/c/134680
>> >>
>> >> _______________________________________________
>> >> OpenStack-dev mailing list
>> >> OpenStack-dev at lists.openstack.org
>> >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>> >>
>> >
>> > _______________________________________________
>> > OpenStack-dev mailing list
>> > OpenStack-dev at lists.openstack.org
>> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>>
>> _______________________________________________
>> OpenStack-dev mailing list
>> OpenStack-dev at lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>
>
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20141209/91bff565/attachment-0001.html>


More information about the OpenStack-dev mailing list