[openstack-dev] [Neutron] Core/Vendor code decomposition
Salvatore Orlando
sorlando at nicira.com
Tue Dec 9 17:53:48 UTC 2014
On 9 December 2014 at 17:32, Armando M. <armamig at gmail.com> wrote:
>
>
> On 9 December 2014 at 09:41, Salvatore Orlando <sorlando at nicira.com>
> wrote:
>
>> I would like to chime into this discussion wearing my plugin developer
>> hat.
>>
>> We (the VMware team) have looked very carefully at the current proposal
>> for splitting off drivers and plugins from the main source code tree.
>> Therefore the concerns you've heard from Gary are not just ramblings but
>> are the results of careful examination of this proposal.
>>
>> While we agree with the final goal, the feeling is that for many plugin
>> maintainers this process change might be too much for what can be
>> accomplished in a single release cycle.
>>
> We actually gave a lot more than a cycle:
>
>
> https://review.openstack.org/#/c/134680/16/specs/kilo/core-vendor-decomposition.rst
> LINE 416
>
> And in all honestly, I can only tell that getting this done by such an
> experienced team like the Neutron team @VMware shouldn't take that long.
>
We are probably not experienced enough. We always love to learn new things.
>
> By the way, if Kyle can do it in his teeny tiny time that he has left
> after his PTL duties, then anyone can do it! :)
>
> https://review.openstack.org/#/c/140191/
>
I think I should be able to use mv & git push as well - I think however
there's a bit more than that to it.
>
> As a member of the drivers team, I am still very supportive of the split,
>> I just want to make sure that it’s made in a sustainable way; I also
>> understand that “sustainability” has been one of the requirements of the
>> current proposal, and therefore we should all be on the same page on this
>> aspect.
>>
>> However, we did a simple exercise trying to assess the amount of work
>> needed to achieve something which might be acceptable to satisfy the
>> process. Without going into too many details, this requires efforts for:
>>
>> - refactor the code to achieve a plugin module simple and thin enough to
>> satisfy the requirements. Unfortunately a radical approach like the one in
>> [1] with a reference to an external library is not pursuable for us
>>
>> - maintaining code repositories outside of the neutron scope and the
>> necessary infrastructure
>>
>> - reinforcing our CI infrastructure, and improve our error detection and
>> log analysis capabilities to improve reaction times upon failures triggered
>> by upstream changes. As you know, even if the plugin interface is
>> solid-ish, the dependency on the db base class increases the chances of
>> upstream changes breaking 3rd party plugins.
>>
>
> No-one is advocating for approach laid out in [1], but a lot of code can
> be moved elsewhere (like the nsxlib) without too much effort. Don't forget
> that not so long ago I was the maintainer of this plugin and the one who
> built the VMware NSX CI; I know very well what it takes to scope this
> effort, and I can support you in the process.
>
Thanks for this clarification. I was sure that you guys were not advocating
for a ninja-split thing, but I wanted just to be sure of that.
I'm also pretty sure our engineering team values your support.
> The feedback from our engineering team is that satisfying the requirements
>> of this new process might not be feasible in the Kilo timeframe, both for
>> existing plugins and for new plugins and drivers that should be upstreamed
>> (there are a few proposed on neutron-specs at the moment, which are all in
>> -2 status considering the impending approval of the split out).
>>
> No new plugins can and will be accepted if they do not adopt the proposed
> model, let's be very clear about this.
>
This is also what I gathered from the proposal. It seems that you're
however stating that there might be some flexibility in defining how much a
plugin complies with the new model. I will need to go back to the drawing
board with the rest of my team and see in which way this can work for us.
> The questions I would like to bring to the wider community are therefore
>> the following:
>>
>> 1 - Is there a possibility of making a further concession on the current
>> proposal, where maintainers are encouraged to experiment with the plugin
>> split in Kilo, but will actually required to do it in the next release?
>>
> This is exactly what the spec is proposing: get started now, and it does
> not matter if you don't finish in time.
>
I think the deprecation note at line 416 still scares people off a bit. To
me your word is enough, no change is needed.
> 2 - What could be considered as a acceptable as a new plugin? I understand
>> that they would be accepted only as “thin integration modules”, which
>> ideally should just be a pointer to code living somewhere else. I’m not
>> questioning the validity of this approach, but it has been brought to my
>> attention that this will actually be troubling for teams which have made an
>> investment in the previous release cycles to upstream plugins following the
>> “old” process
>>
> You are not alone. Other efforts went through the same process [1, 2, 3].
> Adjusting is a way of life. No-one is advocating for throwing away existing
> investment. This proposal actually promotes new and pre-existing investment.
>
> [1] https://review.openstack.org/#/c/104452/
> [2] https://review.openstack.org/#/c/103728/
> [3] https://review.openstack.org/#/c/136091/
>
3 - Regarding the above discussion on "ML2 or not ML2". The point on
>> co-gating is well taken. Eventually we'd like to remove this binding -
>> because I believe the ML2 subteam would also like to have more freedom on
>> their plugin. Do we already have an idea about how doing that without
>> completely moving away from the db_base class approach?
>>
> Sure, if you like to participate in the process, we can only welcome you!
>
I actually asked you if you already had an idea... should I take that as a
no?
> Thanks for your attention and for reading through this
>>
>> Salvatore
>>
>> [1]
>> http://git.openstack.org/cgit/openstack/neutron/tree/neutron/plugins/vmware/plugin.py#n22
>>
>> On 8 December 2014 at 21:51, Maru Newby <marun at redhat.com> wrote:
>>
>>>
>>> On Dec 7, 2014, at 10:51 AM, Gary Kotton <gkotton at vmware.com> wrote:
>>>
>>> > Hi Kyle,
>>> > I am not missing the point. I understand the proposal. I just think
>>> that it has some shortcomings (unless I misunderstand, which will certainly
>>> not be the first time and most definitely not the last). The thinning out
>>> is to have a shim in place. I understand this and this will be the entry
>>> point for the plugin. I do not have a concern for this. My concern is that
>>> we are not doing this with the ML2 off the bat. That should lead by example
>>> as it is our reference architecture. Lets not kid anyone, but we are going
>>> to hit some problems with the decomposition. I would prefer that it be done
>>> with the default implementation. Why?
>>>
>>> The proposal is to move vendor-specific logic out of the tree to
>>> increase vendor control over such code while decreasing load on reviewers.
>>> ML2 doesn’t contain vendor-specific logic - that’s the province of ML2
>>> drivers - so it is not a good target for the proposed decomposition by
>>> itself.
>>>
>>>
>>> > • Cause we will fix them quicker as it is something that prevent
>>> Neutron from moving forwards
>>> > • We will just need to fix in one place first and not in N
>>> (where N is the vendor plugins)
>>> > • This is a community effort – so we will have a lot more eyes
>>> on it
>>> > • It will provide a reference architecture for all new plugins
>>> that want to be added to the tree
>>> > • It will provide a working example for plugin that are already
>>> in tree and are to be replaced by the shim
>>> > If we really want to do this, we can say freeze all development (which
>>> is just approvals for patches) for a few days so that we will can just
>>> focus on this. I stated what I think should be the process on the review.
>>> For those who do not feel like finding the link:
>>> > • Create a stack forge project for ML2
>>> > • Create the shim in Neutron
>>> > • Update devstack for the to use the two repos and the shim
>>> > When #3 is up and running we switch for that to be the gate. Then we
>>> start a stopwatch on all other plugins.
>>>
>>> As was pointed out on the spec (see Miguel’s comment on r15), the ML2
>>> plugin and the OVS mechanism driver need to remain in the main Neutron repo
>>> for now. Neutron gates on ML2+OVS and landing a breaking change in the
>>> Neutron repo along with its corresponding fix to a separate ML2 repo would
>>> be all but impossible under the current integrated gating scheme.
>>> Plugins/drivers that do not gate Neutron have no such constraint.
>>>
>>>
>>> Maru
>>>
>>>
>>> > Sure, I’ll catch you on IRC tomorrow. I guess that you guys will bash
>>> out the details at the meetup. Sadly I will not be able to attend – so you
>>> will have to delay on the tar and feathers.
>>> > Thanks
>>> > Gary
>>> >
>>> >
>>> > From: "mestery at mestery.com" <mestery at mestery.com>
>>> > Reply-To: OpenStack List <openstack-dev at lists.openstack.org>
>>> > Date: Sunday, December 7, 2014 at 7:19 PM
>>> > To: OpenStack List <openstack-dev at lists.openstack.org>
>>> > Cc: "openstack at lists.openstack.org" <openstack at lists.openstack.org>
>>> > Subject: Re: [openstack-dev] [Neutron] Core/Vendor code decomposition
>>> >
>>> > Gary, you are still miss the point of this proposal. Please see my
>>> comments in review. We are not forcing things out of tree, we are thinning
>>> them. The text you quoted in the review makes that clear. We will look at
>>> further decomposing ML2 post Kilo, but we have to be realistic with what we
>>> can accomplish during Kilo.
>>> >
>>> > Find me on IRC Monday morning and we can discuss further if you still
>>> have questions and concerns.
>>> >
>>> > Thanks!
>>> > Kyle
>>> >
>>> > On Sun, Dec 7, 2014 at 2:08 AM, Gary Kotton <gkotton at vmware.com>
>>> wrote:
>>> >> Hi,
>>> >> I have raised my concerns on the proposal. I think that all plugins
>>> should be treated on an equal footing. My main concern is having the ML2
>>> plugin in tree whilst the others will be moved out of tree will be
>>> problematic. I think that the model will be complete if the ML2 was also
>>> out of tree. This will help crystalize the idea and make sure that the
>>> model works correctly.
>>> >> Thanks
>>> >> Gary
>>> >>
>>> >> From: "Armando M." <armamig at gmail.com>
>>> >> Reply-To: OpenStack List <openstack-dev at lists.openstack.org>
>>> >> Date: Saturday, December 6, 2014 at 1:04 AM
>>> >> To: OpenStack List <openstack-dev at lists.openstack.org>, "
>>> openstack at lists.openstack.org" <openstack at lists.openstack.org>
>>> >> Subject: [openstack-dev] [Neutron] Core/Vendor code decomposition
>>> >>
>>> >> Hi folks,
>>> >>
>>> >> For a few weeks now the Neutron team has worked tirelessly on [1].
>>> >>
>>> >> This initiative stems from the fact that as the project matures,
>>> evolution of processes and contribution guidelines need to evolve with it.
>>> This is to ensure that the project can keep on thriving in order to meet
>>> the needs of an ever growing community.
>>> >>
>>> >> The effort of documenting intentions, and fleshing out the various
>>> details of the proposal is about to reach an end, and we'll soon kick the
>>> tires to put the proposal into practice. Since the spec has grown pretty
>>> big, I'll try to capture the tl;dr below.
>>> >>
>>> >> If you have any comment please do not hesitate to raise them here
>>> and/or reach out to us.
>>> >>
>>> >> tl;dr >>>
>>> >>
>>> >> From the Kilo release, we'll initiate a set of steps to change the
>>> following areas:
>>> >> • Code structure: every plugin or driver that exists or wants to
>>> exist as part of Neutron project is decomposed in an slim vendor
>>> integration (which lives in the Neutron repo), plus a bulkier vendor
>>> library (which lives in an independent publicly available repo);
>>> >> • Contribution process: this extends to the following aspects:
>>> >> • Design and Development: the process is largely
>>> unchanged for the part that pertains the vendor integration; the maintainer
>>> team is fully auto governed for the design and development of the vendor
>>> library;
>>> >> • Testing and Continuous Integration: maintainers will
>>> be required to support their vendor integration with 3rd CI testing; the
>>> requirements for 3rd CI testing are largely unchanged;
>>> >> • Defect management: the process is largely unchanged,
>>> issues affecting the vendor library can be tracked with whichever
>>> tool/process the maintainer see fit. In cases where vendor library fixes
>>> need to be reflected in the vendor integration, the usual OpenStack defect
>>> management apply.
>>> >> • Documentation: there will be some changes to the way
>>> plugins and drivers are documented with the intention of promoting
>>> discoverability of the integrated solutions.
>>> >> • Adoption and transition plan: we strongly advise maintainers
>>> to stay abreast of the developments of this effort, as their code, their
>>> CI, etc will be affected. The core team will provide guidelines and support
>>> throughout this cycle the ensure a smooth transition.
>>> >> To learn more, please refer to [1].
>>> >>
>>> >> Many thanks,
>>> >> Armando
>>> >>
>>> >> [1] https://review.openstack.org/#/c/134680
>>> >>
>>> >> _______________________________________________
>>> >> OpenStack-dev mailing list
>>> >> OpenStack-dev at lists.openstack.org
>>> >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>> >>
>>> >
>>> > _______________________________________________
>>> > OpenStack-dev mailing list
>>> > OpenStack-dev at lists.openstack.org
>>> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>
>>>
>>> _______________________________________________
>>> OpenStack-dev mailing list
>>> OpenStack-dev at lists.openstack.org
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>
>>
>>
>> _______________________________________________
>> OpenStack-dev mailing list
>> OpenStack-dev at lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>>
>
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20141209/08f6898c/attachment.html>
More information about the OpenStack-dev
mailing list