Open Stack

Hi,

    Sorry for the late reply. Some comments on Maru's and Amando's points
below:

(1) this model is more complex than linked based

--- as one of the three original members who had action items during the
very first network-policy IRC meeting, I can say that we actually
originally went with the linked-based model. However, as we tried to verify
the model with use cases, we found that while the linked-based is more
intuitive for us infrastructure people (especially networking, as it is
natural for us to apply policy to traffic from src -> dst groups), the
group-based model is more intuitive for app-owners (who deal with
"app-boundaries").

    The ultimate validation (for me at least) came after our conference
presentation where I had a discussion with an operator (from a carrier,
thus "apps" are likely network functions), and he likes group-policy on two
fronts: (a) separation of control with hierarchical contracts, and (b) very
intuitive for app deployment standpoint (wrap the app, or app-tier, with a
policy).

    In terms of implementation complexity - without the optional features
such as label (which we aren't going to add during Juno), I don't think
there is much difference between the two models (APIs, number of new
resources...etc). Mohammad and I did some db and API code initially with
the old model, and the code complexity isn't that much different from what
we have today (sans the label related constructs).

(2) we should iterate code and make them available on gerrit for review as
we have digestible and functional chunks

--- absolutely agreed. And we have reached that agreement during the IRC
meeting yesterday [1]. Particularly for the mapping driver, both Bob
(Kukura) and I reported (also during the IRC meeting[1]) there are still a
lot of work before it is ready. And as we refactor and develop the mapping
driver, we will certainly make it available on gerrit in functional pieces
(along with unit tests).

(3) the mapping driver should utilize Neutron REST API, or more abstract
constructs

--- this is an extremely constructive feedback from Amando. We will
definitely keep this in mind as we refactor and continue development on the
mapping driver. Thanks for that feedback.

(4) why is this project High-Priority?

--- I don't have enough visibility into how Neutron projects are classified
in terms of priority. That said, the effort in general has been on-going
since around H-3 timeframe, and more than 10 developers from seven
different companies have invested to make group-policy what it is today.
Furthermore, from last week's summit, a number of operators and OpenStack
users have shown interest for this feature in OpenStack. So we collectively
as community members would like to plead to the PTL and the rest of the
core team to take these into consideration when making decision on the
relative priority of this project.

Thanks,
- Stephen



[1]
http://eavesdrop.openstack.org/meetings/networking_policy/2014/networking_policy.2014-05-22-18.01.log.html


On Fri, May 23, 2014 at 12:31 AM, Prasad Vellanki <
prasad.vellanki at oneconvergence.com> wrote:

> Great to see the discussions on the ML.
> Mohammad - Good summary.
>
> I would like to make few  points
> 1) The current GP API is tuned towards person deploying the application as
> opposed to the networking person. This is probably a better way as one
> starts to think about self service infrastructure model and person
> deploying applications. We had debated about this on IRC and thought that
> policies/contract applied to the end point group would be easier
> abstraction. The complexity of networking is transformed to the
> implementation while still retaining the flexibility for network admin to
> provide additional policies.
> 2) I agree with mohammad that the feedback received for policy driven
> model was very positive. I heard in one of the BOF sessions a large
> operator commenting on this specifically in a very positive way. It would
> be sad to see this relegated to lesser priority. I am not saying that we
> should not address other issues in neutron but this should be addressed
> with equal priority. Especially when a lot of thought and PoC is already
> being done.
> 3) It was to good see consensus on the IRC today about the process
> particularly around code simplification, checkins and reviews as raised in
> the emails.
>
> thanks
> prasadv
>
>
> On Thu, May 22, 2014 at 9:10 PM, Mohammad Banikazemi <mb at us.ibm.com>wrote:
>
>> Thanks to everyone who participated in the Group Policy meeting [1]
>> earlier today. A lot of good discussion that hopefully will continue with
>> participation from the larger community. I wanted to first make a comment
>> about how the Group Policy work was received outside the Neutron community
>> and then focus on finding a way for us to make progress.
>>
>> I think we had a really good feedback from the larger OpenStack community
>> and I would say a wide support for addition of policy abstractions to
>> Neutron. If the feedback we received at the summit in Hong Kong was mostly
>> positive, in Atlanta the support was overwhelmingly positive in my opinion.
>> I just wanted to make sure this does not get lost in our discussions.
>>
>> Needless to say, we will work on a path to have the Group Policy work
>> included in Neutron in a way that keeps the quality of code in Neutron
>> preserved. To rephrase what Armando said in the IRC meeting earlier today,
>> we all share a common goal and that is to do what's right. I think it
>> may be beneficial that for the moment and for the next few days as we try
>> to find the best path forward, we forget about any particular
>> cycle/milestone/priority and look for the best path forward and see where
>> that leads us. To this end, the group policy team will be setting up a
>> meeting with Armando (and others who are interested) to in particular
>> discuss the possibility of making the code less tightly coupled with
>> Neutron core. We will also consider how we can address Marun's and Mark's
>> concerns by trying to have a simpler but functional set of patches that can
>> be reviewed more effectively such that we can build on them in an iterative
>> manner.
>>
>> Meanwhile, please do continue the discussion on the mailing list.
>>
>> Best,
>>
>> Mohammad
>>
>> [1] https://wiki.openstack.org/wiki/Meetings/Neutron_Group_Policy
>>
>> [image: Inactive hide details for "Armando M." ---05/22/2014 11:24:35
>> PM---On 22 May 2014 13:59, Mandeep Dhami <dhami at noironetworks.com]"Armando
>> M." ---05/22/2014 11:24:35 PM---On 22 May 2014 13:59, Mandeep Dhami <
>> dhami at noironetworks.com> wrote: >
>>
>> From: "Armando M." <armamig at gmail.com>
>> To: "OpenStack Development Mailing List (not for usage questions)" <
>> openstack-dev at lists.openstack.org>,
>> Date: 05/22/2014 11:24 PM
>> Subject: Re: [openstack-dev] [neutron][group-based-policy] Should we
>> revisit the priority of group-based policy?
>> ------------------------------
>>
>>
>>
>> On 22 May 2014 13:59, Mandeep Dhami <dhami at noironetworks.com> wrote:
>> >
>> > Maru's concerns are that:
>> > 1. It is large
>> > 2. It is complex
>> >
>> > And Armando's related concerns are:
>> > 3. Could dev/review cycles be better spent on refactoring
>> > 4. If refactored neutron was available, would a simpler option become
>> more
>> > viable
>>
>> This is not what I meant to say, and if this was the message that came
>> across I apologize for the confusion; let me rephrase:
>>
>> After looking (and relooking) at the initial patches proposed I
>> started to question why the GP plugin functionality was so tightly
>> integrated with the Neutron core functionality; even though I might
>> guess the thinking process, I wonder if such tight coupling was the
>> result of design decisions made without thoroughly considering
>> alternative approaches. Without going too much into details during
>> this email, I can see in the above mentioned patches that lots of
>> plumbing code (like Nova and dhcp notifiers handling code) is put in
>> place to make direct calls to core plugin methods: this spills
>> implementation details across multiple parts of the project; it's
>> fragile because it's prone to ripple effects due to lack of proper
>> encapsulation: if a change is made in the plugin API or its
>> implementation, the whole thing needs to be looked at, end-to-end:
>> this does not scale from a human perspective (probably only a handful
>> of people can really say that they know the Neutron codebase
>> inside-out), it is difficult to maintain, it is difficult to test, it
>> is difficult to extend. etc etc.
>>
>> Instead, I was advocating for an approach where GP and Neutron Core
>> integrate via (a well defined and stable) REST API, or similar (more
>> abstracted) mechanisms; this has obvious benefits because the two
>> become suddenly loosely coupled: a change done in the way Neutron
>> deals with DHCP messages is not going to have any effect to how the GP
>> plugin create resources. Also, any potential refactoring of the
>> Neutron Core will not cause the GP team to take the burden of bringing
>> the current implementation forward.
>>
>> This is why I was proposing that we talk about the introduction of
>> integration hooks, should they (or lack thereof) have been the culprit
>> of such an initial design approach. Please, take my comments as
>> initial reviews to the above patches, if you will :)
>>
>> To be constructive, as a core reviewer who should suggest
>> alternatives, I would invite the people reading this thread to have a
>> look at [1] and [2]: these were introduced by RAX to their cut of
>> Neutron, having in mind exactly what I have been saying: adding
>> functionality with zero impact to existing code. If something along
>> those lines can be achieved, then this would be very beneficial for
>> the progress of the GP effort as it transitions and evolves
>> into/within Neutron, IMO.
>>
>> Having said that, I am making these points without particular
>> reference to the complexity of the GP model being proposed, or the
>> approach being taken to introduce it to the tree. Even though I share
>> some of Maru's points, good architecture and design principles in
>> software development should be followed wherever possible and
>> irrespective of the domain where such development occur.
>>
>> Many thanks,
>> Armando
>>
>>
>> [1] - https://github.com/roaet/wafflehaus
>> [2] - https://github.com/roaet/wafflehaus.neutron
>>
>> >
>> > Let me address them in that order.
>> >
>> > 1. Re: It is large
>> > Group policy has an ambitious goal  - provide devop teams with policy
>> based
>> > controls that are usable at scale and with automation (say a higher
>> > governance layer like Congress). The fact that meeting a large challenge
>> > requires more code is natural. We understand that challenge, and that
>> is why
>> > we did a prototype (as PoC that was demonstrated on the summit). And
>> based
>> > on that learning we are incrementally creating patches for building the
>> > group based policy. Just because a task is large, we as neutron can not
>> shy
>> > away from building it. That will only drive people who need it out side
>> > neutron (as we are seeing with the frustration that the LBaaS team had
>> > because they have a requirement that is "large" as well).
>> >
>> > 2. Re: It is complex
>> > Complexity depends on the context. Our goal was to make the end-user's
>> life
>> > simpler (and more automated). To achieve some of that simplicity, we
>> > required a little more complexity in the implementation. We decide to
>> make
>> > that arbitrage - a little higher complexity in implementation to allow
>> for
>> > simpler usage. But we were careful and did not want to impose that
>> > complexity on every use case - hence a lot of that is optional (and
>> > exercised only if the use case needs it). Unfortunately the model, has
>> to
>> > model all of it so as it not add complexity later in upgrade and
>> backward
>> > compatibility issues. We choose to do architecture upfront, and then
>> > implement it incrementally.
>> >
>> > The team came up with the model currently in model based on that review
>> and
>> > evaluation all the proposals in the document that you refer. It is easy
>> to
>> > make general comments, but unless you participate in the process and
>> sign up
>> > to writing the code, those comments are not going to help with solving
>> the
>> > original problem. And this _is_ open-source. If you disagree, please
>> write
>> > code and the community can decide for itself as to what model is
>> actually
>> > simple to use for them. Curtailing efforts from other developers just
>> > because their engineering trade-offs are different from what you believe
>> > your use-case needs is not why we like open source. We enjoy the mode
>> where
>> > different developers try different things, we experiment, and the
>> software
>> > evolves to what the user demands. Or maybe, multiple models live in
>> harmony.
>> > Let the users decide that.
>> >
>> > 3. Re: Could dev/review cycles be better spent on refactoring
>> > I think that most people agree that policy control is an important
>> feature
>> > that fundamentally improves neutron (by solving the automation and scale
>> > issues). In a large project, multiple sub-projects can, and for a
>> healthy
>> > project should, work in parallel. I understand that the neutron core
>> team is
>> > stretched. But we still need to be able to balance the needs of today
>> > (paying off the technical debt/existing-issues by doing refactoring)
>> with
>> > needs of tomorrow (new features like GP and LBaaS). GP effort was
>> started in
>> > Havana, and now we are trying to get this in Juno. I think that is
>> > reasonable and a long enough cycle for a "high priority" project to be
>> able
>> > to get some core attention. Again I refer to LBaaS experience, as they
>> > struggled with very similar issues.
>> >
>> > 4. Re: If refactored neutron was available, would a simpler option
>> become
>> > more viable
>> > We would love to be able to answer that question. We have been trying to
>> > understand the refactoring work to understand this (see another ML
>> thread)
>> > and we are open to understanding your position on that. We will call the
>> > ad-hoc meeting that you suggested and we would like to understand the
>> > refactoring work that might be reused for simpler policy
>> implementation. At
>> > the same time, we would like to build on what is available today, and
>> when
>> > the required refactored neutron becomes available (say Juno or
>> K-release),
>> > we are more than happy to adapt to it at that time. Serializing all
>> > development around an effort that is still in inception phase is not a
>> good
>> > solution. We are looking forward to participating in the core
>> refactoring
>> > work, and based on the final spec that come up with, we would love to be
>> > able to eventually make the policy implementation simpler.
>> >
>> > Regards,
>> > Mandeep
>> >
>> >
>> >
>> >
>> > On Thu, May 22, 2014 at 11:44 AM, Armando M. <armamig at gmail.com> wrote:
>> >>
>> >> I would second Maru's concerns, and I would also like to add the
>> >> following:
>> >>
>> >> We need to acknowledge the fact that there are certain architectural
>> >> aspects of Neutron as a project that need to be addressed; at the
>> >> summit we talked about the core refactoring, a task oriented API, etc.
>> >> To me these items have been neglected far too much over the past and
>> >> would need a higher priority and a lot more attention during the Juno
>> >> cycle. Being stretched as we are I wonder if dev/review cycles
>> >> wouldn't be better spent devoting more time to these efforts rather
>> >> than GP.
>> >>
>> >> That said, I appreciate that GP is important and needs to move
>> >> forward, but at the same time I am thinking that there must be a
>> >> better way for addressing it and yet relieve some of the pressure that
>> >> GP complexity imposes to the Neutron team. One aspect it was discussed
>> >> at the summit was that the type of approach shown in [2] and [3]
>> >> below, was chosen because of lack of proper integration hooks...so I
>> >> am advocating: let's talk about those first before ruling them out in
>> >> favor of a monolithic approach that seems to violate some engineering
>> >> principles, like modularity and loose decoupling of system components.
>> >>
>> >> I think we didn't have enough time during the summit to iron out some
>> >> of the concerns voiced here, and it seems like the IRC meeting for
>> >> Group Policy would not be the right venue to try and establish a
>> >> common ground among the people driving this effort and the rest of the
>> >> core team.
>> >>
>> >> Shall we try and have an ad-hoc meeting and an ad-hoc agenda to find a
>> >> consensus?
>> >>
>> >> Many thanks,
>> >> Armando
>> >>
>> >> On 22 May 2014 11:38, Maru Newby <marun at redhat.com> wrote:
>> >> >
>> >> > On May 22, 2014, at 11:03 AM, Maru Newby <marun at redhat.com> wrote:
>> >> >
>> >> >> At the summit session last week for group-based policy, there were
>> many
>> >> >> concerns voiced about the approach being undertaken.  I think those
>> concerns
>> >> >> deserve a wider audience, and I'm going to highlight some of them
>> here.
>> >> >>
>> >> >> The primary concern seemed to be related to the complexity of the
>> >> >> approach implemented for the POC.  A number of session participants
>> voiced
>> >> >> concern that the simpler approach documented in the original
>> proposal [1]
>> >> >> (described in the section titled 'Policies applied between groups')
>> had not
>> >> >> been implemented in addition to or instead of what appeared in the
>> POC
>> >> >> (described in the section titled 'Policies applied as a group
>> API').  The
>> >> >> simpler approach was considered by those participants as having the
>> >> >> advantage of clarity and immediate usefulness, whereas the complex
>> approach
>> >> >> was deemed hard to understand and without immediate utility.
>> >> >>
>> >> >> A secondary but no less important concern is related to the impact
>> on
>> >> >> Neutron of the approach implemented in the POC.  The POC was
>> developed
>> >> >> monolithically, without oversight through gerrit, and the resulting
>> patches
>> >> >> were excessive in size (~4700 [2] and ~1500 [3] lines).  Such large
>> patches
>> >> >> are effectively impossible to review.  Even broken down into
>> reviewable
>> >> >> chunks, though, it does not seem realistic to target juno-1 for
>> merging this
>> >> >> kind of complexity.  The impact on stability could be considerable,
>> and it
>> >> >> is questionable whether the necessary review effort should be
>> devoted to
>> >> >> fast-tracking group-based policy at all, let alone an approach that
>> is
>> >> >> considered by many to be unnecessarily complicated.
>> >> >>
>> >> >> The blueprint for group policy [4] is currently listed as a 'High'
>> >> >> priority.  With the above concerns in mind, does it make sense to
>> continue
>> >> >> prioritizing an effort that at present would seem to require
>> considerably
>> >> >> more resources than the benefit it appears to promise?
>> >> >>
>> >> >>
>> >> >> Maru
>> >> >>
>> >> >> 1: https://etherpad.openstack.org/p/group-based-policy
>> >> >
>> >> > Apologies, this link is to the summit session etherpad.  The link to
>> the
>> >> > original proposal is:
>> >> >
>> >> >
>> >> >
>> https://docs.google.com/document/d/1ZbOFxAoibZbJmDWx1oOrOsDcov6Cuom5aaBIrupCD9E/edit
>> >> >
>> >> >> 2: https://review.openstack.org/93853
>> >> >> 3: https://review.openstack.org/93935
>> >> >> 4:
>> >> >>
>> https://blueprints.launchpad.net/neutron/+spec/group-based-policy-abstraction
>> >> >>
>> >> >> _______________________________________________
>> >> >> OpenStack-dev mailing list
>> >> >> OpenStack-dev at lists.openstack.org
>> >> >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>> >> >
>> >> >
>> >> > _______________________________________________
>> >> > OpenStack-dev mailing list
>> >> > OpenStack-dev at lists.openstack.org
>> >> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>> >>
>> >> _______________________________________________
>> >> OpenStack-dev mailing list
>> >> OpenStack-dev at lists.openstack.org
>> >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>> >
>> >
>> >
>> > _______________________________________________
>> > OpenStack-dev mailing list
>> > OpenStack-dev at lists.openstack.org
>> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>> >
>>
>> _______________________________________________
>> OpenStack-dev mailing list
>> OpenStack-dev at lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>>
>>
>> _______________________________________________
>> OpenStack-dev mailing list
>> OpenStack-dev at lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>>
>
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20140523/602cfe7d/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: graycol.gif
Type: image/gif
Size: 105 bytes
Desc: not available
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20140523/602cfe7d/attachment.gif>