Open Stack

Hello Ihar,

I have some comments and questions about your proposal.  My apologies if
any of what I say here results from misunderstandings on my part.

1. I believe there are two sorts of redirection at play here.  The first
involves inter-table traversal while the second allows a frame to exit the
OF pipeline either by being sent to a different port or by being dropped.
Some of what I say next makes use of this distinction.

2. OpenFlow¹s Goto instruction directs a frame from one table to the next.
 A redirection in this sense must be to a higher-numbered table, which is
to say that OF pipeline processing can only go forward (see p.18, para.2
of the 1.4.1 spec 
<https://www.opennetworking.org/images/stories/downloads/sdn-resources/onf-
specifications/openflow/openflow-switch-v1.4.1.pdf>).  However, OvS (at
least v2.0.2) implements a resubmit action, which re-searches another
table‹higher-, lower-, or even same-numbered‹and executes any actions
found there in addition to any subsequent actions in the current flow
entry.  It is by using resubmit that the proposed design could work, as
shown in the ovs-ofctl command posted here
<https://review.openstack.org/#/c/251738/1/neutron/agent/common/ovs_lib.py>
.  (Maybe there are other ways, too.)  The resubmit action is a Nicira
vendor extension that at least at one point, and maybe still, was known to
be implemented only by OvS.  I mention this because I wonder if the
proposed design (and my sample command) calls for flow traversal in a
manner not explicitly supported by OpenFlow and so may not work in future
versions of OvS.

3. Regarding the idea of sorting feature flows by kind: I believe that
what is meant by a 'redirection flow table' is a table that could possibly
remove the frame from OF pipeline processing‹i.e., by forwarding or
dropping it.  Can you correct/confirm?

4. Even though the design promotes playing nice by means of feature flow
kinds, I think that features might nevertheless still step on each others¹
toes due to assumptions made about field content.  I¹m thinking, for
instance, of two features whose in-place frame modifications should be
done in a particular order.  Because of this, I¹m not sure that the
granularity of the proposed design can guarantee feature cooperation.
Maybe it would help to prioritize feature flows as ingress-processing
(that is, the flow should be exercised as early as possible in the
pipeline) versus egress-processing (the opposite) in addition to kind‹or
maybe that is just what  the notion of feature flow kind calls for, at
least in part.  Tied (tangential?) to this is the distinction that
OpenFlow makes between an action list and an action set: the former is a
series of actions that is applied to the frame immediately and in the
order specified in the flow entry; the latter is a proper set of actions
that is applied to the frame only upon its exit from the OF pipeline and
in an order specified by protocol.  (Action set content is modified as the
frame traverses the OF pipeline.)  Should action sets be disallowed?

5. Is it a correct rephrasing of the third bullet of the high-level design
to say: each feature-specific flow entry in table 0 would be triggered
only if the frame's relevant OF metadata has not already been updated as a
result of the frame's previous traversal of the feature table.  I
apologize if I¹m suggesting something here that you didn¹t mean.

Hope this is helpful.
Margaret
--
Margaret Frances
Eng 4, Prodt Dev Engineering



On 12/3/15, 11:29 AM, "Johnston, Nate" <Nate_Johnston at cable.comcast.com>
wrote:

>Ihar,
>
>This is brilliant.  The complexity of doing graceful CRUD on the OVS flow
>table especially when other features are active is so complex, that
>abstracting the management of it into functionality optimized for that
>task is an incredibly good idea, especially for those of us like me who
>are not experts in OVS, and thus have a hard time seeing the edge cases.
>
>Thanks very much for this.
>
>‹N.
>
>> On Dec 3, 2015, at 10:46 AM, Ihar Hrachyshka <ihrachys at redhat.com>
>>wrote:
>> 
>> Hi,
>> 
>> Small update on the RFE. It was approved for Mitaka, assuming we come
>>up with proper details upfront thru neutron-specs process.
>> 
>> In the meantime, we have found more use cases for flow management among
>>features in development: QoS DSCP, also the new OF based firewall
>>driver. Both authors for those new features independently realized that
>>agent does not currently play nice with flows set by external code due
>>to its graceful restart behaviour when rules with unknown cookies are
>>cleaned up. [The agent uses a random session uuid() to mark rules that
>>belong to its current run.]
>> 
>> Before I proceed, full disclosure: I know almost nothing about OpenFlow
>>capabilities, so some pieces below may make no sense. I tried to come up
>>with high level model first and then try to map it to available OF
>>features. Please don¹t hesitate to comment, I like to learn new stuff! ;)
>> 
>> I am thinking lately on the use cases we collected so far. One common
>>need for all features that were seen to be interested in proper
>>integration with Open vSwitch agent is to be able to manage feature
>>specific flows on br-int and br-tun. There are other things that
>>projects may need, like patch ports, though I am still struggling with
>>the question of whether it may be postponed or avoided for phase 1.
>> 
>> There are several specific operation 'kinds' that we should cover for
>>the RFE:
>> - managing flows that modify frames in-place;
>> - managing flows that redirect frames.
>> 
>> There are some things that should be considered to make features
>>cooperate with the agent and other extensions:
>> - feature flows should have proper priorities based on their Œkind¹
>>(f.e. in-place modification probably go before redirections);
>> - feature flows should survive flow reset that may be triggered by the
>>agent;
>> - feature flows should survive flow reset without data plane disruption
>>(=they should support graceful restart:
>>https://review.openstack.org/#/c/182920).
>> 
>> With that in mind, I see the following high level design for the flow
>>tables:
>> 
>> - table 0 serves as a dispatcher for specific features;
>> - each feature gets one or more tables, one per flow Œkind¹ needed;
>> - for each feature table, a new flow entry is added to table 0 that
>>would redirect to feature specific table; the rule will be triggered
>>only if OF metadata is not updated inside the feature table (see the
>>next bullet); the rule will have priority that is defined for the Œkind¹
>>of the operation that is implemented by the table it redirects to;
>> -  each feature table will have default actions that will 1) mark OF
>>metadata for the frame as processed by the feature; 2) redirect back to
>>table 0;
>> - all feature specific flow rules (except dispatcher rules) belong to
>>feature tables;
>> 
>> Now, the workflow for extensions that are interested in setting flows
>>would be:
>> - on initialize() call, extension defines feature tables it will need;
>>it passes the name of the feature table and the Œkind¹ of the actions it
>>will execute; with that, the following is initialized by the agent: 1)
>>table 0 dispatcher entry to redirect frames into feature table; the
>>entry has the priority according to the Œkind¹ of the table; 2) the
>>actual feature table with two default rules (update metadata and push
>>back to table 0);
>> - whenever extension needs to add a new flow rule, it passes the
>>following into the agent: 1) table name; 2) flow specific parameters
>>(actions, priority, ...)
>> 
>> Since the agent will manage setting flows for extensions, it will be
>>able to use the active agent cookie for all feature flows; next time the
>>agent is restarted, it should be able to respin extension flows with no
>>data plane disruption. [Note: we should make sure that on agent restart,
>>we call to extensions *before* we clean up stale flow rules.]
>> 
>> That design will hopefully allow us to abstract interaction with flows
>>from extensions into management code inside the agent. It should
>>guarantee extensions cooperate properly assuming they properly define
>>their priorities thru Œkinds¹ of tables they have.
>> 
>> It is also assumed that existing flow based features integrated into
>>the agent (dvr? anti-spoofing?) will eventually move to the new flow
>>table management model.
>> 
>> I understand that the model does not reflect how do feature processing
>>for existing OF based features in the agent. It may require some smart
>>workarounds to allow non-disruptive migration to new flow table setup.
>> 
>> It would be great to see the design bashed hard before I start to put
>>it into spec format. Especially if it¹s not sane. :)
>> 
>> Ihar
>> 
>> Mathieu Rohon <mathieu.rohon at gmail.com> wrote:
>> 
>>> Thanks ihar!
>>> 
>>> On Thu, Nov 19, 2015 at 2:32 PM, Ihar Hrachyshka <ihrachys at redhat.com>
>>>wrote:
>>> UPD: now that we have some understanding what¹s needed from l2 agent
>>>extension mechanism to cater for interested subprojects (and now that
>>>we see that probably the agent in focus right now is OVS only), we may
>>>move to RFE step. I reported the following RFE for the feature:
>>> 
>>> https://bugs.launchpad.net/neutron/+bug/1517903
>>> 
>>> It may require BP if drivers team will request one.
>>> 
>>> Cheers,
>>> 
>>> Ihar
>>> 
>>> Ihar Hrachyshka <ihrachys at redhat.com> wrote:
>>> 
>>> Reviving the thread.
>>> 
>>> On the design summit session dedicated to agent and plugin extensions
>>>[1] the following was stated for l2 agent extensions (I appreciate if
>>>someone checks me on the following though):
>>> 
>>> - current l2 agent extensions mechanism lacks insight into agent
>>>details like bridges or vlan maps;
>>> 
>>> - in some cases, we don¹t care about extension portability across
>>>multiple agents, so it¹s not of concern if some of them use
>>>implementation details like bridges to set specific flows, or to wire
>>>up some additional ports to them;
>>> 
>>> - that said, we still don¹t want extensions to have unlimited access
>>>to agent details; the rationale for hard constraints on what is seen
>>>inside extensions is that we cannot support backwards compatibility for
>>>*all* possible internal attributes of an agent; instead, we should
>>>explicitly define where we can make an effort to provide stable API
>>>into agent details, and what¹s, on contrary, beyond real life use cases
>>>and hence can be left to be broken/refactored as neutron developers see
>>>fit; this API can be agent specific though;
>>> 
>>> - agent details that are to be passed into extensions should be driven
>>>by actual use cases. There were several subprojects mentioned in the
>>>session that are assumed to lack enough access to agent attributes to
>>>do their job without patching core ovs agent files. Those are: BGP-VPN,
>>>SFC, (anything else?) Those subprojects that are interested in
>>>extending l2 agent extension framework are expected to come up with a
>>>list of things missing in current implementation, so that neutron
>>>developers can agree on proper abstractions to provide missing details
>>>to extensions. For that goal, I set up a new etherpad to collect
>>>feedback from subprojects [2].
>>> 
>>> Once we collect use cases there and agree on agent API for extensions
>>>(even if per agent type), we will implement it and define as stable
>>>API, then pass objects that implement the API into extensions thru
>>>extension manager. If extensions support multiple agent types, they can
>>>still distinguish between which API to use based on agent type string
>>>passed into extension manager.
>>> 
>>> I really hope we start to collect use cases early so that we have time
>>>to polish agent API and make it part of l2 extensions earlier in Mitaka
>>>cycle.
>>> 
>>> [1]: https://etherpad.openstack.org/p/mitaka-neutron-core-extensibility
>>> [2]: https://etherpad.openstack.org/p/l2-agent-extensions-api-expansion
>>> 
>>> Ihar
>>> 
>>> Ihar Hrachyshka <ihrachys at redhat.com> wrote:
>>> 
>>> On 30 Sep 2015, at 12:53, Miguel Angel Ajo <mangelajo at redhat.com>
>>>wrote:
>>> 
>>> 
>>> 
>>> Ihar Hrachyshka wrote:
>>> On 30 Sep 2015, at 12:08, thomas.morin at orange.com wrote:
>>> 
>>> Hi Ihar,
>>> 
>>> Ihar Hrachyshka :
>>> Miguel Angel Ajo :
>>> Do you have a rough idea of what operations you may need to do?
>>> Right now, what bagpipe driver for networking-bgpvpn needs to interact
>>>with is:
>>> - int_br OVSBridge (read-only)
>>> - tun_br OVSBridge (add patch port, add flows)
>>> - patch_int_ofport port number (read-only)
>>> - local_vlan_map dict (read-only)
>>> - setup_entry_for_arp_reply method (called to add static ARP entries)
>>> Sounds very tightly coupled to OVS agent.
>>> Please bear in mind, the extension interface will be available from
>>>different agent types
>>> (OVS, SR-IOV, [eventually LB]), so this interface you're talking about
>>>could also serve as
>>> a translation driver for the agents (where the translation is
>>>possible), I totally understand
>>> that most extensions are specific agent bound, and we must be able to
>>>identify
>>> the agent we're serving back exactly.
>>> Yes, I do have this in mind, but what we've identified for now seems
>>>to be OVS specific.
>>> Indeed it does. Maybe you can try to define the needed pieces in high
>>>level actions, not internal objects you need to access to. Like Œ-
>>>connect endpoint X to Y¹, Œdetermine segmentation id for a network¹ etc.
>>> I've been thinking about this, but would tend to reach the conclusion
>>>that the things we need to interact with are pretty hard to abstract
>>>out into something that would be generic across different agents.
>>>Everything we need to do in our case relates to how the agents use
>>>bridges and represent networks internally: linuxbridge has one bridge
>>>per Network, while OVS has a limited number of bridges playing
>>>different roles for all networks with internal segmentation.
>>> 
>>> To look at the two things you  mention:
>>> - "connect endpoint X to Y" : what we need to do is redirect the
>>>traffic destinated to the gateway of a Neutron network, to the thing
>>>that will do the MPLS forwarding for the right BGP VPN context (called
>>>VRF), in our case br-mpls (that could be done with an OVS table too) ;
>>>that action might be abstracted out to hide the details specific to
>>>OVS, but I'm not sure on how to  name the destination in a way that
>>>would be agnostic to these details, and this is not really relevant to
>>>do until we have a relevant context in which the linuxbridge would pass
>>>packets to something doing MPLS forwarding (OVS is currently the only
>>>option we support for MPLS forwarding, and it does not really make
>>>sense to mix linuxbridge for Neutron L2/L3 and OVS for MPLS)
>>> - "determine segmentation id for a network": this is something really
>>>OVS-agent-specific, the linuxbridge agent uses multiple linux bridges,
>>>and does not rely on internal segmentation
>>> 
>>> Completely abstracting out packet forwarding pipelines in OVS and
>>>linuxbridge agents would possibly allow defining an interface that
>>>agent extension could use without to know about anything specific to
>>>OVS or the linuxbridge, but I believe this is a very significant taks
>>>to tackle.
>>> 
>>> If you look for a clean way to integrate with reference agents, then
>>>it¹s something that we should try to achieve. I agree it¹s not an easy
>>>thing.
>>> 
>>> Just an idea: can we have a resource for traffic forwarding, similar
>>>to security groups? I know folks are not ok with extending security
>>>groups API due to compatibility reasons, so maybe fwaas is the place to
>>>experiment with it.
>>> 
>>> Hopefully it will be acceptable to create an interface, even it
>>>exposes a set of methods specific to the linuxbridge agent and a set of
>>>methods specific to the OVS agent.  That would mean that the agent
>>>extension that can work in both contexts (not our case yet) would check
>>>the agent type before using the first set or the second set.
>>> 
>>> The assumption of the whole idea of l2 agent extensions is that they
>>>are agent agnostic. In case of QoS, we implemented a common QoS
>>>extension that can be plugged in any agent [1], and a set of backend
>>>drivers (atm it¹s just sr-iov [2] and ovs [3]) that are selected based
>>>on the driver type argument passed into the extension manager [4][5].
>>>Your extension could use similar approach to select the backend.
>>> 
>>> [1]: 
>>>https://git.openstack.org/cgit/openstack/neutron/tree/neutron/agent/l2/e
>>>xtensions/qos.py#n169
>>> [2]: 
>>>https://git.openstack.org/cgit/openstack/neutron/tree/neutron/plugins/ml
>>>2/drivers/mech_sriov/agent/extension_drivers/qos_driver.py
>>> [3]: 
>>>https://git.openstack.org/cgit/openstack/neutron/tree/neutron/plugins/ml
>>>2/drivers/openvswitch/agent/extension_drivers/qos_driver.py
>>> [4]: 
>>>https://git.openstack.org/cgit/openstack/neutron/tree/neutron/plugins/ml
>>>2/drivers/openvswitch/agent/ovs_neutron_agent.py#n395
>>> [5]: 
>>>https://git.openstack.org/cgit/openstack/neutron/tree/neutron/plugins/ml
>>>2/drivers/mech_sriov/agent/sriov_nic_agent.py#n155
>>> 
>>> I disagree on the agent-agnostic thing. QoS extension for SR-IOV is
>>>totally not agnostic for OVS or LB, in the QoS case, it's just
>>> accidental that OVS & LB share common bridges now due to the OVS
>>>Hybrid implementation that leverages linux bridge
>>> and iptables.
>>> 
>>> Wait. The QoS extension has nothing agent backend specific. All it
>>>does is it receives rpc updates for tracked resources and pass them
>>>into qos drivers. Those latter are the bits that implement backend
>>>specific operations. So I am not sure why you say the extension itself
>>>is agent specific: any other amqp based agent in the wild can adopt the
>>>extension as-is, only providing a new backend to load.
>>> 
>>> I agree on having a well defined interface, on which API is available
>>>to talking back to each agent, and it has to be common, where
>>> it's possible to be common.
>>> 
>>> It doesn't have to be easy, but it's the way if we want a world where
>>>those commonalities and reusability of extensions can
>>> exist and not be just accidental, but it's not realistic in my opinion
>>>to AIM for it on every shot. I believe we should try where we can
>>> but we should be open to agent specific extensions. The idea of the
>>>extensions is that you can extend specific agents without
>>> being forced to have the main loop hijacked, or eventually having off
>>>tree code plugged into our agents.
>>> 
>>> Partially, yes. The culprit here is how much the extension API should
>>>know about an agent. We can probably make the extension API completely
>>>extendable by allowing agents to pass any random kwargs into the
>>>extension manager that will forward them into extensions. Note that it
>>>breaks current API for extensions and technically breaks it (not that I
>>>know of any external extensions that could be affected so far).
>>> 
>>> There we should add support to identify the type of agent the
>>>extension works with (compatibility, versioning, etc..)
>>> 
>>> We already pass the type into extension manager, and that¹s how we
>>>plug in the proper backend driver in QoS.
>>> 
>>> Does this approach make sense ?
>>> 
>>> -Thomas
>>> 
>>> 
>>>________________________________________________________________________
>>>_________________________________________________
>>> 
>>> Ce message et ses pieces jointes peuvent contenir des informations
>>>confidentielles ou privilegiees et ne doivent donc
>>> pas etre diffuses, exploites ou copies sans autorisation. Si vous avez
>>>recu ce message par erreur, veuillez le signaler
>>> a l'expediteur et le detruire ainsi que les pieces jointes. Les
>>>messages electroniques etant susceptibles d'alteration,
>>> Orange decline toute responsabilite si ce message a ete altere,
>>>deforme ou falsifie. Merci.
>>> 
>>> This message and its attachments may contain confidential or
>>>privileged information that may be protected by law;
>>> they should not be distributed, used or copied without authorisation.
>>> If you have received this email in error, please notify the sender and
>>>delete this message and its attachments.
>>> As emails may be altered, Orange is not liable for messages that have
>>>been modified, changed or falsified.
>>> Thank you.
>>> 
>>> Note that you should really avoid putting that ^^ kind of signature
>>>into your emails intended for public mailing lists. If it¹s
>>>confidential, why do you send it to everyone? And sorry, folks will
>>>copy it without authorisation, for archiving and indexing reasons and
>>>whatnot.
>>> 
>>> Ihar
>>> 
>>>________________________________________________________________________
>>>__
>>> OpenStack Development Mailing List (not for usage questions)
>>> Unsubscribe: 
>>>OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>> 
>>> 
>>> 
>>>________________________________________________________________________
>>>__
>>> OpenStack Development Mailing List (not for usage questions)
>>> Unsubscribe: 
>>>OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>> 
>>> 
>>>________________________________________________________________________
>>>__
>>> OpenStack Development Mailing List (not for usage questions)
>>> Unsubscribe: 
>>>OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>> 
>>> 
>>>________________________________________________________________________
>>>__
>>> OpenStack Development Mailing List (not for usage questions)
>>> Unsubscribe: 
>>>OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>> 
>>> 
>>> 
>>>________________________________________________________________________
>>>__
>>> OpenStack Development Mailing List (not for usage questions)
>>> Unsubscribe: 
>>>OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>> 
>>> 
>>>________________________________________________________________________
>>>__
>>> OpenStack Development Mailing List (not for usage questions)
>>> Unsubscribe: 
>>>OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>> 
>> 
>> 
>>_________________________________________________________________________
>>_
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe: 
>>OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>__________________________________________________________________________
>OpenStack Development Mailing List (not for usage questions)
>Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev