[ptg][neutron] Ussuri PTG summary
Hi Neutron team, First if all thank to all of You for great and very productive week during the PTG in Shanghai. Below is summary of our discussions from whole 3 days. If I forgot about something, please respond to the email and update missing informations. But if You want to have follow up discussion about one of the topics from this summary, please start a new thread to keep this one only as high level summary of the PTG. On boarding =========== Slides from onboarding session can be found at [1] If You have any follow up questions to us about onboarding, or You need help with starting any work in Neutron team, please contact me or Miguel Lavalle by email or on IRC. My IRC nick is slaweq and Miguel's nick is mlavalle. We are available on #openstack-neutron channel @freenode. Train retrospective =================== Good things in Train cycle: * working with this team is still good experience * core team is stable, and we didn't lost any core reviewers during the cycle, * networking is still one of key reasons why people use OpenStack Not good things: * dimished vitality in stadium projects - we had also forum session and follow discussion about this later during the PTG, * gate instability - we have seen many issues which were out of our control, like infra problems, grenade jobs failures, other projects failures, but also many bugs on our side, * we have really a lot of jobs in our check/gate queue. If each of them is failing 5% of times, it's hard to merge any patch as almost every time, one of jobs will fail. Later during the PTG we also discussed that topic and we were looking for some jobs which we maybe can potentially drop from our queues. See below for summary about that, Action items/improvements: * many team meetings each week. We decided to limit number of meetings by: ** consolidate performance subteam meeting into weekly team meeting - this topic will be added to the team meeting's agenda for team meetings on Monday, ** consolidate ovn convergence meeting into weekly team meeting - this topic will be added to the team meeting's agenda for team meetings on Tuesday, ** we need to check if QoS subteam meeting is still needed, * Review process: list of actual review priorities would be useful for the team, we will add "Review-Priority" label to the Neutron reviews board and try to use it during the Ussuri cycle. Openvswitch agent enhancements ============================== We had bunch of topics related to potential improvements for neutron-openvswitch-agent proposed mostly by Liu Yulong. Slides with his proposals are available at [2]. * retire DHCP agent - resyncs of DHCP agent are problematic, especially when agent hosts many networks. Proposal was to add new L2 agent's extension which could be used instead of "regular" DHCP agent and to provide only basic DHCP functionalities. Such solutions would work in the way quite similar to how networking-ovn works today but we would need to implement and maintain own dhcp server application. Problems of this solution are: ** problems with compatibility e.g. with Ironic, ** how it would work with mixed deployments, e.g. with ovs and sriov agents, ** support for dhcp options, Advantages of this solution: ** fully distributed DHCP service, ** no DHCP agents, so less RPC messages on the bus and easier maintanance of the agents, Team's feedback for that is that this is potentially nice solution which may helps in some specific, large scale deploymnets. We can continue discussion about this during Ussuri cycle for sure. * add accepted egress fdb flows We agreed that this is a bug and we should continue work on this to propose some way to fix it. Solution proposed by LIU during this discussion wasn't good as it could potentially break some corner cases. * new API and agent for L2 traffic health check The team asked to add to the spec some more detailed and concrete use cases with explanation how this new API may help operator of the cloud to investigate where the problem actually is. * Local flows cache and batch updating The team agreed that as long as this will be optional solution which operator can opt-in we can give it a try. But spec and discuss details there will be necessary. * stop processing ports twice in ovs-agent We all agreed that this is a bug and should be fixed. But we have to be careful as fixing this bug may cause some other problems e.g. with live-migration - see nova-neutron cross project session. * ovs-agent: batch flow updates with --bundle We all agreed that this can be done as an improvement of existing code. Similar option is already used in openvswitch firewall driver. Neutron - Cyborg cross project session ====================================== Etherpad for the session is at [3]. Cyborg team wants to include Neutron in workflow of spawning VMs with Smart NICs or accelerator cards. From Neutron's side, required change is to allow including "accel" data in port binding profile. As long as this will be well documented what can be placed there, there should be no problem with doing that. Technically we can place almost anything there. Neutron - Kuryr cross project session ===================================== Etherpad for the session is at [4]. Kuryr team proposed 4 improvements for Neutron which would help a lot Kuryr. Ideas are: * Network cascade deletion, * Force subport deletion, * Tag resources at creation time, * Security group creation with rules & bulk security group rule creation All of those ideas makes sense for Neutron team. Tag resources at creation time is even accepted rfe already - see [5] but there was no volunteer to implement it. We will add it to list of our BPs tracked weekly on team meeting. Miguel Lavalle is going to take a look at it during this cycle. For other proposals we need to have RFEs reported first. Starting the process of removing ML2/Linuxbridge ================================================ Currently in Neutron tree we have 4 drivers: * Linuxbridge, * Openvswitch, * macvtap, * sriov. SR-IOV driver is out of discussion here as this driver is addressing slightly different use case than other out drivers. We started discussion about above topic because we don't want to end up with too many drivers in-tree and we also had some discussions (and we have spec for that already) about include networking-ovn as in-tree driver. So with networking-ovn in-tree we would have already 4 drivers which can be used on any hardware: linuxbridge, ovs, macvtap and ovn. Conclusions from the discussion are: * each driver requires proper testing in the gate, so we need to add many new jobs to our check/gate queue, * currently linuxbridge driver don't have a lot of development and feature parity gaps between linuxbridge and ovs drivers is getting bigger and bigger (e.g. dvr, trunk ports), * also macvtap driver don't have a lot of activity in last few cycles. Maybe this one could be also considered as candidate to deprecation, * we need to have process of deprecating some drivers and time horizon for such actions should be at least 2 cycles. * we will not remove any driver completly but rather we will move it to be in stadium process first so it still can be maintained by people who are interested in it. Actions to do after this discussion: * Miguel Lavalle will contact RAX and Godaddy (we know that those are Linuxbridge users currently) to ask about their feedback about this, * if there are any other companies using LB driver, Nate Johnston is willing to help conctating them, please reach to him in such case. * we may ratify marking linuxbridge as deprecated in the team meeting during Ussuri cycle if nothing surprising pops in. Encrypted(IPSec) tenant networks ================================ Interesting topic proposed but we need to have RFE and spec with more detailed informations about it to continue discussions. Medatada service over IPv6 ========================== This is continuation of old RFE [6]. The only real problem is to choose proper IPv6 address which will be well known address used e.g. by cloud-init. Original spec proposed fe80::a9fe:a9fe as IPv6 address to access metadata service. We decided to be bold and define the standard. Bence Romsics and Miguel Lavalle volunteered to reach out to cloud-init maintainers to discuss that. walkthrough of OVN ================== Since some time we have in review spec about ml2/ovs and ovn convergence. See [7] for details. List of parity gaps between those backends is available at [8]. During the discussion we talked about things like: * migration from ml2/ovs to ml2/ovn - some scripts are already done in [9], * migration from ml2/lb to ml2/ovn - there was no any work done in this topic so far but it should be doable also if someone would need it and want to invest own time for that, * include networking-ovn as in-tree neutron driver and reasons why it could be good idea. Main reasons of that are: ** that would help growing networking-ovn community, ** would help to maintain a healthy project team, ** the default drivers have always been in-tree, However such inclusion may also hurt modularity/logical separation/dependency management/packaging/etc so we need to consider it really carefully and consider all points of view and opinions. Next action item on this topic is to write more detailed summary of this topic and send it to ML and ask wider audience for feedback. IPv6 devstack tempest test configuration vs OVN =============================================== Generally team supports idea which was described during this session and we should change sligtly IPv6 config on e.g. devstack deployments. Neutron - Edge SIG session ========================== We discussed about RFE [10]. This will require also changes on placement side. See [11] for details. Also some cyborg and ovn related changes may be relevant to topics related to Edge. Currently specs which we have are only related to ML2/OVS solution. Neutron - Nova cross project session ==================================== Etherpad for this session is on [12]. Summary written already by gibi can be found at [13]. On [14] You can find image which shows in visual way problem with live-migration of instances with SR-IOV ports. Policy handling in Neutron ========================== The goal of the session was to plan on Neutron's side similar effort to what services like nova are doing now to use new roles like reader and scopes, like project, domain, system provided by Keystone. Miguel Lavalle volunteered to work on this for Neutron and to be part of popup team for cross project collaboration on this topic. Neutron performance improvements ================================ Miguel Lavalle shown us his new profiling decorator [15] and how we all can use it to profile some of API calls in Neutron. Reevaluate Stadium projects =========================== This was follow up discussion after forum session. Notes from forum session can be found at [16]. Nate also prepared some good data about stadium projects activity in last cycles. See at [17] and [18] for details. We all agreed that projects which are in (relatively) good condition now are: * networking-ovn, * networking-odl, * ovsdbapp Projects in bad condition are other projects, like: * neutron-interconnection, * networking-sfc, * networking-bagpipe/bgpvpn, * networking-midonet, * neutron-fwaas and neutron-fwaas-dashboard, * neutron-dynamic-routing, * neutron-vpnaas and neutron-vpnaas-dashboard, We decided to immediately remove neutron-interconnection project as it was never really implemented. For other of those projects, we will send emails to ML to ask for potential maintainers of those projects. If there will be no any volunteers to maintain some of those projects, we will deprecated them and move to "x/" namespace in 2 cycles. Floating IP's On Routed Networks ================================ There is still interest of doing this. Lajos Katona started adding some scenario tests for routed networks already as we need improved test coverage for this feature. Miguel Lavalle said that he will possibly try to work on implementing this in Ussuri cycle. L3 agent enhancement ==================== We talked about couple potential improvements of existing L3 agent, all proposed by LIU Yulong. * retire metering-agent It seems that there is some interest in metering agent recently so we shouldn't probably consider of retiring it for now. We also talked about adding new "tc based" driver to the metering agent and this discussion can be continue on rfe bug [19]. * Centralized DNAT (non-DVR) traffic (floating IP) Scale-out This is proposal of new DVR solution. Some details of this new solution are available at [20]. We agreed that this proposal is trying to solve some very specific use case, and it seems to be very complicated solution with many potential corner cases to address. As a community we don't want to introduce and maintain such complicated new L3 design. * Lazy-load agent side router resources when no related service port Team wants to see RFE with detailed description of the exact problem which this is trying to solve and than continue discussion on such RFE. Zuul jobs ========= In this session we talked about jobs which we can potentially promote to be voting (and we didn't found any of such) and about jobs which we maybe can potentially remove from our queues. Here is what we agreed: * we have 2 iptables_hybrid jobs - one on Fedora and one on Ubuntu - we will drop one of those jobs and left only one of them, * drop neutron-grenade job as it is running still on py27 - we have grenade-py3 which is the same job but run on py36 already, * as it is begin of the cycle, we will switch in devstack neutron uwsgi to be default choice and we will remove "-uwsgi" jobs from queue, * we should compare our single node and multinode variants of same jobs and maybe promote multinode jobs to be voting and then remove single node job - I volunteered to do that, * remove our existing experimental jobs as those jobs are mostly broken and nobody is run those jobs in experimental queue actually, * Yamamoto will check failing networking-midonet job and propose patch to make it passing again, * we will change neutron-tempest-plugin jobs for branch in EM phase to always use certain tempest-plugin and tempest tag, than we will remove those jobs from check and gate queue in master branch, Stateless security groups ========================= Old RFE [21] was approved for neutron-fwaas project but we all agreed that this should be now implemented for security groups in core Neutron. People from Nuage are interested in work on this in upstream. We should probably also explore how easy/hard it will be to implement it in networking-ovn backend. Old, stagnant specs =================== During this session we decided to abandon many of old specs which were proposed long time ago and there is currently no any activity and interest in continue working on them. If anyone would be interested in continue work on some of them, feel free to contact neutron core team on irc or through email and we can always reopen such patch. Community Goal things ===================== We discussed about currently proposed community goals and who can take care of which goal on Neutron's side. Currently there are proposals of community goals as below: * python3 readiness - Nate will take care of this, * move jobs definitions to zuul v3 - I will take care of it. In core neutron and neutron-tempest-plugin we are (mostly) done. On stadium projects' side this will require some work to do, * Project specific PTL and contributor guides - Miguel Lavalle will take care of this goal as former PTL, We will track progress of community goals weekly in our team meetings. Neutron-lib =========== As some time ago our main neutron-lib maintainer (Boden) leaved from the project, we need some new volunteers to continue work on it. Todo list is available on [22]. This should be mostly important for people who are maintaining stadium projects or some 3rd party drivers/plugins so if You are doing things like that, please check list from [22] and reach out to us on ML or #openstack-neutron IRC channel. [1] https://www.slideshare.net/SawomirKaposki/neutron-on-boarding-room [2] https://github.com/gotostack/shanghai_ptg/blob/master/shanghai_neutron_ptg_t... [3] https://etherpad.openstack.org/p/Shanghai-Neutron-Cyborg-xproj [4] https://etherpad.openstack.org/p/kuryr-neutron-nice-to-have [5] https://bugs.launchpad.net/neutron/+bug/1815933 [6] https://bugs.launchpad.net/neutron/+bug/1460177 [7] https://review.opendev.org/#/c/658414/ [8] https://etherpad.openstack.org/p/ML2-OVS-OVN-Convergence [9] https://github.com/openstack/networking-ovn/tree/master/migration [10] https://bugs.launchpad.net/neutron/+bug/1832526 [11] http://lists.openstack.org/pipermail/openstack-discuss/2019-October/009991.h... [12] https://etherpad.openstack.org/p/ptg-ussuri-xproj-nova-neutron [13] http://lists.openstack.org/pipermail/openstack-discuss/2019-November/010654.... [14] https://imgur.com/a/12PrQ9W [15] https://review.opendev.org/678438 [16] https://etherpad.openstack.org/p/PVG-Neutron-stadium-projects-the-path-forwa... [17] https://ethercalc.openstack.org/neutron-stadium-train-metrics [18] https://ibb.co/SBzDGdD [19] https://bugs.launchpad.net/neutron/+bug/1817881 [20] https://imgur.com/a/6MeNUNb [21] https://bugs.launchpad.net/neutron/+bug/1753466 [22] https://etherpad.openstack.org/p/neutron-lib-volunteers-and-punch-list -- Slawek Kaplonski Senior software engineer Red Hat
Many thanks for the summaries. It’s really helpful for those who could not be in the discussions. CERN are also using ML2/Linuxbridge so we’d welcome being involved in any deprecation discussions and migration paths. Tim
On 12 Nov 2019, at 14:53, Slawek Kaplonski <skaplons@redhat.com> wrote:
Hi Neutron team,
First if all thank to all of You for great and very productive week during the PTG in Shanghai. Below is summary of our discussions from whole 3 days. If I forgot about something, please respond to the email and update missing informations. But if You want to have follow up discussion about one of the topics from this summary, please start a new thread to keep this one only as high level summary of the PTG.
...
Starting the process of removing ML2/Linuxbridge ================================================
Currently in Neutron tree we have 4 drivers: * Linuxbridge, * Openvswitch, * macvtap, * sriov. SR-IOV driver is out of discussion here as this driver is addressing slightly different use case than other out drivers.
We started discussion about above topic because we don't want to end up with too many drivers in-tree and we also had some discussions (and we have spec for that already) about include networking-ovn as in-tree driver. So with networking-ovn in-tree we would have already 4 drivers which can be used on any hardware: linuxbridge, ovs, macvtap and ovn. Conclusions from the discussion are: * each driver requires proper testing in the gate, so we need to add many new jobs to our check/gate queue, * currently linuxbridge driver don't have a lot of development and feature parity gaps between linuxbridge and ovs drivers is getting bigger and bigger (e.g. dvr, trunk ports), * also macvtap driver don't have a lot of activity in last few cycles. Maybe this one could be also considered as candidate to deprecation, * we need to have process of deprecating some drivers and time horizon for such actions should be at least 2 cycles. * we will not remove any driver completly but rather we will move it to be in stadium process first so it still can be maintained by people who are interested in it.
Actions to do after this discussion: * Miguel Lavalle will contact RAX and Godaddy (we know that those are Linuxbridge users currently) to ask about their feedback about this, * if there are any other companies using LB driver, Nate Johnston is willing to help conctating them, please reach to him in such case. * we may ratify marking linuxbridge as deprecated in the team meeting during Ussuri cycle if nothing surprising pops in.
Appreciate the summary as well. For what it's worth, the ML2/LinuxBridge combo has been a very stable setup for us since its inception, and I'd hate to see it deprecated and removed for the sake of removing something. Last I checked, trunk ports were supported with the ML2/LinuxBridge driver. And while of course DVR is not a supported feature, a good number of our ML2/LXB environments forgo Neutron routers altogether in favor of putting VMs on the provider network. It has shown to be as performant as vanilla OVS, and a simpler model to implement and support as an operator. Just my two cents. Thanks, James Denton Network Engineer Rackspace Private Cloud james.denton@rackspace.com On 11/12/19, 3:41 PM, "Tim Bell" <Tim.Bell@cern.ch> wrote: CAUTION: This message originated externally, please use caution when clicking on links or opening attachments! Many thanks for the summaries. It’s really helpful for those who could not be in the discussions. CERN are also using ML2/Linuxbridge so we’d welcome being involved in any deprecation discussions and migration paths. Tim > On 12 Nov 2019, at 14:53, Slawek Kaplonski <skaplons@redhat.com> wrote: > > Hi Neutron team, > > First if all thank to all of You for great and very productive week during the > PTG in Shanghai. > Below is summary of our discussions from whole 3 days. > If I forgot about something, please respond to the email and update missing > informations. But if You want to have follow up discussion about one of the > topics from this summary, please start a new thread to keep this one only as > high level summary of the PTG. > > ... > Starting the process of removing ML2/Linuxbridge > ================================================ > > Currently in Neutron tree we have 4 drivers: > * Linuxbridge, > * Openvswitch, > * macvtap, > * sriov. > SR-IOV driver is out of discussion here as this driver is > addressing slightly different use case than other out drivers. > > We started discussion about above topic because we don't want to end up with too > many drivers in-tree and we also had some discussions (and we have spec for that > already) about include networking-ovn as in-tree driver. > So with networking-ovn in-tree we would have already 4 drivers which can be used > on any hardware: linuxbridge, ovs, macvtap and ovn. > Conclusions from the discussion are: > * each driver requires proper testing in the gate, so we need to add many new > jobs to our check/gate queue, > * currently linuxbridge driver don't have a lot of development and feature > parity gaps between linuxbridge and ovs drivers is getting bigger and bigger > (e.g. dvr, trunk ports), > * also macvtap driver don't have a lot of activity in last few cycles. Maybe > this one could be also considered as candidate to deprecation, > * we need to have process of deprecating some drivers and time horizon for such > actions should be at least 2 cycles. > * we will not remove any driver completly but rather we will move it to be in > stadium process first so it still can be maintained by people who are > interested in it. > > Actions to do after this discussion: > * Miguel Lavalle will contact RAX and Godaddy (we know that those are > Linuxbridge users currently) to ask about their feedback about this, > * if there are any other companies using LB driver, Nate Johnston is willing to > help conctating them, please reach to him in such case. > * we may ratify marking linuxbridge as deprecated in the team meeting during > Ussuri cycle if nothing surprising pops in. >
Hi, On Wed, Nov 13, 2019 at 04:31:20AM +0000, James Denton wrote:
Appreciate the summary as well.
For what it's worth, the ML2/LinuxBridge combo has been a very stable setup for us since its inception, and I'd hate to see it deprecated and removed for the sake of removing something. Last I checked, trunk ports were supported with the ML2/LinuxBridge driver. And while of course DVR is not a supported feature, a good number of our ML2/LXB environments forgo Neutron routers altogether in favor of putting VMs on the provider network. It has shown to be as performant as vanilla OVS, and a simpler model to implement and support as an operator.
You're right. Trunk ports are ofcourse supported by LB agent. But e.g. some of QoS rules aren't supported by this backend.
Just my two cents.
Thanks,
James Denton Network Engineer Rackspace Private Cloud james.denton@rackspace.com
On 11/12/19, 3:41 PM, "Tim Bell" <Tim.Bell@cern.ch> wrote:
CAUTION: This message originated externally, please use caution when clicking on links or opening attachments!
Many thanks for the summaries. It’s really helpful for those who could not be in the discussions.
CERN are also using ML2/Linuxbridge so we’d welcome being involved in any deprecation discussions and migration paths.
Tim
> On 12 Nov 2019, at 14:53, Slawek Kaplonski <skaplons@redhat.com> wrote: > > Hi Neutron team, > > First if all thank to all of You for great and very productive week during the > PTG in Shanghai. > Below is summary of our discussions from whole 3 days. > If I forgot about something, please respond to the email and update missing > informations. But if You want to have follow up discussion about one of the > topics from this summary, please start a new thread to keep this one only as > high level summary of the PTG. > > ...
> Starting the process of removing ML2/Linuxbridge > ================================================ > > Currently in Neutron tree we have 4 drivers: > * Linuxbridge, > * Openvswitch, > * macvtap, > * sriov. > SR-IOV driver is out of discussion here as this driver is > addressing slightly different use case than other out drivers. > > We started discussion about above topic because we don't want to end up with too > many drivers in-tree and we also had some discussions (and we have spec for that > already) about include networking-ovn as in-tree driver. > So with networking-ovn in-tree we would have already 4 drivers which can be used > on any hardware: linuxbridge, ovs, macvtap and ovn. > Conclusions from the discussion are: > * each driver requires proper testing in the gate, so we need to add many new > jobs to our check/gate queue, > * currently linuxbridge driver don't have a lot of development and feature > parity gaps between linuxbridge and ovs drivers is getting bigger and bigger > (e.g. dvr, trunk ports), > * also macvtap driver don't have a lot of activity in last few cycles. Maybe > this one could be also considered as candidate to deprecation, > * we need to have process of deprecating some drivers and time horizon for such > actions should be at least 2 cycles. > * we will not remove any driver completly but rather we will move it to be in > stadium process first so it still can be maintained by people who are > interested in it. > > Actions to do after this discussion: > * Miguel Lavalle will contact RAX and Godaddy (we know that those are > Linuxbridge users currently) to ask about their feedback about this, > * if there are any other companies using LB driver, Nate Johnston is willing to > help conctating them, please reach to him in such case. > * we may ratify marking linuxbridge as deprecated in the team meeting during > Ussuri cycle if nothing surprising pops in. >
-- Slawek Kaplonski Senior software engineer Red Hat
Hi, After the neutron PTG session on the future of ML2/LinuxBridge, I discussed this topic in the ops-meetup PTG room (L.43- in [1]). - A lot of needs for Linux Bridge driver was expressed in the room. - LB is for simple network and many ops need it to keep deployment simple including a provider network without L3 feature. - The stats on Linux Bridge usage were shared as well. LB still has a large user base. 40% use Linux Bridge driver according to a survey in Wed's ops(?) session and the user survey last Oct shows 33% use Linux Bridge driver (63% use OVS based) [2]. This discussion does not mean the deprecation of the linux bridge driver. In my understanding, the main motivation is how the neutron team can keep the reference implementations simple. One example is that the features supported in the linux bridge driver are behind those in the OVS driver and some developers think this is the lack of the interest for the linux bridge driver, but this may show that most/not small number of linux bridge users just want simple features. That's my understanding in the PTG. Hope it helps the discussion :) Thanks, Akihiro [1] https://etherpad.openstack.org/p/shanghai-ptg-ops-meetup [2] Deployment Decisions in https://www.openstack.org/analytics On Wed, Nov 13, 2019 at 8:21 PM Slawek Kaplonski <skaplons@redhat.com> wrote:
Hi,
On Wed, Nov 13, 2019 at 04:31:20AM +0000, James Denton wrote:
Appreciate the summary as well.
For what it's worth, the ML2/LinuxBridge combo has been a very stable setup for us since its inception, and I'd hate to see it deprecated and removed for the sake of removing something. Last I checked, trunk ports were supported with the ML2/LinuxBridge driver. And while of course DVR is not a supported feature, a good number of our ML2/LXB environments forgo Neutron routers altogether in favor of putting VMs on the provider network. It has shown to be as performant as vanilla OVS, and a simpler model to implement and support as an operator.
You're right. Trunk ports are ofcourse supported by LB agent. But e.g. some of QoS rules aren't supported by this backend.
Just my two cents.
Thanks,
James Denton Network Engineer Rackspace Private Cloud james.denton@rackspace.com
On 11/12/19, 3:41 PM, "Tim Bell" <Tim.Bell@cern.ch> wrote:
CAUTION: This message originated externally, please use caution when clicking on links or opening attachments!
Many thanks for the summaries. It’s really helpful for those who could not be in the discussions.
CERN are also using ML2/Linuxbridge so we’d welcome being involved in any deprecation discussions and migration paths.
Tim
> On 12 Nov 2019, at 14:53, Slawek Kaplonski <skaplons@redhat.com> wrote: > > Hi Neutron team, > > First if all thank to all of You for great and very productive week during the > PTG in Shanghai. > Below is summary of our discussions from whole 3 days. > If I forgot about something, please respond to the email and update missing > informations. But if You want to have follow up discussion about one of the > topics from this summary, please start a new thread to keep this one only as > high level summary of the PTG. > > ...
> Starting the process of removing ML2/Linuxbridge > ================================================ > > Currently in Neutron tree we have 4 drivers: > * Linuxbridge, > * Openvswitch, > * macvtap, > * sriov. > SR-IOV driver is out of discussion here as this driver is > addressing slightly different use case than other out drivers. > > We started discussion about above topic because we don't want to end up with too > many drivers in-tree and we also had some discussions (and we have spec for that > already) about include networking-ovn as in-tree driver. > So with networking-ovn in-tree we would have already 4 drivers which can be used > on any hardware: linuxbridge, ovs, macvtap and ovn. > Conclusions from the discussion are: > * each driver requires proper testing in the gate, so we need to add many new > jobs to our check/gate queue, > * currently linuxbridge driver don't have a lot of development and feature > parity gaps between linuxbridge and ovs drivers is getting bigger and bigger > (e.g. dvr, trunk ports), > * also macvtap driver don't have a lot of activity in last few cycles. Maybe > this one could be also considered as candidate to deprecation, > * we need to have process of deprecating some drivers and time horizon for such > actions should be at least 2 cycles. > * we will not remove any driver completly but rather we will move it to be in > stadium process first so it still can be maintained by people who are > interested in it. > > Actions to do after this discussion: > * Miguel Lavalle will contact RAX and Godaddy (we know that those are > Linuxbridge users currently) to ask about their feedback about this, > * if there are any other companies using LB driver, Nate Johnston is willing to > help conctating them, please reach to him in such case. > * we may ratify marking linuxbridge as deprecated in the team meeting during > Ussuri cycle if nothing surprising pops in. >
-- Slawek Kaplonski Senior software engineer Red Hat
On Tue, 2019-11-12 at 14:53 +0100, Slawek Kaplonski wrote:
Stateless security groups =========================
Old RFE [21] was approved for neutron-fwaas project but we all agreed that this should be now implemented for security groups in core Neutron. People from Nuage are interested in work on this in upstream. We should probably also explore how easy/hard it will be to implement it in networking-ovn backend.
for what its worth we implemented this 4 years ago and it was breifly used in production trial deployment in a telco deployment but i dont think it ever went to full production as they went wtih sriov instead https://review.opendev.org/#/c/264131/ as part of this RFE https://bugs.launchpad.net/neutron/+bug/1531205 which was closed as wont fix https://bugs.launchpad.net/neutron/+bug/1531205/comments/14 as it was view that this was not the correct long term direction for the community. this is the summit presentation for austin for anyone that does not rememebr this effort https://www.openstack.org/videos/summits/austin-2016/tired-of-iptables-based... im not sure how the new proposal differeres form our previous proposal for the same feautre but the main pushback we got was that the securtiy group api is assumed to be stateful and that is why this was rejected. form our mesurments at the time we expected the stateless approch to scale better then contrack driver so it woudl be nice to see a stateless approch avialable. i never got around to deleteing our implemenation form networking-ovs-dpdk https://opendev.org/x/networking-ovs-dpdk/src/branch/master/networking_ovs_d... but i has not been tested our updated really for the last 2 years but it could be used as a basis of this effort if nuage does not have a poc already.
Hi, Akihiro, thanks for you summary. We use the linuxbridge driver because its simplicity and the match with the old nova-network schema (yes, are we still migrating). The functionality gap between ovs driver and linuxbridge is a good think in my view. It allows operators to chose the best solution considering their deployment use case and scale. Slawek, Miguel please keep us in the discussions. Belmiro CERN On Wed, Nov 13, 2019 at 7:22 PM Sean Mooney <smooney@redhat.com> wrote:
Stateless security groups =========================
Old RFE [21] was approved for neutron-fwaas project but we all agreed
On Tue, 2019-11-12 at 14:53 +0100, Slawek Kaplonski wrote: that this
should be now implemented for security groups in core Neutron. People from Nuage are interested in work on this in upstream. We should probably also explore how easy/hard it will be to implement it in networking-ovn backend.
for what its worth we implemented this 4 years ago and it was breifly used in production trial deployment in a telco deployment but i dont think it ever went to full production as they went wtih sriov instead https://review.opendev.org/#/c/264131/ as part of this RFE https://bugs.launchpad.net/neutron/+bug/1531205 which was closed as wont fix https://bugs.launchpad.net/neutron/+bug/1531205/comments/14 as it was view that this was not the correct long term direction for the community. this is the summit presentation for austin for anyone that does not rememebr this effort
https://www.openstack.org/videos/summits/austin-2016/tired-of-iptables-based...
im not sure how the new proposal differeres form our previous proposal for the same feautre but the main pushback we got was that the securtiy group api is assumed to be stateful and that is why this was rejected. form our mesurments at the time we expected the stateless approch to scale better then contrack driver so it woudl be nice to see a stateless approch avialable. i never got around to deleteing our implemenation form networking-ovs-dpdk
https://opendev.org/x/networking-ovs-dpdk/src/branch/master/networking_ovs_d... but i has not been tested our updated really for the last 2 years but it could be used as a basis of this effort if nuage does not have a poc already.
Hi Belmiro, The Neutron team is fully cognizant that we have operators large and small using Linuxbridge. No decision will be made without involving you Regards On Thu, Nov 14, 2019 at 3:59 AM Belmiro Moreira < moreira.belmiro.email.lists@gmail.com> wrote:
Hi, Akihiro, thanks for you summary.
We use the linuxbridge driver because its simplicity and the match with the old nova-network schema (yes, are we still migrating).
The functionality gap between ovs driver and linuxbridge is a good think in my view. It allows operators to chose the best solution considering their deployment use case and scale.
Slawek, Miguel please keep us in the discussions.
Belmiro CERN
On Wed, Nov 13, 2019 at 7:22 PM Sean Mooney <smooney@redhat.com> wrote:
Stateless security groups =========================
Old RFE [21] was approved for neutron-fwaas project but we all agreed
On Tue, 2019-11-12 at 14:53 +0100, Slawek Kaplonski wrote: that this
should be now implemented for security groups in core Neutron. People from Nuage are interested in work on this in upstream. We should probably also explore how easy/hard it will be to implement it in networking-ovn backend.
for what its worth we implemented this 4 years ago and it was breifly used in production trial deployment in a telco deployment but i dont think it ever went to full production as they went wtih sriov instead https://review.opendev.org/#/c/264131/ as part of this RFE https://bugs.launchpad.net/neutron/+bug/1531205 which was closed as wont fix https://bugs.launchpad.net/neutron/+bug/1531205/comments/14 as it was view that this was not the correct long term direction for the community. this is the summit presentation for austin for anyone that does not rememebr this effort
https://www.openstack.org/videos/summits/austin-2016/tired-of-iptables-based...
im not sure how the new proposal differeres form our previous proposal for the same feautre but the main pushback we got was that the securtiy group api is assumed to be stateful and that is why this was rejected. form our mesurments at the time we expected the stateless approch to scale better then contrack driver so it woudl be nice to see a stateless approch avialable. i never got around to deleteing our implemenation form networking-ovs-dpdk
https://opendev.org/x/networking-ovs-dpdk/src/branch/master/networking_ovs_d... but i has not been tested our updated really for the last 2 years but it could be used as a basis of this effort if nuage does not have a poc already.
Hi, In http://kaplonski.pl/files/neutron_team_photo_shanghai_2019.zip You can find some good quality photos from Neutron team dinner which we had in Shanghai. Thx LIU Yulong for those pictures :) On Tue, Nov 12, 2019 at 02:53:11PM +0100, Slawek Kaplonski wrote:
Hi Neutron team,
First if all thank to all of You for great and very productive week during the PTG in Shanghai. Below is summary of our discussions from whole 3 days. If I forgot about something, please respond to the email and update missing informations. But if You want to have follow up discussion about one of the topics from this summary, please start a new thread to keep this one only as high level summary of the PTG.
On boarding ===========
Slides from onboarding session can be found at [1] If You have any follow up questions to us about onboarding, or You need help with starting any work in Neutron team, please contact me or Miguel Lavalle by email or on IRC. My IRC nick is slaweq and Miguel's nick is mlavalle. We are available on #openstack-neutron channel @freenode.
Train retrospective ===================
Good things in Train cycle: * working with this team is still good experience * core team is stable, and we didn't lost any core reviewers during the cycle, * networking is still one of key reasons why people use OpenStack
Not good things: * dimished vitality in stadium projects - we had also forum session and follow discussion about this later during the PTG, * gate instability - we have seen many issues which were out of our control, like infra problems, grenade jobs failures, other projects failures, but also many bugs on our side, * we have really a lot of jobs in our check/gate queue. If each of them is failing 5% of times, it's hard to merge any patch as almost every time, one of jobs will fail. Later during the PTG we also discussed that topic and we were looking for some jobs which we maybe can potentially drop from our queues. See below for summary about that,
Action items/improvements: * many team meetings each week. We decided to limit number of meetings by: ** consolidate performance subteam meeting into weekly team meeting - this topic will be added to the team meeting's agenda for team meetings on Monday, ** consolidate ovn convergence meeting into weekly team meeting - this topic will be added to the team meeting's agenda for team meetings on Tuesday, ** we need to check if QoS subteam meeting is still needed,
* Review process: list of actual review priorities would be useful for the team, we will add "Review-Priority" label to the Neutron reviews board and try to use it during the Ussuri cycle.
Openvswitch agent enhancements ==============================
We had bunch of topics related to potential improvements for neutron-openvswitch-agent proposed mostly by Liu Yulong. Slides with his proposals are available at [2].
* retire DHCP agent - resyncs of DHCP agent are problematic, especially when agent hosts many networks. Proposal was to add new L2 agent's extension which could be used instead of "regular" DHCP agent and to provide only basic DHCP functionalities. Such solutions would work in the way quite similar to how networking-ovn works today but we would need to implement and maintain own dhcp server application.
Problems of this solution are: ** problems with compatibility e.g. with Ironic, ** how it would work with mixed deployments, e.g. with ovs and sriov agents, ** support for dhcp options,
Advantages of this solution: ** fully distributed DHCP service, ** no DHCP agents, so less RPC messages on the bus and easier maintanance of the agents,
Team's feedback for that is that this is potentially nice solution which may helps in some specific, large scale deploymnets. We can continue discussion about this during Ussuri cycle for sure.
* add accepted egress fdb flows We agreed that this is a bug and we should continue work on this to propose some way to fix it. Solution proposed by LIU during this discussion wasn't good as it could potentially break some corner cases.
* new API and agent for L2 traffic health check The team asked to add to the spec some more detailed and concrete use cases with explanation how this new API may help operator of the cloud to investigate where the problem actually is.
* Local flows cache and batch updating The team agreed that as long as this will be optional solution which operator can opt-in we can give it a try. But spec and discuss details there will be necessary.
* stop processing ports twice in ovs-agent We all agreed that this is a bug and should be fixed. But we have to be careful as fixing this bug may cause some other problems e.g. with live-migration - see nova-neutron cross project session.
* ovs-agent: batch flow updates with --bundle We all agreed that this can be done as an improvement of existing code. Similar option is already used in openvswitch firewall driver.
Neutron - Cyborg cross project session ======================================
Etherpad for the session is at [3]. Cyborg team wants to include Neutron in workflow of spawning VMs with Smart NICs or accelerator cards. From Neutron's side, required change is to allow including "accel" data in port binding profile. As long as this will be well documented what can be placed there, there should be no problem with doing that. Technically we can place almost anything there.
Neutron - Kuryr cross project session =====================================
Etherpad for the session is at [4]. Kuryr team proposed 4 improvements for Neutron which would help a lot Kuryr. Ideas are: * Network cascade deletion, * Force subport deletion, * Tag resources at creation time, * Security group creation with rules & bulk security group rule creation
All of those ideas makes sense for Neutron team. Tag resources at creation time is even accepted rfe already - see [5] but there was no volunteer to implement it. We will add it to list of our BPs tracked weekly on team meeting. Miguel Lavalle is going to take a look at it during this cycle. For other proposals we need to have RFEs reported first.
Starting the process of removing ML2/Linuxbridge ================================================
Currently in Neutron tree we have 4 drivers: * Linuxbridge, * Openvswitch, * macvtap, * sriov. SR-IOV driver is out of discussion here as this driver is addressing slightly different use case than other out drivers.
We started discussion about above topic because we don't want to end up with too many drivers in-tree and we also had some discussions (and we have spec for that already) about include networking-ovn as in-tree driver. So with networking-ovn in-tree we would have already 4 drivers which can be used on any hardware: linuxbridge, ovs, macvtap and ovn. Conclusions from the discussion are: * each driver requires proper testing in the gate, so we need to add many new jobs to our check/gate queue, * currently linuxbridge driver don't have a lot of development and feature parity gaps between linuxbridge and ovs drivers is getting bigger and bigger (e.g. dvr, trunk ports), * also macvtap driver don't have a lot of activity in last few cycles. Maybe this one could be also considered as candidate to deprecation, * we need to have process of deprecating some drivers and time horizon for such actions should be at least 2 cycles. * we will not remove any driver completly but rather we will move it to be in stadium process first so it still can be maintained by people who are interested in it.
Actions to do after this discussion: * Miguel Lavalle will contact RAX and Godaddy (we know that those are Linuxbridge users currently) to ask about their feedback about this, * if there are any other companies using LB driver, Nate Johnston is willing to help conctating them, please reach to him in such case. * we may ratify marking linuxbridge as deprecated in the team meeting during Ussuri cycle if nothing surprising pops in.
Encrypted(IPSec) tenant networks ================================
Interesting topic proposed but we need to have RFE and spec with more detailed informations about it to continue discussions.
Medatada service over IPv6 ==========================
This is continuation of old RFE [6]. The only real problem is to choose proper IPv6 address which will be well known address used e.g. by cloud-init. Original spec proposed fe80::a9fe:a9fe as IPv6 address to access metadata service. We decided to be bold and define the standard. Bence Romsics and Miguel Lavalle volunteered to reach out to cloud-init maintainers to discuss that.
walkthrough of OVN ==================
Since some time we have in review spec about ml2/ovs and ovn convergence. See [7] for details. List of parity gaps between those backends is available at [8]. During the discussion we talked about things like: * migration from ml2/ovs to ml2/ovn - some scripts are already done in [9], * migration from ml2/lb to ml2/ovn - there was no any work done in this topic so far but it should be doable also if someone would need it and want to invest own time for that, * include networking-ovn as in-tree neutron driver and reasons why it could be good idea.
Main reasons of that are: ** that would help growing networking-ovn community, ** would help to maintain a healthy project team, ** the default drivers have always been in-tree,
However such inclusion may also hurt modularity/logical separation/dependency management/packaging/etc so we need to consider it really carefully and consider all points of view and opinions.
Next action item on this topic is to write more detailed summary of this topic and send it to ML and ask wider audience for feedback.
IPv6 devstack tempest test configuration vs OVN ===============================================
Generally team supports idea which was described during this session and we should change sligtly IPv6 config on e.g. devstack deployments.
Neutron - Edge SIG session ==========================
We discussed about RFE [10]. This will require also changes on placement side. See [11] for details. Also some cyborg and ovn related changes may be relevant to topics related to Edge. Currently specs which we have are only related to ML2/OVS solution.
Neutron - Nova cross project session ====================================
Etherpad for this session is on [12]. Summary written already by gibi can be found at [13]. On [14] You can find image which shows in visual way problem with live-migration of instances with SR-IOV ports.
Policy handling in Neutron ==========================
The goal of the session was to plan on Neutron's side similar effort to what services like nova are doing now to use new roles like reader and scopes, like project, domain, system provided by Keystone. Miguel Lavalle volunteered to work on this for Neutron and to be part of popup team for cross project collaboration on this topic.
Neutron performance improvements ================================
Miguel Lavalle shown us his new profiling decorator [15] and how we all can use it to profile some of API calls in Neutron.
Reevaluate Stadium projects ===========================
This was follow up discussion after forum session. Notes from forum session can be found at [16]. Nate also prepared some good data about stadium projects activity in last cycles. See at [17] and [18] for details. We all agreed that projects which are in (relatively) good condition now are: * networking-ovn, * networking-odl, * ovsdbapp
Projects in bad condition are other projects, like: * neutron-interconnection, * networking-sfc, * networking-bagpipe/bgpvpn, * networking-midonet, * neutron-fwaas and neutron-fwaas-dashboard, * neutron-dynamic-routing, * neutron-vpnaas and neutron-vpnaas-dashboard,
We decided to immediately remove neutron-interconnection project as it was never really implemented. For other of those projects, we will send emails to ML to ask for potential maintainers of those projects. If there will be no any volunteers to maintain some of those projects, we will deprecated them and move to "x/" namespace in 2 cycles.
Floating IP's On Routed Networks ================================
There is still interest of doing this. Lajos Katona started adding some scenario tests for routed networks already as we need improved test coverage for this feature. Miguel Lavalle said that he will possibly try to work on implementing this in Ussuri cycle.
L3 agent enhancement ====================
We talked about couple potential improvements of existing L3 agent, all proposed by LIU Yulong.
* retire metering-agent It seems that there is some interest in metering agent recently so we shouldn't probably consider of retiring it for now. We also talked about adding new "tc based" driver to the metering agent and this discussion can be continue on rfe bug [19].
* Centralized DNAT (non-DVR) traffic (floating IP) Scale-out This is proposal of new DVR solution. Some details of this new solution are available at [20]. We agreed that this proposal is trying to solve some very specific use case, and it seems to be very complicated solution with many potential corner cases to address. As a community we don't want to introduce and maintain such complicated new L3 design.
* Lazy-load agent side router resources when no related service port Team wants to see RFE with detailed description of the exact problem which this is trying to solve and than continue discussion on such RFE.
Zuul jobs =========
In this session we talked about jobs which we can potentially promote to be voting (and we didn't found any of such) and about jobs which we maybe can potentially remove from our queues. Here is what we agreed: * we have 2 iptables_hybrid jobs - one on Fedora and one on Ubuntu - we will drop one of those jobs and left only one of them, * drop neutron-grenade job as it is running still on py27 - we have grenade-py3 which is the same job but run on py36 already, * as it is begin of the cycle, we will switch in devstack neutron uwsgi to be default choice and we will remove "-uwsgi" jobs from queue, * we should compare our single node and multinode variants of same jobs and maybe promote multinode jobs to be voting and then remove single node job - I volunteered to do that, * remove our existing experimental jobs as those jobs are mostly broken and nobody is run those jobs in experimental queue actually, * Yamamoto will check failing networking-midonet job and propose patch to make it passing again, * we will change neutron-tempest-plugin jobs for branch in EM phase to always use certain tempest-plugin and tempest tag, than we will remove those jobs from check and gate queue in master branch,
Stateless security groups =========================
Old RFE [21] was approved for neutron-fwaas project but we all agreed that this should be now implemented for security groups in core Neutron. People from Nuage are interested in work on this in upstream. We should probably also explore how easy/hard it will be to implement it in networking-ovn backend.
Old, stagnant specs ===================
During this session we decided to abandon many of old specs which were proposed long time ago and there is currently no any activity and interest in continue working on them. If anyone would be interested in continue work on some of them, feel free to contact neutron core team on irc or through email and we can always reopen such patch.
Community Goal things =====================
We discussed about currently proposed community goals and who can take care of which goal on Neutron's side. Currently there are proposals of community goals as below: * python3 readiness - Nate will take care of this, * move jobs definitions to zuul v3 - I will take care of it. In core neutron and neutron-tempest-plugin we are (mostly) done. On stadium projects' side this will require some work to do, * Project specific PTL and contributor guides - Miguel Lavalle will take care of this goal as former PTL,
We will track progress of community goals weekly in our team meetings.
Neutron-lib ===========
As some time ago our main neutron-lib maintainer (Boden) leaved from the project, we need some new volunteers to continue work on it. Todo list is available on [22]. This should be mostly important for people who are maintaining stadium projects or some 3rd party drivers/plugins so if You are doing things like that, please check list from [22] and reach out to us on ML or #openstack-neutron IRC channel.
[1] https://www.slideshare.net/SawomirKaposki/neutron-on-boarding-room [2] https://github.com/gotostack/shanghai_ptg/blob/master/shanghai_neutron_ptg_t... [3] https://etherpad.openstack.org/p/Shanghai-Neutron-Cyborg-xproj [4] https://etherpad.openstack.org/p/kuryr-neutron-nice-to-have [5] https://bugs.launchpad.net/neutron/+bug/1815933 [6] https://bugs.launchpad.net/neutron/+bug/1460177 [7] https://review.opendev.org/#/c/658414/ [8] https://etherpad.openstack.org/p/ML2-OVS-OVN-Convergence [9] https://github.com/openstack/networking-ovn/tree/master/migration [10] https://bugs.launchpad.net/neutron/+bug/1832526 [11] http://lists.openstack.org/pipermail/openstack-discuss/2019-October/009991.h... [12] https://etherpad.openstack.org/p/ptg-ussuri-xproj-nova-neutron [13] http://lists.openstack.org/pipermail/openstack-discuss/2019-November/010654.... [14] https://imgur.com/a/12PrQ9W [15] https://review.opendev.org/678438 [16] https://etherpad.openstack.org/p/PVG-Neutron-stadium-projects-the-path-forwa... [17] https://ethercalc.openstack.org/neutron-stadium-train-metrics [18] https://ibb.co/SBzDGdD [19] https://bugs.launchpad.net/neutron/+bug/1817881 [20] https://imgur.com/a/6MeNUNb [21] https://bugs.launchpad.net/neutron/+bug/1753466 [22] https://etherpad.openstack.org/p/neutron-lib-volunteers-and-punch-list
-- Slawek Kaplonski Senior software engineer Red Hat
-- Slawek Kaplonski Senior software engineer Red Hat
participants (7)
-
Akihiro Motoki
-
Belmiro Moreira
-
James Denton
-
Miguel Lavalle
-
Sean Mooney
-
Slawek Kaplonski
-
Tim Bell