[neutron][ptg] 2024.1/Caracal vPTG summary
Sorry for the slowness of getting this out.
Here is a summary of the discussions we had over our two days of meetings, as well as other related sessions. The etherpad link can be found at [0].
Monday, October 23rd --------------------
# OS DBPerf Cross project session [1]
This session was to talk about devstack performance issues that are believed to be caused by excessive querying by Neutron and Keystone.
By default, when devstack "stacks", DB counter values are collected on each of the subsystems. Keystone and Neutron are outliers on how many SELECTs they issue during this process. As this is such a high-level number (4x to 10x other users), more work is required to understand what to fix, or if it's just normal behavior. For example, there is data from just a "stack" in addition to a tempest run that might be able to narrow things down here.
Slawek and Miguel have agreed to work with the oslo.db team and Mike Bayer to get a better understanding of things here.
Tuesday, October 24th ---------------------
# Horizon
For the last few cycles the Neutron team (Lajos) has been trying to deprecate usage of neutronclient. While this work is mostly complete, there is usage in Horizon that needs to be addressed.
Lajos and Amotoki will be working on this. Full notes at [2]
Wednesday, October 25th -----------------------
# Retrospective
As is customary the team did a retrospective of the good, bad and what could be improved.
In short, the Neutron team has met its goals from last cycle of completing Sqlalchemy 2.0 and RBAC work, while continuing to deliver a few new features and a lot of bug fixes. Most of the team are long time members which helps.
One place we are working on is our job stability, and we continue to look at how to increase our review velocity. Our initial proposal on the latter is to have review marathons, perhaps on a monthly basis, since it has the double effect of closing bugs in the process.
# EOL/EM cadence
Train just went EOL. Decision was to propose Ussuri for EOL this cycle.
# Bugs
The total number of open bugs against Neutron has crept up since the beginning of Bobcat, from 726 to 760. I asked people to go through any bugs they own and make sure they are in the right state or should be closed. This is something we've done in previous cycles that has reduced our count by 1/3.
# Gate
We now have voting SLURP jobs! I.e. from Antelope to Caracal.
Rodolfo has also been working on fullstack performance to try and reduce the runtime since we hit the 3 hour limit occasionally. Action item was taken by Slawek to see if there was overlap with other tests to see what could be removed.
There was a question on what versions of OVS and OVN we use in our gate, and if there is a rule on when we should bump it, especially since sometimes new features require newer versions. Action item was taken by me to create a requirements doc similar to what Nova does for libvirt.
# Neutronclient deprecation
As discussed above, Horizon will be the focus this cycle.
# OVN drivers for stadium projects
There have been efforts recently to make stadium projects work with OVN, for example VPNaaS, FWaas and Taas. Main ask for for review cycles from core contributors.
# Hidden SG rules in OVN
Background is at [3]. Short story is that there are a minimum set of things that must be allowed for a VM to configure and use the network, like ARP, DHCP, ND, RA, RS and MLD. ML2/OVS adds hidden SG rules for these, and OVN allows some via ACLs. Our api-ref actually documents these requirements since stateless SGs needed to add them to function properly. The only service that is not enabled by default with stateless SGs is metadata. Question was regarding if we should make these rules visible in a read-only state.
# S-RBAC
This is a community goal and Neutron has completed the first phase and is part of phase 2 (service role), additional testing and nova/neutron token passing TBD. Phase 3 is the Project Manager role, which is not planned for this cycle.
Thursday, October 26th ----------------------
# RFE reviews
Discussed three active RFEs, two related to neutron-dynamic-routing, and one for OVN interconnection. Further discussion to take place in reviews and/or drivers meeting.
# Metadata changes
I noticed there were a number of changes to the metadata code in flight, so we went through them to try and create order out of the chaos. Basically, we will try and merge any bug fixes first since they need to be backported, followed by features and refactors. See the etherpad for more details.
# OVN DB sync utility
This tool is typically run after a migration or upgrade to make sure the Neutron and OVN DBs are in sync. A number of bugs have been fixed and backported, but there might be more. An action item was taken to try and make it more robust, and also make sure usage is documented.
# Active-Active L3 gateway multi-homing
Frode Nordahl gave an update on this OVN work that started in the Bobcat cycle, necessary to support multiple external gateways. A little over half the code is merged, plan is to complete the rest this cycle.
# Nova cross-project
Sean Mooney proposed that Neutron starts making certain extensions mandatory, for example, multiple port bindings, that make migration possible. Since then other features have been added that require it as well. Should we stop supporting backends that have not implemented it?
We agreed this is a good step forward to what should be the minimum requirements of Neutron. In addition:
- Nova will deprecate the legacy code path that doesn't use multiple bindings and log a warning, to be deleted in E release - Neutron will do the same for any similar code - Sean will provide a blueprint explaining which API extensions will be mandatory by Nova, as well as file a Neutron RFE bug - Eventually, we'll provide release notes and a ML thread for explaining all of this (we being the Nova and Neutron PTLs)
Complete notes in Nova etherpad [4].
Thanks, and if you think I missed something please send updates or corrections.
-Brian
[0] https://etherpad.opendev.org/p/oct2023-ptg-neutron [1] https://etherpad.opendev.org/p/oct2023-ptg-os-dbperf-crossproject [2] https://etherpad.opendev.org/p/horizon-caracal-ptg [3] https://bugs.launchpad.net/neutron/+bug/1926515 [4] https://etherpad.opendev.org/p/nova-caracal-ptg#L80
Hi Brian,
what about the OVN VPNaaS integration, are there any plans to merge it in the Caracal cycle? I believe that this is a blocker for many. We for our part would definitely need support going forward.
Thank you!
Best regards, Justin Lamp
Am 07.11.23 um 23:39 schrieb Brian Haley:
Sorry for the slowness of getting this out.
Here is a summary of the discussions we had over our two days of meetings, as well as other related sessions. The etherpad link can be found at [0].
Monday, October 23rd
# OS DBPerf Cross project session [1]
This session was to talk about devstack performance issues that are believed to be caused by excessive querying by Neutron and Keystone.
By default, when devstack "stacks", DB counter values are collected on each of the subsystems. Keystone and Neutron are outliers on how many SELECTs they issue during this process. As this is such a high-level number (4x to 10x other users), more work is required to understand what to fix, or if it's just normal behavior. For example, there is data from just a "stack" in addition to a tempest run that might be able to narrow things down here.
Slawek and Miguel have agreed to work with the oslo.db team and Mike Bayer to get a better understanding of things here.
Tuesday, October 24th
# Horizon
For the last few cycles the Neutron team (Lajos) has been trying to deprecate usage of neutronclient. While this work is mostly complete, there is usage in Horizon that needs to be addressed.
Lajos and Amotoki will be working on this. Full notes at [2]
Wednesday, October 25th
# Retrospective
As is customary the team did a retrospective of the good, bad and what could be improved.
In short, the Neutron team has met its goals from last cycle of completing Sqlalchemy 2.0 and RBAC work, while continuing to deliver a few new features and a lot of bug fixes. Most of the team are long time members which helps.
One place we are working on is our job stability, and we continue to look at how to increase our review velocity. Our initial proposal on the latter is to have review marathons, perhaps on a monthly basis, since it has the double effect of closing bugs in the process.
# EOL/EM cadence
Train just went EOL. Decision was to propose Ussuri for EOL this cycle.
# Bugs
The total number of open bugs against Neutron has crept up since the beginning of Bobcat, from 726 to 760. I asked people to go through any bugs they own and make sure they are in the right state or should be closed. This is something we've done in previous cycles that has reduced our count by 1/3.
# Gate
We now have voting SLURP jobs! I.e. from Antelope to Caracal.
Rodolfo has also been working on fullstack performance to try and reduce the runtime since we hit the 3 hour limit occasionally. Action item was taken by Slawek to see if there was overlap with other tests to see what could be removed.
There was a question on what versions of OVS and OVN we use in our gate, and if there is a rule on when we should bump it, especially since sometimes new features require newer versions. Action item was taken by me to create a requirements doc similar to what Nova does for libvirt.
# Neutronclient deprecation
As discussed above, Horizon will be the focus this cycle.
# OVN drivers for stadium projects
There have been efforts recently to make stadium projects work with OVN, for example VPNaaS, FWaas and Taas. Main ask for for review cycles from core contributors.
# Hidden SG rules in OVN
Background is at [3]. Short story is that there are a minimum set of things that must be allowed for a VM to configure and use the network, like ARP, DHCP, ND, RA, RS and MLD. ML2/OVS adds hidden SG rules for these, and OVN allows some via ACLs. Our api-ref actually documents these requirements since stateless SGs needed to add them to function properly. The only service that is not enabled by default with stateless SGs is metadata. Question was regarding if we should make these rules visible in a read-only state.
# S-RBAC
This is a community goal and Neutron has completed the first phase and is part of phase 2 (service role), additional testing and nova/neutron token passing TBD. Phase 3 is the Project Manager role, which is not planned for this cycle.
Thursday, October 26th
# RFE reviews
Discussed three active RFEs, two related to neutron-dynamic-routing, and one for OVN interconnection. Further discussion to take place in reviews and/or drivers meeting.
# Metadata changes
I noticed there were a number of changes to the metadata code in flight, so we went through them to try and create order out of the chaos. Basically, we will try and merge any bug fixes first since they need to be backported, followed by features and refactors. See the etherpad for more details.
# OVN DB sync utility
This tool is typically run after a migration or upgrade to make sure the Neutron and OVN DBs are in sync. A number of bugs have been fixed and backported, but there might be more. An action item was taken to try and make it more robust, and also make sure usage is documented.
# Active-Active L3 gateway multi-homing
Frode Nordahl gave an update on this OVN work that started in the Bobcat cycle, necessary to support multiple external gateways. A little over half the code is merged, plan is to complete the rest this cycle.
# Nova cross-project
Sean Mooney proposed that Neutron starts making certain extensions mandatory, for example, multiple port bindings, that make migration possible. Since then other features have been added that require it as well. Should we stop supporting backends that have not implemented it?
We agreed this is a good step forward to what should be the minimum requirements of Neutron. In addition:
- Nova will deprecate the legacy code path that doesn't use multiple
bindings and log a warning, to be deleted in E release
- Neutron will do the same for any similar code
- Sean will provide a blueprint explaining which API extensions will
be mandatory by Nova, as well as file a Neutron RFE bug
- Eventually, we'll provide release notes and a ML thread for
explaining all of this (we being the Nova and Neutron PTLs)
Complete notes in Nova etherpad [4].
Thanks, and if you think I missed something please send updates or corrections.
-Brian
[0] https://etherpad.opendev.org/p/oct2023-ptg-neutron [1] https://etherpad.opendev.org/p/oct2023-ptg-os-dbperf-crossproject [2] https://etherpad.opendev.org/p/horizon-caracal-ptg [3] https://bugs.launchpad.net/neutron/+bug/1926515 [4] https://etherpad.opendev.org/p/nova-caracal-ptg#L80
-- Justin Lamp Systems Engineer
NETWAYS Managed Services GmbH | Deutschherrnstr. 15-19 | D-90429 Nuernberg Tel: +49 911 92885-0 | Fax: +49 911 92885-77 CEO: Julian Hein, Bernd Erk, Sebastian Saemann | AG Nuernberg HRB25207 https://www.netways.de | justin.lamp@netways.de
** Open Source Camp on Kubernetes - February | Nuremberg - https://opensourcecamp.de ** ** stackconf 2024 - June | Berlin - https://stackconf.eu ** ** OSMC 2024 - November | Nuremberg - https://osmc.de ** ** NETWAYS Web Services - https://nws.netways.de ** ** NETWAYS Trainings - https://netways.de/trainings **
Hi Justin,
On 11/13/23 5:41 AM, Justin Lamp wrote:
Hi Brian,
what about the OVN VPNaaS integration, are there any plans to merge it in the Caracal cycle? I believe that this is a blocker for many. We for our part would definitely need support going forward.
Yes, the status of this work [0] was brought-up in the scope of the neutron stadium projects. I believe it is close to merging, just needs another +2, being such a large change just might take a little bit. I will put it on the agenda for our weekly meeting tomorrow to see if someone can look at it.
Thanks,
-Brian
[0] https://review.opendev.org/c/openstack/neutron-vpnaas/+/765353
Thank you!
Best regards, Justin Lamp
Am 07.11.23 um 23:39 schrieb Brian Haley:
Sorry for the slowness of getting this out.
Here is a summary of the discussions we had over our two days of meetings, as well as other related sessions. The etherpad link can be found at [0].
Monday, October 23rd
# OS DBPerf Cross project session [1]
This session was to talk about devstack performance issues that are believed to be caused by excessive querying by Neutron and Keystone.
By default, when devstack "stacks", DB counter values are collected on each of the subsystems. Keystone and Neutron are outliers on how many SELECTs they issue during this process. As this is such a high-level number (4x to 10x other users), more work is required to understand what to fix, or if it's just normal behavior. For example, there is data from just a "stack" in addition to a tempest run that might be able to narrow things down here.
Slawek and Miguel have agreed to work with the oslo.db team and Mike Bayer to get a better understanding of things here.
Tuesday, October 24th
# Horizon
For the last few cycles the Neutron team (Lajos) has been trying to deprecate usage of neutronclient. While this work is mostly complete, there is usage in Horizon that needs to be addressed.
Lajos and Amotoki will be working on this. Full notes at [2]
Wednesday, October 25th
# Retrospective
As is customary the team did a retrospective of the good, bad and what could be improved.
In short, the Neutron team has met its goals from last cycle of completing Sqlalchemy 2.0 and RBAC work, while continuing to deliver a few new features and a lot of bug fixes. Most of the team are long time members which helps.
One place we are working on is our job stability, and we continue to look at how to increase our review velocity. Our initial proposal on the latter is to have review marathons, perhaps on a monthly basis, since it has the double effect of closing bugs in the process.
# EOL/EM cadence
Train just went EOL. Decision was to propose Ussuri for EOL this cycle.
# Bugs
The total number of open bugs against Neutron has crept up since the beginning of Bobcat, from 726 to 760. I asked people to go through any bugs they own and make sure they are in the right state or should be closed. This is something we've done in previous cycles that has reduced our count by 1/3.
# Gate
We now have voting SLURP jobs! I.e. from Antelope to Caracal.
Rodolfo has also been working on fullstack performance to try and reduce the runtime since we hit the 3 hour limit occasionally. Action item was taken by Slawek to see if there was overlap with other tests to see what could be removed.
There was a question on what versions of OVS and OVN we use in our gate, and if there is a rule on when we should bump it, especially since sometimes new features require newer versions. Action item was taken by me to create a requirements doc similar to what Nova does for libvirt.
# Neutronclient deprecation
As discussed above, Horizon will be the focus this cycle.
# OVN drivers for stadium projects
There have been efforts recently to make stadium projects work with OVN, for example VPNaaS, FWaas and Taas. Main ask for for review cycles from core contributors.
# Hidden SG rules in OVN
Background is at [3]. Short story is that there are a minimum set of things that must be allowed for a VM to configure and use the network, like ARP, DHCP, ND, RA, RS and MLD. ML2/OVS adds hidden SG rules for these, and OVN allows some via ACLs. Our api-ref actually documents these requirements since stateless SGs needed to add them to function properly. The only service that is not enabled by default with stateless SGs is metadata. Question was regarding if we should make these rules visible in a read-only state.
# S-RBAC
This is a community goal and Neutron has completed the first phase and is part of phase 2 (service role), additional testing and nova/neutron token passing TBD. Phase 3 is the Project Manager role, which is not planned for this cycle.
Thursday, October 26th
# RFE reviews
Discussed three active RFEs, two related to neutron-dynamic-routing, and one for OVN interconnection. Further discussion to take place in reviews and/or drivers meeting.
# Metadata changes
I noticed there were a number of changes to the metadata code in flight, so we went through them to try and create order out of the chaos. Basically, we will try and merge any bug fixes first since they need to be backported, followed by features and refactors. See the etherpad for more details.
# OVN DB sync utility
This tool is typically run after a migration or upgrade to make sure the Neutron and OVN DBs are in sync. A number of bugs have been fixed and backported, but there might be more. An action item was taken to try and make it more robust, and also make sure usage is documented.
# Active-Active L3 gateway multi-homing
Frode Nordahl gave an update on this OVN work that started in the Bobcat cycle, necessary to support multiple external gateways. A little over half the code is merged, plan is to complete the rest this cycle.
# Nova cross-project
Sean Mooney proposed that Neutron starts making certain extensions mandatory, for example, multiple port bindings, that make migration possible. Since then other features have been added that require it as well. Should we stop supporting backends that have not implemented it?
We agreed this is a good step forward to what should be the minimum requirements of Neutron. In addition:
- Nova will deprecate the legacy code path that doesn't use multiple
bindings and log a warning, to be deleted in E release
- Neutron will do the same for any similar code
- Sean will provide a blueprint explaining which API extensions will
be mandatory by Nova, as well as file a Neutron RFE bug
- Eventually, we'll provide release notes and a ML thread for
explaining all of this (we being the Nova and Neutron PTLs)
Complete notes in Nova etherpad [4].
Thanks, and if you think I missed something please send updates or corrections.
-Brian
[0] https://etherpad.opendev.org/p/oct2023-ptg-neutron [1] https://etherpad.opendev.org/p/oct2023-ptg-os-dbperf-crossproject [2] https://etherpad.opendev.org/p/horizon-caracal-ptg [3] https://bugs.launchpad.net/neutron/+bug/1926515 [4] https://etherpad.opendev.org/p/nova-caracal-ptg#L80
-- Justin Lamp Systems Engineer
NETWAYS Managed Services GmbH | Deutschherrnstr. 15-19 | D-90429 Nuernberg Tel: +49 911 92885-0 | Fax: +49 911 92885-77 CEO: Julian Hein, Bernd Erk, Sebastian Saemann | AG Nuernberg HRB25207 https://www.netways.de | justin.lamp@netways.de
** Open Source Camp on Kubernetes - February | Nuremberg - https://opensourcecamp.de ** ** stackconf 2024 - June | Berlin - https://stackconf.eu ** ** OSMC 2024 - November | Nuremberg - https://osmc.de ** ** NETWAYS Web Services - https://nws.netways.de ** ** NETWAYS Trainings - https://netways.de/trainings **
participants (2)
-
Brian Haley
-
Justin Lamp