[neutron][interop][refstack] New tests and capabilities to track in interop
Hi everyone, I would like to further discuss the topics we covered with the neutron team during the PTG [1]. * adding address_group API capability It's tested by tests in neutron-tempest-plugin. First question is if tests which are not directly in tempest can be a part of a non-add-on marketing program? It's possible to move them to tempest though, by the time we do so, could they be marked as advisory? * Shall we include QoS tempest tests since we don't know what share of vendors enable QoS? Could it be an add-on? These tests are also in neutron-tempest-plugin, I assume we're talking about neutron_tempest_plugin.api.test_qos tests. If we want to include these tests, which program should they belong to? Do we wanna create a new one? [1] https://etherpad.opendev.org/p/neutron-xena-ptg Thanks, -- Martin Kopec Senior Software Quality Engineer Red Hat EMEA
Comments inline. From: Martin Kopec <mkopec@redhat.com> Sent: Monday, April 26, 2021 10:48 AM To: openstack-discuss; Kanevsky, Arkady; Slawek Kaplonski Subject: [neutron][interop][refstack] New tests and capabilities to track in interop [EXTERNAL EMAIL] Hi everyone, I would like to further discuss the topics we covered with the neutron team during the PTG [1]. * adding address_group API capability It's tested by tests in neutron-tempest-plugin. First question is if tests which are not directly in tempest can be a part of a non-add-on marketing program? AK – yes as long as it is commonly supported by the clouds and ditto for users. As long as it meet criteria as defined by Interop. It's possible to move them to tempest though, by the time we do so, could they be marked as advisory? AK – I like that approach. * Shall we include QoS tempest tests since we don't know what share of vendors enable QoS? Could it be an add-on? These tests are also in neutron-tempest-plugin, I assume we're talking about neutron_tempest_plugin.api.test_qos tests. If we want to include these tests, which program should they belong to? Do we wanna create a new one? AK – by definition they belong to OpenStack powered compute and platform. AT some old time ago we talked about program for specific verticals. NFV was considered first but did not go anywhere. Lack of bandwidth to make it happen as well as lack of interest in OpenSTack since OPNFV (currently Anuket) was handling it. [1] https://etherpad.opendev.org/p/neutron-xena-ptg [etherpad.opendev.org]<https://urldefense.com/v3/__https:/etherpad.opendev.org/p/neutron-xena-ptg__;!!LpKI!0ffVYPqRJsO-sTwYbNveGON8O2EebE4UmuCl29kaboANM9lkOijmEBWZ2wBhV7kI6NSL$> Thanks, -- Martin Kopec Senior Software Quality Engineer Red Hat EMEA
---- On Mon, 26 Apr 2021 10:48:08 -0500 Martin Kopec <mkopec@redhat.com> wrote ----
Hi everyone, I would like to further discuss the topics we covered with the neutron team during the PTG [1]. * adding address_group API capability It's tested by tests in neutron-tempest-plugin. First question is if tests which arenot directly in tempest can be a part of a non-add-on marketing program?It's possible to move them to tempest though, by the time we do so, could they be marked as advisory?
* Shall we include QoS tempest tests since we don't know what share of vendorsenable QoS? Could it be an add-on?These tests are also in neutron-tempest-plugin, I assume we're talking aboutneutron_tempest_plugin.api.test_qos tests.If we want to include these tests, which program should they belong to? Do we wannacreate a new one?
I remember the discussion on the location of tests required by the interop group 2-3 years back (when heat and dns adds-on were added). We all agreed that having tests in tempest plugins is all fine, they do not need to be in Tempest as such. That is why heat and dns or now manila tests stays in their respective plugins. For neutron also, as there are existing neutron tempest plugin tests we do not need to move them to Tempest and refstack can run it from neutron-tempest-plugin, if we do move then we might break existing upstream or downstream testing scripts running those existing tests from the current location. -gmann
[1] https://etherpad.opendev.org/p/neutron-xena-ptg
Thanks, -- Martin Kopec Senior Software Quality Engineer Red Hat EMEA
Agree. But can we run both tempest and project plug-ins together for the same project? -----Original Message----- From: Ghanshyam Mann <gmann@ghanshyammann.com> Sent: Tuesday, April 27, 2021 1:15 PM To: Martin Kopec Cc: openstack-discuss; Kanevsky, Arkady; Slawek Kaplonski Subject: Re: [neutron][interop][refstack] New tests and capabilities to track in interop [EXTERNAL EMAIL] ---- On Mon, 26 Apr 2021 10:48:08 -0500 Martin Kopec <mkopec@redhat.com> wrote ---- > Hi everyone, > I would like to further discuss the topics we covered with the neutron team during > the PTG [1].
* adding address_group API capability > It's tested by tests in neutron-tempest-plugin. First question is if tests which arenot directly in tempest can be a part of a non-add-on marketing program?It's possible to move them to tempest though, by the time we do so, could they be > marked as advisory?
* Shall we include QoS tempest tests since we don't know what share of vendorsenable QoS? Could it be an add-on?These tests are also in neutron-tempest-plugin, I assume we're talking aboutneutron_tempest_plugin.api.test_qos tests.If we want to include these tests, which program should they belong to? Do we wannacreate a new one?
I remember the discussion on the location of tests required by the interop group 2-3 years back (when heat and dns adds-on were added). We all agreed that having tests in tempest plugins is all fine, they do not need to be in Tempest as such. That is why heat and dns or now manila tests stays in their respective plugins. For neutron also, as there are existing neutron tempest plugin tests we do not need to move them to Tempest and refstack can run it from neutron-tempest-plugin, if we do move then we might break existing upstream or downstream testing scripts running those existing tests from the current location. -gmann
[1] https://urldefense.com/v3/__https://etherpad.opendev.org/p/neutron-xena-ptg_... [etherpad[.]opendev[.]org] > > Thanks, > -- > Martin Kopec > Senior Software Quality Engineer > Red Hat EMEA > > > > > >
---- On Tue, 27 Apr 2021 14:33:12 -0500 Kanevsky, Arkady <Arkady.Kanevsky@dell.com> wrote ----
Agree. But can we run both tempest and project plug-ins together for the same project?
Yes, we can run. There is no restriction on that, we can provide regex matching those tests in Tempest run CLI or tox commands. -gmann
-----Original Message----- From: Ghanshyam Mann <gmann@ghanshyammann.com> Sent: Tuesday, April 27, 2021 1:15 PM To: Martin Kopec Cc: openstack-discuss; Kanevsky, Arkady; Slawek Kaplonski Subject: Re: [neutron][interop][refstack] New tests and capabilities to track in interop
[EXTERNAL EMAIL]
---- On Mon, 26 Apr 2021 10:48:08 -0500 Martin Kopec <mkopec@redhat.com> wrote ---- > Hi everyone, > I would like to further discuss the topics we covered with the neutron team during > the PTG [1].
* adding address_group API capability > It's tested by tests in neutron-tempest-plugin. First question is if tests which arenot directly in tempest can be a part of a non-add-on marketing program?It's possible to move them to tempest though, by the time we do so, could they be > marked as advisory?
* Shall we include QoS tempest tests since we don't know what share of vendorsenable QoS? Could it be an add-on?These tests are also in neutron-tempest-plugin, I assume we're talking aboutneutron_tempest_plugin.api.test_qos tests.If we want to include these tests, which program should they belong to? Do we wannacreate a new one?
I remember the discussion on the location of tests required by the interop group 2-3 years back (when heat and dns adds-on were added). We all agreed that having tests in tempest plugins is all fine, they do not need to be in Tempest as such. That is why heat and dns or now manila tests stays in their respective plugins.
For neutron also, as there are existing neutron tempest plugin tests we do not need to move them to Tempest and refstack can run it from neutron-tempest-plugin, if we do move then we might break existing upstream or downstream testing scripts running those existing tests from the current location.
-gmann
[1] https://urldefense.com/v3/__https://etherpad.opendev.org/p/neutron-xena-ptg_... [etherpad[.]opendev[.]org] > > Thanks, > -- > Martin Kopec > Senior Software Quality Engineer > Red Hat EMEA > > > > > >
Hi, Dnia wtorek, 27 kwietnia 2021 21:38:03 CEST Ghanshyam Mann pisze:
---- On Tue, 27 Apr 2021 14:33:12 -0500 Kanevsky, Arkady <Arkady.Kanevsky@dell.com> wrote ----
Agree. But can we run both tempest and project plug-ins together for the same project?
Yes, we can run. There is no restriction on that, we can provide regex matching those tests in Tempest run CLI or tox commands.
Great news for us as we don't need to move the tests to tempest :) Thx.
-gmann
-----Original Message----- From: Ghanshyam Mann <gmann@ghanshyammann.com> Sent: Tuesday, April 27, 2021 1:15 PM To: Martin Kopec Cc: openstack-discuss; Kanevsky, Arkady; Slawek Kaplonski Subject: Re: [neutron][interop][refstack] New tests and capabilities to
track in interop
[EXTERNAL EMAIL]
---- On Mon, 26 Apr 2021 10:48:08 -0500 Martin Kopec <mkopec@redhat.com>
wrote ---- > Hi everyone, > I would like to further discuss the topics we
covered with the neutron team during > the PTG [1]. >
* adding address_group API capability > It's tested by tests in neutron-tempest-plugin. First question is if tests which arenot directly in tempest can be a part of a non-add-on marketing program?It's possible to move them to tempest though, by the time we do so, could they be > marked as advisory?
* Shall we include QoS tempest tests since we don't know what share of vendorsenable QoS? Could it be an add-on?These tests are also in neutron-tempest-plugin, I assume we're talking aboutneutron_tempest_plugin.api.test_qos tests.If we want to include these tests, which program should they belong to? Do we wannacreate a new one? > I remember the discussion on the location of tests required by the interop group 2-3 years back (when heat and dns adds-on were added). We all agreed that having tests in tempest plugins is all fine, they do not need to be in Tempest as such. That is why heat and dns or now manila tests stays in their respective plugins.
For neutron also, as there are existing neutron tempest plugin tests we do not need to move them to Tempest and refstack can run it from neutron-tempest-plugin, if we do move then we might break existing upstream or downstream testing scripts running those existing tests from the current location.
-gmann
[1] https://urldefense.com/v3/__https://etherpad.opendev.org/p/neutron-xena-ptg_... 5M$ [etherpad[.]opendev[.]org] > > Thanks, > -- > Martin Kopec > Senior Software Quality Engineer > Red Hat EMEA > > > > > >
-- Slawek Kaplonski Principal Software Engineer Red Hat
Hi, Dnia poniedziałek, 26 kwietnia 2021 17:48:08 CEST Martin Kopec pisze:
Hi everyone,
I would like to further discuss the topics we covered with the neutron team during the PTG [1].
* adding address_group API capability It's tested by tests in neutron-tempest-plugin. First question is if tests which are not directly in tempest can be a part of a non-add-on marketing program? It's possible to move them to tempest though, by the time we do so, could they be marked as advisory?
* Shall we include QoS tempest tests since we don't know what share of vendors enable QoS? Could it be an add-on? These tests are also in neutron-tempest-plugin, I assume we're talking about neutron_tempest_plugin.api.test_qos tests. If we want to include these tests, which program should they belong to? Do we wanna create a new one?
[1] https://etherpad.opendev.org/p/neutron-xena-ptg
Thanks, -- Martin Kopec Senior Software Quality Engineer Red Hat EMEA
First of all, sorry that it took so long for me but I finally looked into Neutron related tests and capabilities and I think we can possibly add few things there: - For "networks-security-groups-CRUD" we can add "address_groups" API. It is now supported by ML2 plugin [1]. In the neutron-tempest-plugin we just have some scenario test [2] but we would probably need also API tests for that, correct? - For networks-l3-CRUD we can optionally add port_forwarding API. This can be added by service plugin [3] so it may not be enabled in all deployments. But maybe there is some "optional feature" category in the RefStack, and if so, this could be included there. Tests for that are in neutron-tempest-plugin [4] and [5]. - There are also 2 other service plugins, which I think could be included as "optional feature" in the RefStack, but IMO don't fit exactly in any of the existing groups. Those are QoS [6] and Trunks [7]. Tests for both are in the neutron-tempest-plugin as well: Qos: [8] and [9], Trunk [10], [11] and [12]. Please let me know what do You think about it and if that would be ok and if You want me to propose some patches with that or maybe You will propose them. [1] https://review.opendev.org/c/openstack/neutron-lib/+/741784[1] [2] https://review.opendev.org/c/openstack/neutron-tempest-plugin/+/777833[2] [3] https://github.com/openstack/neutron/blob/master/neutron/services/portforwar... pf_plugin.py[3] [4] https://github.com/openstack/neutron-tempest-plugin/blob/master/ neutron_tempest_plugin/api/test_port_forwardings.py[4] [5] https://github.com/openstack/neutron-tempest-plugin/blob/master/ neutron_tempest_plugin/api/test_port_forwarding_negative.py[5] [6] https://github.com/openstack/neutron/blob/master/neutron/services/qos/ qos_plugin.py[6] [7] https://github.com/openstack/neutron/blob/master/neutron/services/trunk/ plugin.py[7] [8] https://github.com/openstack/neutron-tempest-plugin/blob/master/ neutron_tempest_plugin/api/test_qos.py[8] [9] https://github.com/openstack/neutron-tempest-plugin/blob/master/ neutron_tempest_plugin/api/test_qos_negative.py[9] [10] https://github.com/openstack/neutron-tempest-plugin/blob/master/ neutron_tempest_plugin/api/test_trunk.py[10] [11] https://github.com/openstack/neutron-tempest-plugin/blob/master/ neutron_tempest_plugin/api/test_trunk_details.py[11] [12] https://github.com/openstack/neutron-tempest-plugin/blob/master/ neutron_tempest_plugin/api/test_trunk_negative.py[12] -- Slawek Kaplonski Principal Software Engineer Red Hat -------- [1] https://review.opendev.org/c/openstack/neutron-lib/+/741784 [2] https://review.opendev.org/c/openstack/neutron-tempest-plugin/+/777833 [3] https://github.com/openstack/neutron/blob/master/neutron/services/portforwar... pf_plugin.py [4] https://github.com/openstack/neutron-tempest-plugin/blob/master/ neutron_tempest_plugin/api/test_port_forwardings.py [5] https://github.com/openstack/neutron-tempest-plugin/blob/master/ neutron_tempest_plugin/api/test_port_forwarding_negative.py [6] https://github.com/openstack/neutron/blob/master/neutron/services/qos/ qos_plugin.py [7] https://github.com/openstack/neutron/blob/master/neutron/services/trunk/plug... [8] https://github.com/openstack/neutron-tempest-plugin/blob/master/ neutron_tempest_plugin/api/test_qos.py [9] https://github.com/openstack/neutron-tempest-plugin/blob/master/ neutron_tempest_plugin/api/test_qos_negative.py [10] https://github.com/openstack/neutron-tempest-plugin/blob/master/ neutron_tempest_plugin/api/test_trunk.py [11] https://github.com/openstack/neutron-tempest-plugin/blob/master/ neutron_tempest_plugin/api/test_trunk_details.py [12] https://github.com/openstack/neutron-tempest-plugin/blob/master/ neutron_tempest_plugin/api/test_trunk_negative.py
Hi Slawek, thanks for getting back to us and sharing new potential tests and capabilities from neutron-tempest-plugin. Let's first discuss tests which are in tempest directly please. We have done an analysis where we have cross checked tests we have in our guidelines with the ones (api and non-admin ones) present in tempest at the tempest checkout we currently use and here are the results: https://etherpad.opendev.org/p/refstack-test-analysis There are 110 and tempest.api.network tests which we don't have in any guideline yet. Could you please have a look at the list of the tests? Would it make sense to include them in a guideline? Would they extend any network capabilities we have in OpenStack Powered Platform program or would we need to create a new one(s)? https://opendev.org/osf/interop/src/branch/master/next.json Thank you, On Mon, 24 May 2021 at 16:33, Slawek Kaplonski <skaplons@redhat.com> wrote:
Hi,
Dnia poniedziałek, 26 kwietnia 2021 17:48:08 CEST Martin Kopec pisze:
Hi everyone,
I would like to further discuss the topics we covered with the neutron team
during
the PTG [1].
* adding address_group API capability
It's tested by tests in neutron-tempest-plugin. First question is if tests
which are
not directly in tempest can be a part of a non-add-on marketing program?
It's possible to move them to tempest though, by the time we do so, could
they be
marked as advisory?
* Shall we include QoS tempest tests since we don't know what share of
vendors
enable QoS? Could it be an add-on?
These tests are also in neutron-tempest-plugin, I assume we're talking about
neutron_tempest_plugin.api.test_qos tests.
If we want to include these tests, which program should they belong to? Do
we wanna
create a new one?
Thanks,
--
Martin Kopec
Senior Software Quality Engineer
Red Hat EMEA
First of all, sorry that it took so long for me but I finally looked into Neutron related tests and capabilities and I think we can possibly add few things there:
- For "networks-security-groups-CRUD" we can add "address_groups" API. It is now supported by ML2 plugin [1]. In the neutron-tempest-plugin we just have some scenario test [2] but we would probably need also API tests for that, correct?
- For networks-l3-CRUD we can optionally add port_forwarding API. This can be added by service plugin [3] so it may not be enabled in all deployments. But maybe there is some "optional feature" category in the RefStack, and if so, this could be included there. Tests for that are in neutron-tempest-plugin [4] and [5].
- There are also 2 other service plugins, which I think could be included as "optional feature" in the RefStack, but IMO don't fit exactly in any of the existing groups. Those are QoS [6] and Trunks [7]. Tests for both are in the neutron-tempest-plugin as well: Qos: [8] and [9], Trunk [10], [11] and [12].
Please let me know what do You think about it and if that would be ok and if You want me to propose some patches with that or maybe You will propose them.
[1] https://review.opendev.org/c/openstack/neutron-lib/+/741784
[2] https://review.opendev.org/c/openstack/neutron-tempest-plugin/+/777833
[3] https://github.com/openstack/neutron/blob/master/neutron/services/portforwar...
[4] https://github.com/openstack/neutron-tempest-plugin/blob/master/neutron_temp...
[5] https://github.com/openstack/neutron-tempest-plugin/blob/master/neutron_temp...
[6] https://github.com/openstack/neutron/blob/master/neutron/services/qos/qos_pl...
[7] https://github.com/openstack/neutron/blob/master/neutron/services/trunk/plug...
[8] https://github.com/openstack/neutron-tempest-plugin/blob/master/neutron_temp...
[9] https://github.com/openstack/neutron-tempest-plugin/blob/master/neutron_temp...
[10] https://github.com/openstack/neutron-tempest-plugin/blob/master/neutron_temp...
[11] https://github.com/openstack/neutron-tempest-plugin/blob/master/neutron_temp...
[12] https://github.com/openstack/neutron-tempest-plugin/blob/master/neutron_temp...
--
Slawek Kaplonski
Principal Software Engineer
Red Hat
-- Martin Kopec Senior Software Quality Engineer Red Hat EMEA
Hi, Dnia środa, 2 czerwca 2021 14:16:50 CEST Martin Kopec pisze:
Hi Slawek,
thanks for getting back to us and sharing new potential tests and capabilities from neutron-tempest-plugin. Let's first discuss tests which are in tempest directly please.
We have done an analysis where we have cross checked tests we have in our guidelines with the ones (api and non-admin ones) present in tempest at the tempest checkout we currently use and here are the results: https://etherpad.opendev.org/p/refstack-test-analysis There are 110 and tempest.api.network tests which we don't have in any guideline yet. Could you please have a look at the list of the tests? Would it make sense to include them in a guideline? Would they extend any network capabilities we have in OpenStack Powered Platform program or would we need to create a new one(s)? https://opendev.org/osf/interop/src/branch/master/next.json
Sure. I took a look at that list today. I think that: * tests from the group tempest.api.network.test_allowed_address_pair could be added to the "networks-l2-CRUD". Allowed_address_pairs is API extension, but it is supported by ML2 plugin since very long time, and should be available in all clouds which are using ML2 plugin. * tests from tempest.api.network.test_dhcp_ipv6 can probably be included in "IPAM drivers" section as now I think all clouds should supports IPv6 :) * tempest.api.network.test_floating_ips - those tests could be probably added to the "Core API L3 extension" section, but I'm not sure what are the guidlines for negative tests in the refstack, * Tests from tempest.api.network.test_networks.BulkNetwork* - are similar to the other L2 CRUD tests but are testing basic bulk CRUD operations for Networks. So It could be IMO included in the "networks-l2-CRUD" section * same for all other tests from tempest.api.network.test_networks and tempest.api.network.test_networks_negative modules * Tests from tempest.api.network.test_ports can probably also be included in the "network-l2-CRUD" section as filtering is supported by core Neutron db modules, * Tests from the tempest.api.network.test_routers module can probably go to the network-l3-CRUD section, That are the tests which I think that may be included somehow in the refstack. But I'm not refstack expert so please forgive me if I included here too many of them or if some of them are not approriate to be there :)
Thank you,
On Mon, 24 May 2021 at 16:33, Slawek Kaplonski <skaplons@redhat.com> wrote:
Hi,
Dnia poniedziałek, 26 kwietnia 2021 17:48:08 CEST Martin Kopec pisze:
Hi everyone,
I would like to further discuss the topics we covered with the neutron
team
during
the PTG [1].
* adding address_group API capability
It's tested by tests in neutron-tempest-plugin. First question is if
tests
which are
not directly in tempest can be a part of a non-add-on marketing program?
It's possible to move them to tempest though, by the time we do so, could
they be
marked as advisory?
* Shall we include QoS tempest tests since we don't know what share of
vendors
enable QoS? Could it be an add-on?
These tests are also in neutron-tempest-plugin, I assume we're talking
about
neutron_tempest_plugin.api.test_qos tests.
If we want to include these tests, which program should they belong to?
Do
we wanna
create a new one?
participants (4)
-
Ghanshyam Mann
-
Kanevsky, Arkady
-
Martin Kopec
-
Slawek Kaplonski