[dev][nova][placement][qa] opinion on adding placement tests support in Tempest
Hi All, As you all know, placement tests (in nova repo or in new placement repo) are implemented using gabbi functional test[1]. There is a patch from amodi about adding placement test in Tempest [2]. Currently, Tempest does not support the placement endpoints. To support placement API in Tempest, it does not need much work (i have listed the todo items in gerrit patch.). Before we start or proceed with the discussion in QA, i would like to get the nova(placement) team opinion on adding the placement support in Tempest. Obviously, we should not duplicate the testing effort between what existing gabbi tests cover or what going to be added in Tempest which we can take care while adding the new tests. [1] https://github.com/openstack/nova/tree/5f648dda49a6d5fe5ecfd7dddcb5f7dc3d6b5... https://github.com/openstack/placement/tree/34f03c297e28ae2c46ab11debdb4bbe6... [2] https://review.openstack.org/#/c/621645/2 -gmann
On Tue, 4 Dec 2018, Ghanshyam Mann wrote:
Before we start or proceed with the discussion in QA, i would like to get the nova(placement) team opinion on adding the placement support in Tempest. Obviously, we should not duplicate the testing effort between what existing gabbi tests cover or what going to be added in Tempest which we can take care while adding the new tests.
My feeling on this is that what should be showing up in tempest with regard to placement tests are things that demonstrate and prove end to end scenarios in which placement is involved as a critical part, but is in the background. For example, things like the emerging minimal bandwidth functionality that involves all three of nova, placement and neutron. I don't think we need extensive testing in Tempest of the placement API itself, as that's already well covered by the existing functional tests, nor do I think it makes much sense to cover the common scheduling scenarios between nova and placement as those are also well covered and will continue to be covered even with placement extracted [1]. Existing Tempests tests that do things like launching, resizing, migrating servers already touch placement so may be sufficient. If we wanted to make these more complete adding verification of resource providers and their inventories before and after the tests might be useful. [1] https://review.openstack.org/#/c/617941/ -- Chris Dent ٩◔̯◔۶ https://anticdent.org/ freenode: cdent tw: @anticdent
Hi, I started to add a scenario test to neutron-tempest-plugin for bandwidth and for that some placement client to tempest. I plan to upload (possibly as WIP) first the client part and after that the scenario for bandwidth and for routed networks as that one uses placement as well. Regards Lajos On 2018. 12. 04. 12:13, Chris Dent wrote:
On Tue, 4 Dec 2018, Ghanshyam Mann wrote:
Before we start or proceed with the discussion in QA, i would like to get the nova(placement) team opinion on adding the placement support in Tempest. Obviously, we should not duplicate the testing effort between what existing gabbi tests cover or what going to be added in Tempest which we can take care while adding the new tests.
My feeling on this is that what should be showing up in tempest with regard to placement tests are things that demonstrate and prove end to end scenarios in which placement is involved as a critical part, but is in the background. For example, things like the emerging minimal bandwidth functionality that involves all three of nova, placement and neutron.
I don't think we need extensive testing in Tempest of the placement API itself, as that's already well covered by the existing functional tests, nor do I think it makes much sense to cover the common scheduling scenarios between nova and placement as those are also well covered and will continue to be covered even with placement extracted [1].
Existing Tempests tests that do things like launching, resizing, migrating servers already touch placement so may be sufficient. If we wanted to make these more complete adding verification of resource providers and their inventories before and after the tests might be useful.
On 12/04/2018 06:13 AM, Chris Dent wrote:
On Tue, 4 Dec 2018, Ghanshyam Mann wrote:
Before we start or proceed with the discussion in QA, i would like to get the nova(placement) team opinion on adding the placement support in Tempest. Obviously, we should not duplicate the testing effort between what existing gabbi tests cover or what going to be added in Tempest which we can take care while adding the new tests.
My feeling on this is that what should be showing up in tempest with regard to placement tests are things that demonstrate and prove end to end scenarios in which placement is involved as a critical part, but is in the background. For example, things like the emerging minimal bandwidth functionality that involves all three of nova, placement and neutron.
I don't think we need extensive testing in Tempest of the placement API itself, as that's already well covered by the existing functional tests, nor do I think it makes much sense to cover the common scheduling scenarios between nova and placement as those are also well covered and will continue to be covered even with placement extracted [1].
Existing Tempests tests that do things like launching, resizing, migrating servers already touch placement so may be sufficient. If we wanted to make these more complete adding verification of resource providers and their inventories before and after the tests might be useful.
Fully agree with Chris' assessment on this. Best, -jay
On 12/04/2018 06:13 AM, Chris Dent wrote:
On Tue, 4 Dec 2018, Ghanshyam Mann wrote:
Before we start or proceed with the discussion in QA, i would like to get the nova(placement) team opinion on adding the placement support in Tempest. Obviously, we should not duplicate the testing effort between what existing gabbi tests cover or what going to be added in Tempest which we can take care while adding the new tests.
My feeling on this is that what should be showing up in tempest with regard to placement tests are things that demonstrate and prove end to end scenarios in which placement is involved as a critical part, but is in the background. For example, things like the emerging minimal bandwidth functionality that involves all three of nova, placement and neutron.
I don't think we need extensive testing in Tempest of the placement API itself, as that's already well covered by the existing functional tests, nor do I think it makes much sense to cover the common scheduling scenarios between nova and placement as those are also well covered and will continue to be covered even with placement extracted [1].
Existing Tempests tests that do things like launching, resizing, migrating servers already touch placement so may be sufficient. If we wanted to make these more complete adding verification of resource providers and their inventories before and after the tests might be useful.
Fully agree with Chris' assessment on this.
I don't disagree either. However, I do think that there are cases where it may make sense to be _able_ to hit the placement endpoint from tempest in order to verify that certain things are happening, even in a scenario that involves other services. For example, if we're testing nova's request filter stuff, we may very well need to hit the placement endpoint to validate that aggregate information is being mirrored, and/or that adding a trait to a provider properly results in some scheduling behavior. So, if the question is "should a tempest test be able to hit the placement endpoint?" I would say "yes". If the question is "should tempest have tests that only hit placement to validate proper behavior", I'd agree that functional tests in placement probably cover that sufficiently. I *think* that gmann's question in the email was actually about placement endpoint support, which is the former, and I think is probably legit. --Dan
On Tue, 4 Dec 2018, Dan Smith wrote:
On 12/04/2018 06:13 AM, Chris Dent wrote:
Existing Tempests tests that do things like launching, resizing, migrating servers already touch placement so may be sufficient. If we wanted to make these more complete adding verification of resource providers and their inventories before and after the tests might be useful.
[snip]
I don't disagree either. However, I do think that there are cases where it may make sense to be _able_ to hit the placement endpoint from tempest in order to verify that certain things are happening, even in a scenario that involves other services.
[snip] Based on conversation with Dan in IRC, we decided it might be useful to clarify that Dan and I are in agreement. It had seemed to me that he was saying something different from me, but we're both basically saying "yes, tempest needs to be able to talk to placement to confirm what it's holding because that's useful sometimes" and "no, tempest doesn't need to verify the workings of placement api itself". Which boils out to this:
I *think* that gmann's question in the email was actually about placement endpoint support, which is the former, and I think is probably legit.
Yes. -- Chris Dent ٩◔̯◔۶ https://anticdent.org/ freenode: cdent tw: @anticdent
Great! There is already a patch from Lajos [1]. I'd like resource_provider_aggregates_client to be added too. (/resource_providers/{uuid}/aggregates) [1] https://review.openstack.org/#/c/622316/ On Tue, Dec 4, 2018 at 1:32 PM Chris Dent <cdent+os@anticdent.org> wrote:
On Tue, 4 Dec 2018, Dan Smith wrote:
On 12/04/2018 06:13 AM, Chris Dent wrote:
Existing Tempests tests that do things like launching, resizing, migrating servers already touch placement so may be sufficient. If we wanted to make these more complete adding verification of resource providers and their inventories before and after the tests might be useful.
[snip]
I don't disagree either. However, I do think that there are cases where it may make sense to be _able_ to hit the placement endpoint from tempest in order to verify that certain things are happening, even in a scenario that involves other services.
[snip]
Based on conversation with Dan in IRC, we decided it might be useful to clarify that Dan and I are in agreement. It had seemed to me that he was saying something different from me, but we're both basically saying "yes, tempest needs to be able to talk to placement to confirm what it's holding because that's useful sometimes" and "no, tempest doesn't need to verify the workings of placement api itself".
Which boils out to this:
I *think* that gmann's question in the email was actually about placement endpoint support, which is the former, and I think is probably legit.
Yes.
-- Chris Dent ٩◔̯◔۶ https://anticdent.org/ freenode: cdent tw: @anticdent
Thanks, I uploaded the patch only to start the conversation early, I plan to add all the necessary methods to cover the needs now for bandwidth, and have a basic framework for adding more things later. Regards Lajos On 2018. 12. 04. 19:50, Archit Modi wrote: Great! There is already a patch from Lajos [1]. I'd like resource_provider_aggregates_client to be added too. (/resource_providers/{uuid}/aggregates) [1] https://review.openstack.org/#/c/622316/ On Tue, Dec 4, 2018 at 1:32 PM Chris Dent <cdent+os@anticdent.org<mailto:cdent%2Bos@anticdent.org>> wrote: On Tue, 4 Dec 2018, Dan Smith wrote:
On 12/04/2018 06:13 AM, Chris Dent wrote:
Existing Tempests tests that do things like launching, resizing, migrating servers already touch placement so may be sufficient. If we wanted to make these more complete adding verification of resource providers and their inventories before and after the tests might be useful.
[snip]
I don't disagree either. However, I do think that there are cases where it may make sense to be _able_ to hit the placement endpoint from tempest in order to verify that certain things are happening, even in a scenario that involves other services.
[snip] Based on conversation with Dan in IRC, we decided it might be useful to clarify that Dan and I are in agreement. It had seemed to me that he was saying something different from me, but we're both basically saying "yes, tempest needs to be able to talk to placement to confirm what it's holding because that's useful sometimes" and "no, tempest doesn't need to verify the workings of placement api itself". Which boils out to this:
I *think* that gmann's question in the email was actually about placement endpoint support, which is the former, and I think is probably legit.
Yes. -- Chris Dent ٩◔̯◔۶ https://anticdent.org/ freenode: cdent tw: @anticdent
---- On Wed, 05 Dec 2018 03:30:56 +0900 Chris Dent <cdent+os@anticdent.org> wrote ----
On Tue, 4 Dec 2018, Dan Smith wrote:
On 12/04/2018 06:13 AM, Chris Dent wrote:
Existing Tempests tests that do things like launching, resizing, migrating servers already touch placement so may be sufficient. If we wanted to make these more complete adding verification of resource providers and their inventories before and after the tests might be useful.
[snip]
I don't disagree either. However, I do think that there are cases where it may make sense to be _able_ to hit the placement endpoint from tempest in order to verify that certain things are happening, even in a scenario that involves other services.
[snip]
Based on conversation with Dan in IRC, we decided it might be useful to clarify that Dan and I are in agreement. It had seemed to me that he was saying something different from me, but we're both basically saying "yes, tempest needs to be able to talk to placement to confirm what it's holding because that's useful sometimes" and "no, tempest doesn't need to verify the workings of placement api itself".
Yeah, that is what we wanted. I think I mentioned that in my original mail but that sentence was not so clear. There will not be any overlap/duplicate tests between what existing functional test cover and what Tempest is going to cover. Tempest will need to talk to placement for extra verification in Tempest tests. Verify only placement API working is not in scope of Tempest. Which is nothing but : - Adding placement service clients with unit test coverage only - Those service client will be added on need basis. We do not want to maintain the unused service clients. - Use those service clients to talk to placement for extra verification in tests.
Which boils out to this:
I *think* that gmann's question in the email was actually about placement endpoint support, which is the former, and I think is probably legit.
Yes.
Great, we all are on same page now. -gmann
-- Chris Dent ٩◔̯◔۶ https://anticdent.org/ freenode: cdent tw: @anticdent
participants (6)
-
Archit Modi
-
Chris Dent
-
Dan Smith
-
Ghanshyam Mann
-
Jay Pipes
-
Lajos Katona