Open Stack

Armando M. <armamig at gmail.com> wrote:

> Hi Neutrinos,
>
> I would like to share a proposal with you on how we could scale our  
> ever-growing testing needs, and at the same time, provide guidance to the  
> developers who care about the quality of the work they produce, and how  
> they can protect it from the dreadful regressions!
>
> In [1] you will see a Decision Tree: the objective is to guide us in  
> deciding what is the best testing strategy for a feature, and what type  
> of support to expect from our CI engine. In a nutshell we have the  
> following basic options we can choose from:
> 	• Unit testing
> 	• Functional testing
> 	• API testing (*)
> 	• Fullstack testing
> 	• Scenario testing (**)
> The bottom line is: we should tap into our existing infrastructure as  
> much as we can to try to minimize the amount of CI moving parts that  
> allow us to test the numerous features that Neutron has. Some work has to  
> happen (as highlighted below) before we can make this decision process as  
> smooth as possible. In particular:
>
> (*) The API testing framework allows us to execute tempest-style API  
> tests against a Neutron live server. Assaf and I are working to address  
> the overlap between Tempest and this framework, and to ensure that  
> there's a clear demarcation to what test belongs to Tempest and what test  
> belongs to Neutron codebases respectively. More will follow in another  
> thread, so stay tuned. On the other hand, there's some work that needs to  
> go into this testing framework to make the server load up non-default  
> extensions.
>

I think we already load up "non-default" extensions as needed (think QoS or  
LBaaS):
https://github.com/openstack/neutron/blob/master/neutron/tests/contrib/gate_hook.sh#L25

> (**) This is something that Ihar started in [2]. I am still unclear on  
> how we intend to leverage this job and yet keep scenario-like tests for  
> hardcore Neutron features in the Neutron tree. Ihar is thinking QoS, I  
> may be thinking dpdk, someone else might be thinking to something yet  
> again different (don't get bogged down on the actual examples). We'll  
> iterate on [2], but I see it as an integral part of our end-to-end  
> testing strategy. More to follow.
>

Side note: I don’t believe dpdk job would need tempest in the first place  
since it does not touch API in any way, so could and should be covered with  
functional/fullstack just fine. So let’s better talk e.g. ‘flavors’ here if  
we need another example in addition to QoS.

> As for new additional jobs that may come and go: I believe that we have  
> to revisit our approach to introduce experimental and non-voting jobs,  
> and how we make them stable and ultimately running as gate jobs and, long  
> term, having the features they tests supplant the ones they are meant to  
> supersede (think pecan, or dvr). I have an idea on how to address the  
> non-voting neglect conundrum, but I'll tackle it in another thread.
>
> Finally, a word about projects that need testing with Neutron, advanced  
> services and beyond. The same way the neutron-lib initiative is tackling  
> the python side of the Neutron codebase as of now, we'll have to work to  
> identify the common pieces of functionality that will allow the reuse of  
> the testing machinery across multiple project, in a modular and decouple  
> fashion, so that when Neutron related projects want to integrate with the  
> Neutron CI engine, they will do so using good patterns rather than bad  
> ones; I believe that [3] is one of them, and that's why I have been  
> pushing back.
>

Apart from in-neutron work on common testing infrastructure, we need to  
think of how we manage gate jobs for all stadium projects. F.e. most of  
them are forced into adopting a huge hack-around called  
tools/tox_install.sh that would install proper neutron for them. Instead of  
reimplementing the script for every subproject, we better work with infra  
to define common job macros that would manage neutron installation for us.  
It would simplify our lives a lot if only we would not need to maintain the  
script in each and every stadium repo.

> Feedback welcome, as I am sure I may overlooked something.
>
> Cheers,
> Armando
>
> [1] https://wiki.openstack.org/wiki/Network/Testing
> [2] https://review.openstack.org/#/c/247697/
> [3] https://review.openstack.org/#/c/242925/
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev