Open Stack

hi,

On Thu, Dec 3, 2015 at 11:19 AM, Armando M. <armamig at gmail.com> wrote:
> Hi Neutrinos,
>
> I would like to share a proposal with you on how we could scale our
> ever-growing testing needs, and at the same time, provide guidance to the
> developers who care about the quality of the work they produce, and how they
> can protect it from the dreadful regressions!
>
> In [1] you will see a Decision Tree: the objective is to guide us in
> deciding what is the best testing strategy for a feature, and what type of
> support to expect from our CI engine. In a nutshell we have the following
> basic options we can choose from:
>
> Unit testing
> Functional testing
> API testing (*)
> Fullstack testing
> Scenario testing (**)
>
> The bottom line is: we should tap into our existing infrastructure as much
> as we can to try to minimize the amount of CI moving parts that allow us to
> test the numerous features that Neutron has. Some work has to happen (as
> highlighted below) before we can make this decision process as smooth as
> possible. In particular:
>
> (*) The API testing framework allows us to execute tempest-style API tests
> against a Neutron live server. Assaf and I are working to address the
> overlap between Tempest and this framework, and to ensure that there's a
> clear demarcation to what test belongs to Tempest and what test belongs to
> Neutron codebases respectively. More will follow in another thread, so stay
> tuned. On the other hand, there's some work that needs to go into this
> testing framework to make the server load up non-default extensions.
>
> (**) This is something that Ihar started in [2]. I am still unclear on how
> we intend to leverage this job and yet keep scenario-like tests for hardcore
> Neutron features in the Neutron tree. Ihar is thinking QoS, I may be
> thinking dpdk, someone else might be thinking to something yet again
> different (don't get bogged down on the actual examples). We'll iterate on
> [2], but I see it as an integral part of our end-to-end testing strategy.
> More to follow.
>
> As for new additional jobs that may come and go: I believe that we have to
> revisit our approach to introduce experimental and non-voting jobs, and how
> we make them stable and ultimately running as gate jobs and, long term,
> having the features they tests supplant the ones they are meant to supersede
> (think pecan, or dvr). I have an idea on how to address the non-voting
> neglect conundrum, but I'll tackle it in another thread.
>
> Finally, a word about projects that need testing with Neutron, advanced
> services and beyond. The same way the neutron-lib initiative is tackling the
> python side of the Neutron codebase as of now, we'll have to work to
> identify the common pieces of functionality that will allow the reuse of the
> testing machinery across multiple project, in a modular and decouple
> fashion, so that when Neutron related projects want to integrate with the
> Neutron CI engine, they will do so using good patterns rather than bad ones;
> I believe that [3] is one of them, and that's why I have been pushing back.

can you explain what distinguishes good ones and bad ones?

>
> Feedback welcome, as I am sure I may overlooked something.
>
> Cheers,
> Armando
>
> [1] https://wiki.openstack.org/wiki/Network/Testing
> [2] https://review.openstack.org/#/c/247697/
> [3] https://review.openstack.org/#/c/242925/
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>