[openstack-dev] [Neutron] Scaling our testing needs
vikschw at gmail.com
Thu Dec 3 05:24:25 UTC 2015
It would be better if we can document the scenarios / cases where
individual tests make sense. As a developer this confuses me the most.
On Thu, Dec 3, 2015 at 10:21 AM, Takashi Yamamoto <yamamoto at midokura.com>
> On Thu, Dec 3, 2015 at 11:19 AM, Armando M. <armamig at gmail.com> wrote:
> > Hi Neutrinos,
> > I would like to share a proposal with you on how we could scale our
> > ever-growing testing needs, and at the same time, provide guidance to the
> > developers who care about the quality of the work they produce, and how
> > can protect it from the dreadful regressions!
> > In  you will see a Decision Tree: the objective is to guide us in
> > deciding what is the best testing strategy for a feature, and what type
> > support to expect from our CI engine. In a nutshell we have the following
> > basic options we can choose from:
> > Unit testing
> > Functional testing
> > API testing (*)
> > Fullstack testing
> > Scenario testing (**)
> > The bottom line is: we should tap into our existing infrastructure as
> > as we can to try to minimize the amount of CI moving parts that allow us
> > test the numerous features that Neutron has. Some work has to happen (as
> > highlighted below) before we can make this decision process as smooth as
> > possible. In particular:
> > (*) The API testing framework allows us to execute tempest-style API
> > against a Neutron live server. Assaf and I are working to address the
> > overlap between Tempest and this framework, and to ensure that there's a
> > clear demarcation to what test belongs to Tempest and what test belongs
> > Neutron codebases respectively. More will follow in another thread, so
> > tuned. On the other hand, there's some work that needs to go into this
> > testing framework to make the server load up non-default extensions.
> > (**) This is something that Ihar started in . I am still unclear on
> > we intend to leverage this job and yet keep scenario-like tests for
> > Neutron features in the Neutron tree. Ihar is thinking QoS, I may be
> > thinking dpdk, someone else might be thinking to something yet again
> > different (don't get bogged down on the actual examples). We'll iterate
> > , but I see it as an integral part of our end-to-end testing strategy.
> > More to follow.
> > As for new additional jobs that may come and go: I believe that we have
> > revisit our approach to introduce experimental and non-voting jobs, and
> > we make them stable and ultimately running as gate jobs and, long term,
> > having the features they tests supplant the ones they are meant to
> > (think pecan, or dvr). I have an idea on how to address the non-voting
> > neglect conundrum, but I'll tackle it in another thread.
> > Finally, a word about projects that need testing with Neutron, advanced
> > services and beyond. The same way the neutron-lib initiative is tackling
> > python side of the Neutron codebase as of now, we'll have to work to
> > identify the common pieces of functionality that will allow the reuse of
> > testing machinery across multiple project, in a modular and decouple
> > fashion, so that when Neutron related projects want to integrate with the
> > Neutron CI engine, they will do so using good patterns rather than bad
> > I believe that  is one of them, and that's why I have been pushing
> can you explain what distinguishes good ones and bad ones?
> > Feedback welcome, as I am sure I may overlooked something.
> > Cheers,
> > Armando
> >  https://wiki.openstack.org/wiki/Network/Testing
> >  https://review.openstack.org/#/c/247697/
> >  https://review.openstack.org/#/c/242925/
> > OpenStack Development Mailing List (not for usage questions)
> > Unsubscribe:
> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
-------------- next part --------------
An HTML attachment was scrubbed...
More information about the OpenStack-dev