Open Stack

It would be better if we can document the scenarios / cases where
individual tests make sense. As a developer this confuses me the most.

On Thu, Dec 3, 2015 at 10:21 AM, Takashi Yamamoto <yamamoto at midokura.com>
wrote:

> hi,
>
> On Thu, Dec 3, 2015 at 11:19 AM, Armando M. <armamig at gmail.com> wrote:
> > Hi Neutrinos,
> >
> > I would like to share a proposal with you on how we could scale our
> > ever-growing testing needs, and at the same time, provide guidance to the
> > developers who care about the quality of the work they produce, and how
> they
> > can protect it from the dreadful regressions!
> >
> > In [1] you will see a Decision Tree: the objective is to guide us in
> > deciding what is the best testing strategy for a feature, and what type
> of
> > support to expect from our CI engine. In a nutshell we have the following
> > basic options we can choose from:
> >
> > Unit testing
> > Functional testing
> > API testing (*)
> > Fullstack testing
> > Scenario testing (**)
> >
> > The bottom line is: we should tap into our existing infrastructure as
> much
> > as we can to try to minimize the amount of CI moving parts that allow us
> to
> > test the numerous features that Neutron has. Some work has to happen (as
> > highlighted below) before we can make this decision process as smooth as
> > possible. In particular:
> >
> > (*) The API testing framework allows us to execute tempest-style API
> tests
> > against a Neutron live server. Assaf and I are working to address the
> > overlap between Tempest and this framework, and to ensure that there's a
> > clear demarcation to what test belongs to Tempest and what test belongs
> to
> > Neutron codebases respectively. More will follow in another thread, so
> stay
> > tuned. On the other hand, there's some work that needs to go into this
> > testing framework to make the server load up non-default extensions.
> >
> > (**) This is something that Ihar started in [2]. I am still unclear on
> how
> > we intend to leverage this job and yet keep scenario-like tests for
> hardcore
> > Neutron features in the Neutron tree. Ihar is thinking QoS, I may be
> > thinking dpdk, someone else might be thinking to something yet again
> > different (don't get bogged down on the actual examples). We'll iterate
> on
> > [2], but I see it as an integral part of our end-to-end testing strategy.
> > More to follow.
> >
> > As for new additional jobs that may come and go: I believe that we have
> to
> > revisit our approach to introduce experimental and non-voting jobs, and
> how
> > we make them stable and ultimately running as gate jobs and, long term,
> > having the features they tests supplant the ones they are meant to
> supersede
> > (think pecan, or dvr). I have an idea on how to address the non-voting
> > neglect conundrum, but I'll tackle it in another thread.
> >
> > Finally, a word about projects that need testing with Neutron, advanced
> > services and beyond. The same way the neutron-lib initiative is tackling
> the
> > python side of the Neutron codebase as of now, we'll have to work to
> > identify the common pieces of functionality that will allow the reuse of
> the
> > testing machinery across multiple project, in a modular and decouple
> > fashion, so that when Neutron related projects want to integrate with the
> > Neutron CI engine, they will do so using good patterns rather than bad
> ones;
> > I believe that [3] is one of them, and that's why I have been pushing
> back.
>
> can you explain what distinguishes good ones and bad ones?
>
> >
> > Feedback welcome, as I am sure I may overlooked something.
> >
> > Cheers,
> > Armando
> >
> > [1] https://wiki.openstack.org/wiki/Network/Testing
> > [2] https://review.openstack.org/#/c/247697/
> > [3] https://review.openstack.org/#/c/242925/
> >
> >
> __________________________________________________________________________
> > OpenStack Development Mailing List (not for usage questions)
> > Unsubscribe:
> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20151203/81b2ae6c/attachment-0001.html>