Open Stack

On Wed, Mar 26, 2014 at 9:02 PM, Maru Newby <marun at redhat.com> wrote:

>
> On Mar 26, 2014, at 12:59 PM, Joe Gordon <joe.gordon0 at gmail.com> wrote:
>
> >
> >
> >
> > On Tue, Mar 25, 2014 at 1:49 AM, Maru Newby <marun at redhat.com> wrote:
> >
> > On Mar 21, 2014, at 9:01 AM, David Kranz <dkranz at redhat.com> wrote:
> >
> > > On 03/20/2014 04:19 PM, Rochelle.RochelleGrober wrote:
> > >>
> > >>> -----Original Message-----
> > >>> From: Malini Kamalambal [mailto:malini.kamalambal at RACKSPACE.COM]
> > >>> Sent: Thursday, March 20, 2014 12:13 PM
> > >>>
> > >>> 'project specific functional testing' in the Marconi context is
> > >>> treating
> > >>> Marconi as a complete system, making Marconi API calls & verifying
> the
> > >>> response - just like an end user would, but without keystone. If one
> of
> > >>> these tests fail, it is because there is a bug in the Marconi code ,
> > >>> and
> > >>> not because its interaction with Keystone caused it to fail.
> > >>>
> > >>> "That being said there are certain cases where having a project
> > >>> specific
> > >>> functional test makes sense. For example swift has a functional test
> > >>> job
> > >>> that
> > >>> starts swift in devstack. But, those things are normally handled on a
> > >>> per
> > >>> case
> > >>> basis. In general if the project is meant to be part of the larger
> > >>> OpenStack
> > >>> ecosystem then Tempest is the place to put functional testing. That
> way
> > >>> you know
> > >>> it works with all of the other components. The thing is in openstack
> > >>> what
> > >>> seems
> > >>> like a project isolated functional test almost always involves
> another
> > >>> project
> > >>> in real use cases. (for example keystone auth with api requests)
> > >>>
> > >>> "
> >
> >
> >
> > >>>
> > >>> One of the concerns we heard in the review was 'having the functional
> > >>> tests elsewhere (I.e within the project itself) does not count and
> they
> > >>> have to be in Tempest'.
> > >>> This has made us as a team wonder if we should migrate all our
> > >>> functional
> > >>> tests to Tempest.
> > >>> But from Matt's response, I think it is reasonable to continue in our
> > >>> current path & have the functional tests in Marconi coexist  along
> with
> > >>> the tests in Tempest.
> > >>>
> > >> I think that what is being asked, really is that the functional tests
> could be a single set of tests that would become a part of the tempest
> repository and that these tests would have an ENV variable as part of the
> configuration that would allow either "no Keystone" or "Keystone" or some
> such, if that is the only configuration issue that separates running the
> tests isolated vs. integrated.  The functional tests need to be as much as
> possible a single set of tests to reduce duplication and remove the
> likelihood of two sets getting out of sync with each other/development.  If
> they only run in the integrated environment, that's ok, but if you want to
> run them isolated to make debugging easier, then it should be a
> configuration option and a separate test job.
> > >>
> > >> So, if my assumptions are correct, QA only requires functional tests
> for integrated runs, but if the project QAs/Devs want to run isolated for
> dev and devtest purposes, more power to them.  Just keep it a single set of
> functional tests and put them in the Tempest repository so that if a
> failure happens, anyone can find the test and do the debug work without
> digging into a separate project repository.
> > >>
> > >> Hopefully, the tests as designed could easily take a new
> configuration directive and a short bit of work with OS QA will get the
> integrated FTs working as well as the isolated ones.
> > >>
> > >> --Rocky
> > > This issue has been much debated. There are some active members of our
> community who believe that all the functional tests should live outside of
> tempest in the projects, albeit with the same idea that such tests could be
> run either as part of today's "real" tempest runs or mocked in various ways
> to allow component isolation or better performance. Maru Newby posted a
> patch with an example of one way to do this but I think it expired and I
> don't have a pointer.
> >
> > I think the best place for functional api tests to be maintained is in
> the projects themselves.  The domain expertise required to write api tests
> is likely to be greater among project resources, and they should be tasked
> with writing api tests pre-merge.  The current 'merge-first, test-later'
> procedure of maintaining api tests in the Tempest repo makes that
> impossible.  Worse, the cost of developing functional api tests is higher
> in the integration environment that is the Tempest default.
> >
> >
> > If an API is made and documented properly what domain expertise would be
> needed to use it? The opposite is true for tempest and the tests
> themselves. The tempest team focuses on just tests so they know how to
> write good tests and are able to leverage common underlying framework code.
>
> Given that documentation is typically finalized only late in the cycle,
> are you suggesting that we forego api testing until possibly well after the
> code has been written?  Plus, it is a rare api that doesn't have to evolve
> in response to real-world experience with early implementations.  The
> sooner we can write functional api tests, the sooner we can identify
> shortcomings that need to be addressed - and the less costly they will be
> to fix.
>


I agree with this sentiment, but I think this can be solved via tempest as
well.


> As to expertise, would you agree that we have a much greater need for QA
> expertise than the existing team can hope to meet?  I would argue that this
> is the case, and believe that the more we can diffuse a testing mentality
> from the QA team to the projects, the better off OpenStack will be.
>  Tempest has proven a very effective guardian against regression, but that
> role can only grow more difficult with time and a lack of focus will
> complicate the effort unnecessarily.
>


If Tempest doesn't have enough human resources, then teams should work
closer with tempest to fix that.


>
> As to framework, the talk of 'libifying' Tempest promises to allow the
> projects the same benefit of common framework that Tempest currently enjoys.
>
> >
> > Yes 'merge-first, test-later' is bad. But There are other ways of fixing
> this then moving the tests out of tempest. Hopefully cross project
> dependencies in zuul will help make this workflow easier.
> >
> >
> >
> > The patch in question [1] proposes allowing pre-merge functional api
> test maintenance and test reuse in an integration environment.
> >
> >
> > m.
> >
> > 1: https://review.openstack.org/#/c/72585/
> >
> > > IMO there are valid arguments on both sides, but I hope every one
> could agree that functional tests should not be arbitrarily split between
> projects and tempest as they are now. The Tempest README states a desire
> for "complete coverage of the OpenStack API" but Tempest is not close to
> that. We have been discussing and then ignoring this issue for some time
> but I think the recent action to say that Tempest will be used to determine
> if something can use the OpenStack trademark will force more completeness
> on tempest (more tests, that is). I think we need to resolve this issue but
> it won't be easy and modifying existing api tests to be more flexible will
> be a lot of work. But at least new projects could get on the right path
> sooner.
> >
> >
> > This touches on another point that I forgot to bring up before.  As a
> cloud deployer I want a test suite that allows me to exercise all of my
> APIs to make sure everything is working. Although OpenStack runs extensive
> tests upstream, upstream cannot test every possible configuration, so as a
> deployer I would want rerun the same test suite. This is exactly why
> TripleO is getting ready to run tempest. Does this test suite have to just
> tempest? Perhaps not, but if we go down the road of moving more API tests
> into individual projects I think we still need easy way for a cloud
> deployer to run all the API tests against a live cloud from a single
> interface.
>
> I don't think project-maintained functional api tests would in any way
> preclude Tempest from continuing to be the go-to test suite for
> deployers/operators.  There is nothing stopping project-maintained
> functional api tests from being consumed by Tempest.  Tempest could
> explicitly depend on the projects, either the whole package or a special
> package that contained only the api tests.  Or Tempest could copy the api
> tests, as is done with oslo incubator.  In either case, the process could
> be purely mechanical.
>


I agree with you here, if we do decide to go down the route you are
proposing (functional tests outside of tempest repo), then having this
mechanism sorted out should be a pre-requisite.



>
>
> m.
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20140327/769c781d/attachment.html>