Open Stack

On Thu, Mar 20, 2014 at 1:19 PM, Rochelle.RochelleGrober <
rochelle.grober at huawei.com> wrote:

>
>
> > -----Original Message-----
> > From: Malini Kamalambal [mailto:malini.kamalambal at RACKSPACE.COM]
> > Sent: Thursday, March 20, 2014 12:13 PM
> >
> > 'project specific functional testing' in the Marconi context is
> > treating
> > Marconi as a complete system, making Marconi API calls & verifying the
> > response - just like an end user would, but without keystone. If one of
> > these tests fail, it is because there is a bug in the Marconi code ,
> > and
> > not because its interaction with Keystone caused it to fail.
> >
> > "That being said there are certain cases where having a project
> > specific
> > functional test makes sense. For example swift has a functional test
> > job
> > that
> > starts swift in devstack. But, those things are normally handled on a
> > per
> > case
> > basis. In general if the project is meant to be part of the larger
> > OpenStack
> > ecosystem then Tempest is the place to put functional testing. That way
> > you know
> > it works with all of the other components. The thing is in openstack
> > what
> > seems
> > like a project isolated functional test almost always involves another
> > project
> > in real use cases. (for example keystone auth with api requests)
> >
> > "
> >
> > One of the concerns we heard in the review was 'having the functional
> > tests elsewhere (I.e within the project itself) does not count and they
> > have to be in Tempest'.
> > This has made us as a team wonder if we should migrate all our
> > functional
> > tests to Tempest.
> > But from Matt's response, I think it is reasonable to continue in our
> > current path & have the functional tests in Marconi coexist  along with
> > the tests in Tempest.
> >
>
>
While there always exceptions to this rule, in general I think functional
tests belong in tempest for all of the reasons discussed above.



> I think that what is being asked, really is that the functional tests
> could be a single set of tests that would become a part of the tempest
> repository and that these tests would have an ENV variable as part of the
> configuration that would allow either "no Keystone" or "Keystone" or some
> such, if that is the only configuration issue that separates running the
> tests isolated vs. integrated.  The functional tests need to be as much as
> possible a single set of tests to reduce duplication and remove the
> likelihood of two sets getting out of sync with each other/development.  If
> they only run in the integrated environment, that's ok, but if you want to
> run them isolated to make debugging easier, then it should be a
> configuration option and a separate test job.
>
> So, if my assumptions are correct, QA only requires functional tests for
> integrated runs, but if the project QAs/Devs want to run isolated for dev
> and devtest purposes, more power to them.  Just keep it a single set of
> functional tests and put them in the Tempest repository so that if a
> failure happens, anyone can find the test and do the debug work without
> digging into a separate project repository.
>
> Hopefully, the tests as designed could easily take a new configuration
> directive and a short bit of work with OS QA will get the integrated FTs
> working as well as the isolated ones.
>
> --Rocky
>
> > On 3/20/14 1:59 PM, "Matthew Treinish" <mtreinish at kortar.org> wrote:
> >
> > >On Thu, Mar 20, 2014 at 11:35:15AM +0000, Malini Kamalambal wrote:
> > >> Hello all,
> > >>
> > >> I have been working on adding tests in Tempest for Marconi, for the
> > >>last few months.
> > >> While there are many amazing people to work with, the process has
> > been
> > >>more difficult than I expected.
> > >>
> > >> Couple of pain-points and suggestions to make the process easier for
> > >>myself & future contributors.
> > >>
> > >> 1. The QA requirements for a project to graduate needs details
> > beyond
> > >>the "Project must have a *basic* devstack-gate job set up"
> > >> 2. The scope of Tempest needs clarification  - what tests should be
> > in
> > >>Tempest vs. in the individual projects? Or should they be in both
> > >>tempest and the project?
> > >>
> > >> See details below.
> > >>
> > >> 1. There is little documentation on graduation requirement from a QA
> > >>perspective beyond 'Project must have a basic devstack-gate job set
> > up'.
> > >>
> > >> As a result, I hear different interpretations on what a basic
> > devstack
> > >>gate job is.
> > >> This topic was discussed in one of the QA meetings a few weeks back
> > [1].
> > >> Based on the discussion there, having a basic job - such as one that
> > >>will let us know 'if a keystone change broke marconi' was  good
> > enough.
> > >> My efforts in getting Marconi meet graduation requirements w.r.t
> > >>Tempest was based on the above discussion.
> > >>
> > >> However, my conversations with the TC during Marconi's graduation
> > >>review  lead me to believe that these requirements aren't yet
> > formalized.
> > >> We were told that we needed to have more test coverage in tempest, &
> > >>having them elsewhere (i.e. functional tests in the Marconi project
> > >>itself) was not good enough.
> > >
> > >So having only looked at the Marconi ML thread and not the actual TC
> > >meeting
> > >minutes I might be missing the whole picture. But, from what I saw
> > when I
> > >looked
> > >at both a marconi commit and a tempest commit is that there is no
> > gating
> > >marconi
> > >devstack-gate job on marconi commits. It's only non-voting in the
> > check
> > >pipeline.
> > >Additionally, there isn't a non-voting job on tempest or devstack-
> > gate.
> > >For
> > >example, look at how savanna has it's tempest jobs setup and this is
> > what
> > >marconi
> > >needs to have.
> > >
> > >>
> > >> I will never debate the value of having good test coverage - after
> > all
> > >>I define myself professionally as a QA ;)
> > >> I am proud of the unit and functional test suites & the test
> > coverage
> > >>we have in Marconi [2].
> > >> Marconi team is continuing its efforts in this direction.
> > >> We are looking forward to adding more tests in Tempest and making
> > sure
> > >>Marconi is in par with the community standards.
> > >>
> > >> But what frustrates me is that the test requirements seem to evolve,
> > >>catching  new contributors by surprise.
> > >>
> > >> It will really help to have these requirements documented in detail
> > -
> > >>answering at least the following questions,
> > >> a. What tests are needed to graduate - API, Scenario, CLI?
> > >> b. How much coverage is good enough to graduate?
> > >>
> > >> That will make sure that contributors focus their time & energy in
> > the
> > >>right direction.
> > >> I am willing to lead the effort to document the QA-level graduation
> > >>requirements for a project and help solidify them.
> > >
> > >Testing contributions will always be an iterative process. The actual
> > test
> > >coverage doesn't matter as much up front. The graduation requirement
> > as I
> > >understood it was just to have the glue in place and to verify that
> > >everything
> > >runs. As long as there is steady contribution and interaction from the
> > >marconi
> > >community with tempest IMO that matters far more then actually having
> > >complete
> > >coverage upfront.
> > >
> > >>
> > >> 2. Clarify the scope of Tempest  - what tests should be in Tempest
> > vs
> > >>in the individual projects ?
> > >>
> > >> It sounds like the scope of tempest is to make sure that,
> > >> a. Projects are functionally tested (AND)
> > >> b. Openstack components (a.k.a projects) do not have integration
> > issues.
> > >>
> > >> Assuming my understanding is correct, does it make sense to have the
> > >>project specific functional tests in Tempest?
> > >> Troubleshooting failures related to project specific functionality
> > >>requires deep understanding of the individual projects.
> > >> Isn't it better to leave it to the individual projects to make sure
> > >>that they are functional?
> > >> That will help the contributors to Tempest spend their time on what
> > >>only Tempest can do -i.e. identify integration issues.
> > >
> > >What do you mean by project specific functional testing? What makes
> > >debugging
> > >a marconi failure in a tempest gate job any more involved than
> > debugging a
> > >nova or neutron failure? Part of the point of having an integrated
> > gate is
> > >saying that the project works well with all the others in OpenStack.
> > IMO
> > >that's
> > >not just in project functionality but also in community. When there is
> > an
> > >issue
> > >with a gate job everyone comes together to work on it. For example if
> > you
> > >have
> > >a keystone patch that breaks a marconi test in check there is open
> > >communication
> > >about what happened and how to fix it.
> > >
> > >That being said there are certain cases where having a project
> > specific
> > >functional test makes sense. For example swift has a functional test
> > job
> > >that
> > >starts swift in devstack. But, those things are normally handled on a
> > per
> > >case
> > >basis. In general if the project is meant to be part of the larger
> > >OpenStack
> > >ecosystem then Tempest is the place to put functional testing. That
> > way
> > >you know
> > >it works with all of the other components. The thing is in openstack
> > what
> > >seems
> > >like a project isolated functional test almost always involves another
> > >project
> > >in real use cases. (for example keystone auth with api requests)
> > >
> > >Now for the boundary between what kind of test belongs where isn't
> > always
> > >clear
> > >and every project has a different rule of thumb on that. There really
> > >isn't a
> > >hard and fast rule for tempest on what's allowed as long as it fits
> > >within the
> > >criteria for the different test categories. It's handled on a per case
> > >basis.
> > >Honestly, though I normally recommend putting as much in tempest as
> > you
> > >can
> > >because a big advantage of having an external test suite is that it
> > keeps
> > >you
> > >honest because the tests are in a separate repo.
> > >
> > >-Matt Treinish
> > >
> > >_______________________________________________
> > >OpenStack-dev mailing list
> > >OpenStack-dev at lists.openstack.org
> > >http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >
> >
> > _______________________________________________
> > OpenStack-dev mailing list
> > OpenStack-dev at lists.openstack.org
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20140320/c46805ba/attachment.html>