[openstack-dev] [tempest][nova][defcore] Add option to disable some strict response checking for interop testing

Arkady_Kanevsky at DELL.com Arkady_Kanevsky at DELL.com
Fri Jul 1 04:12:39 UTC 2016


There is a version of Tempest that is released as part of OpenStack release.
Agree with Mark that we should stick to versions parity.

-----Original Message-----
From: Mark Voelker [mailto:mvoelker at vmware.com]
Sent: Monday, June 20, 2016 8:25 AM
To: OpenStack Development Mailing List (not for usage questions)
Subject: Re: [openstack-dev] [tempest][nova][defcore] Add option to disable some strict response checking for interop testing


> On Jun 20, 2016, at 8:46 AM, Doug Hellmann wrote:
>
> Excerpts from Mark Voelker's message of 2016-06-16 20:33:36 +0000:
>
>> On Tue, Jun 14, 2016 at 05:42:16PM -0400, Doug Hellmann wrote:
>>
>>
>>> I don't think DefCore actually needs to change old versions of
>>> Tempest, but maybe Chris or Mark can verify that?
>>
>> So if I'm groking this correctly, there's kind of two scenarios being
>> painted here. One is the "LCD" approach where we use the
>> $osversion-eol version of Tempest, where $osversion matches the
>> oldest version covered in a Guideline. The other is to use the
>> start-of-$osversion version of Tempest where $osversion is the
>> OpenStack version after the most recent one in the Guideline. The
>> former may result in some fairly long-lived flags, and the latter is
>> actually not terribly different than what we do today I think.
>> Let me try to talk through both...
>>
>> In some cases, tests get flagged in the Guidelines because of bugs in
>> the test or because the test needs refactoring. The underlying
>> Capabilities the those tests are testing actually work fine. Once we
>> identify such an issue, the test can be fixed...in master. Under the
>> first scenario, this potentially creates some very long-lived flags:
>>
>> 2016.01 is the most current Guideline right now covers Juno, Kilo,
>> Liberty (and Mitaka after it was released). It's one of the two
>> Guidelines that you can use if you want an OpenStack Powered license
>> from he Foundation, $vendor wants to run it against their shiny new
>> Mitaka cloud. They run the Juno EOL version of Tempest (tag=8), they
>> find a test issue, and we flag it. A few weeks later, a fix lands in
>> Tempest. Several months later the next Guideline rolls
>> around: the oldest covered release is Kilo and we start telling
>> people to use the Kilo-EOL version of Tempest. That doesn't have the
>> fix, so the flag stays. Another six months goes by and we get a
>> Guideline and we're up to the Liberty-EOL version of Tempest. No
>> fix, flag stays. Six more months, and now we're at Mitaka-EOL, and
>> that's the first version that includes the fix.
>>
>> Generally speaking long lived flags aren't so great because it means
>> the tests are not required...which means there's less or no assurance
>> that the capabilities they test for actually work in the clouds that
>> adhere to those Guidelines. So, the worst-case scenario here looks
>> kind of ugly.
>>
>> As Matt correctly pointed out though, the capabilities DefCore
>> selects for are generally pretty stable API's that are long-lived
>> across many releases, so we haven't run into a lot of issues running
>> pretty new versions of Tempest against older clouds to date. In fact
>> I'm struggling to think of a time we've flagged something because
>> someone complained the test wasn't runnable against an older release
>> covered by the Guideline in question. I can think of plenty of times
>> where we've flagged something due to a test issue though...keep in mind
>> we're still in pretty formative times with DefCore here where these
>> tests are starting to be used in a new way for the first time.
>> Anyway, as Matt points out we could potentially use a much newer
>> Tempest tag: tag=11 (which is the start of Newton development and is
>> a roughly 2 month old version of Tempest). Next Guideline rolls
>> around, we use the tag for start-of-ocata, and we get the fix and can
>> drop the flag.
>>
>> Today, RefStack client by default checks out a specific SHA of
>> Tempest [1] (it actually did use a tag at some point in the past, and
>> still can). When we see a fix for a flagged test go in, we or the
>> Refstack folks can do a quick test to make sure everything's in order
>> and then update that SHA to match the version with the fix. That way
>> we're relatively sure we have a version that works today, and will
>> work when we drop the flag in the next Guideline too. When we
>> finalize that next Guideline, we also update the test-repositories
>> section of the new Guideline that Matt pointed to earlier to reflect
>> the best-known version on the day the Guideline was sent to the Board
>> for approval. One added benefit of this approach is that people
>> running the tests today may get a version of Tempest that includes a
>> fix for a flagged test. A flagged test isn't required, but it does
>> get run-and now will show a passing result, so we have data that says
>> "this provider actually does support this capability (even though
>> it's flagged), and the test does indeed seem to be working."
>>
>>
>> So, that's actually not hugely different from the second scenario I
>> think? Or did I miss something there?
>
> What I was proposing is that we keep the certification rules in sync
> with OpenStack versions. If a vendor stays on an old version of
> OpenStack, they can keep using the old (matching) version of Tempest
> to certify. It's not clear from your description if that's allowed
> now.

Yes, it's allowed: per earlier, DefCore doesn't prescribe a specific version of Tempest that one *must* use when testing to get a license agreement (though we do recommend a version via what RefStack client checks out by default). However pragmatically, using a much older version of Tempest is likely to preclude you from using newer Guidelines for the reasons stated above, one of which is: tests get unflagged in newer Guidelines as tests get fixed. If you use an older Tempest you won't have those fixes so you'll have to use an older Guideline in which those tests are still flagged.

>
> We would want to expose the version of OpenStack, the version of
> Tempest, any flagged tests, and any configuration settings that change
> test behavior (such as the ones we're talking about in this
> thread) in the results reported for the cloud.
>
> If a vendor *wants* to use a newer version of Tempest with an older
> version of OpenStack (using Mitaka Tempest to test a Liberty cloud),
> that seems fine, and we should highlight that.
>
> What effect would that change have on the DefCore program (or is this
> what's allowed now)?

If you're just asking for more information (reporting of version information): very little. If you want to say "you must use exactly version X of Tempest if you're using version Y of OpenStack", potentially more depending on the version you propose (e.g. the long-lived flag problem).

At Your Service,

Mark T. Voelker


>
> Doug
>
>>
>>
>> [1]
>> http://git.openstack.org/cgit/openstack/refstack-client/tree/setup_en
>> v#n3
>>
>> At Your Service,
>>
>> Mark T. Voelker
>
> ______________________________________________________________________
> ____ OpenStack Development Mailing List (not for usage questions)
> Unsubscribe:
> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20160701/cd999a73/attachment.html>


More information about the OpenStack-dev mailing list