Open Stack

> On Jun 20, 2016, at 8:46 AM, Doug Hellmann <doug at doughellmann.com> wrote:
> 
> Excerpts from Mark Voelker's message of 2016-06-16 20:33:36 +0000:
> 
>> On Tue, Jun 14, 2016 at 05:42:16PM -0400, Doug Hellmann wrote:
>> 
>> 
>>> I don't think DefCore actually needs to change old versions of Tempest,
>>> but maybe Chris or Mark can verify that?
>> 
>> So if I’m groking this correctly, there’s kind of two scenarios
>> being painted here.  One is the “LCD” approach where we use the
>> $osversion-eol version of Tempest, where $osversion matches the
>> oldest version covered in a Guideline.  The other is to use the
>> start-of-$osversion version of Tempest where $osversion is the
>> OpenStack version after the most recent one in the Guideline.  The
>> former may result in some fairly long-lived flags, and the latter
>> is actually not terribly different than what we do today I think.
>> Let me try to talk through both...
>> 
>> In some cases, tests get flagged in the Guidelines because of
>> bugs in the test or because the test needs refactoring.  The
>> underlying Capabilities the those tests are testing actually work
>> fine.  Once we identify such an issue, the test can be fixed…in
>> master.  Under the first scenario, this potentially creates some
>> very long-lived flags:
>> 
>> 2016.01 is the most current Guideline right now covers Juno, Kilo,
>> Liberty (and Mitaka after it was released).  It’s one of the two
>> Guidelines that you can use if you want an OpenStack Powered license
>> from he Foundation, $vendor wants to run it against their shiny new
>> Mitaka cloud.  They run the Juno EOL version of Tempest (tag=8),
>> they find a test issue, and we flag it.  A few weeks later, a fix
>> lands in Tempest.  Several months later the next Guideline rolls
>> around: the oldest covered release is Kilo and we start telling
>> people to use the Kilo-EOL version of Tempest.  That doesn’t have
>> the fix, so the flag stays.  Another six months goes by and we get
>> a Guideline and we’re up to the Liberty-EOL version of Tempest.  No
>> fix, flag stays.  Six more months, and now we’re at Mitaka-EOL, and
>> that's the first version that includes the fix.
>> 
>> Generally speaking long lived flags aren’t so great because it
>> means the tests are not required…which means there’s less or no
>> assurance that the capabilities they test for actually work in the
>> clouds that adhere to those Guidelines.  So, the worst-case scenario
>> here looks kind of ugly.
>> 
>> As Matt correctly pointed out though, the capabilities DefCore
>> selects for are generally pretty stable API’s that are long-lived
>> across many releases, so we haven’t run into a lot of issues running
>> pretty new versions of Tempest against older clouds to date.  In
>> fact I’m struggling to think of a time we’ve flagged something
>> because someone complained the test wasn’t runnable against an older
>> release covered by the Guideline in question.  I can think of plenty
>> of times where we’ve flagged something due to a test issue though…keep
>> in mind we’re still in pretty formative times with DefCore here
>> where these tests are starting to be used in a new way for the first
>> time.  Anyway, as Matt points out we could potentially use a much
>> newer Tempest tag: tag=11 (which is the start of Newton development
>> and is a roughly 2 month old version of Tempest).  Next Guideline
>> rolls around, we use the tag for start-of-ocata, and we get the fix
>> and can drop the flag.
>> 
>> Today, RefStack client by default checks out a specific SHA of
>> Tempest [1] (it actually did use a tag at some point in the past,
>> and still can).  When we see a fix for a flagged test go in, we or
>> the Refstack folks can do a quick test to make sure everything’s
>> in order and then update that SHA to match the version with the
>> fix.  That way we’re relatively sure we have a version that works
>> today, and will work when we drop the flag in the next Guideline
>> too.  When we finalize that next Guideline, we also update the
>> test-repositories section of the new Guideline that Matt pointed
>> to earlier to reflect the best-known version on the day the Guideline
>> was sent to the Board for approval.  One added benefit of this
>> approach is that people running the tests today may get a version
>> of Tempest that includes a fix for a flagged test.  A flagged test
>> isn’t required, but it does get run—and now will show a passing
>> result, so we have data that says “this provider actually does
>> support this capability (even though it’s flagged), and the test
>> does indeed seem to be working."
>> 
>> 
>> So, that’s actually not hugely different from the second scenario
>> I think?  Or did I miss something there?
> 
> What I was proposing is that we keep the certification rules in
> sync with OpenStack versions. If a vendor stays on an old version
> of OpenStack, they can keep using the old (matching) version of
> Tempest to certify.  It's not clear from your description if that's
> allowed now.

Yes, it’s allowed: per earlier, DefCore doesn’t prescribe a specific version of Tempest that one *must* use when testing to get a license agreement (though we do recommend a version via what RefStack client checks out by default).  However pragmatically, using a much older version of Tempest is likely to preclude you from using newer Guidelines for the reasons stated above, one of which is: tests get unflagged in newer Guidelines as tests get fixed.  If you use an older Tempest you won’t have those fixes so you’ll have to use an older Guideline in which those tests are still flagged.

> 
> We would want to expose the version of OpenStack, the version of
> Tempest, any flagged tests, and any configuration settings that
> change test behavior (such as the ones we're talking about in this
> thread) in the results reported for the cloud.
> 
> If a vendor *wants* to use a newer version of Tempest with an older
> version of OpenStack (using Mitaka Tempest to test a Liberty cloud),
> that seems fine, and we should highlight that.
> 
> What effect would that change have on the DefCore program (or is
> this what's allowed now)?

If you’re just asking for more information (reporting of version information): very little.  If you want to say “you must use exactly version X of Tempest if you’re using version Y of OpenStack”, potentially more depending on the version you propose (e.g. the long-lived flag problem).  

At Your Service,

Mark T. Voelker


> 
> Doug
> 
>> 
>> 
>> [1] http://git.openstack.org/cgit/openstack/refstack-client/tree/setup_env#n3
>> 
>> At Your Service,
>> 
>> Mark T. Voelker
> 
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev