Open Stack

Excerpts from Mark Voelker's message of 2016-06-16 20:33:36 +0000:

> On Tue, Jun 14, 2016 at 05:42:16PM -0400, Doug Hellmann wrote:
> 
> 
> > I don't think DefCore actually needs to change old versions of Tempest,
> > but maybe Chris or Mark can verify that?
> 
> So if I’m groking this correctly, there’s kind of two scenarios
> being painted here.  One is the “LCD” approach where we use the
> $osversion-eol version of Tempest, where $osversion matches the
> oldest version covered in a Guideline.  The other is to use the
> start-of-$osversion version of Tempest where $osversion is the
> OpenStack version after the most recent one in the Guideline.  The
> former may result in some fairly long-lived flags, and the latter
> is actually not terribly different than what we do today I think.
> Let me try to talk through both...
> 
> In some cases, tests get flagged in the Guidelines because of
> bugs in the test or because the test needs refactoring.  The
> underlying Capabilities the those tests are testing actually work
> fine.  Once we identify such an issue, the test can be fixed…in
> master.  Under the first scenario, this potentially creates some
> very long-lived flags:
> 
> 2016.01 is the most current Guideline right now covers Juno, Kilo,
> Liberty (and Mitaka after it was released).  It’s one of the two
> Guidelines that you can use if you want an OpenStack Powered license
> from he Foundation, $vendor wants to run it against their shiny new
> Mitaka cloud.  They run the Juno EOL version of Tempest (tag=8),
> they find a test issue, and we flag it.  A few weeks later, a fix
> lands in Tempest.  Several months later the next Guideline rolls
> around: the oldest covered release is Kilo and we start telling
> people to use the Kilo-EOL version of Tempest.  That doesn’t have
> the fix, so the flag stays.  Another six months goes by and we get
> a Guideline and we’re up to the Liberty-EOL version of Tempest.  No
> fix, flag stays.  Six more months, and now we’re at Mitaka-EOL, and
> that's the first version that includes the fix.
>
> Generally speaking long lived flags aren’t so great because it
> means the tests are not required…which means there’s less or no
> assurance that the capabilities they test for actually work in the
> clouds that adhere to those Guidelines.  So, the worst-case scenario
> here looks kind of ugly.
> 
> As Matt correctly pointed out though, the capabilities DefCore
> selects for are generally pretty stable API’s that are long-lived
> across many releases, so we haven’t run into a lot of issues running
> pretty new versions of Tempest against older clouds to date.  In
> fact I’m struggling to think of a time we’ve flagged something
> because someone complained the test wasn’t runnable against an older
> release covered by the Guideline in question.  I can think of plenty
> of times where we’ve flagged something due to a test issue though…keep
> in mind we’re still in pretty formative times with DefCore here
> where these tests are starting to be used in a new way for the first
> time.  Anyway, as Matt points out we could potentially use a much
> newer Tempest tag: tag=11 (which is the start of Newton development
> and is a roughly 2 month old version of Tempest).  Next Guideline
> rolls around, we use the tag for start-of-ocata, and we get the fix
> and can drop the flag.
> 
> Today, RefStack client by default checks out a specific SHA of
> Tempest [1] (it actually did use a tag at some point in the past,
> and still can).  When we see a fix for a flagged test go in, we or
> the Refstack folks can do a quick test to make sure everything’s
> in order and then update that SHA to match the version with the
> fix.  That way we’re relatively sure we have a version that works
> today, and will work when we drop the flag in the next Guideline
> too.  When we finalize that next Guideline, we also update the
> test-repositories section of the new Guideline that Matt pointed
> to earlier to reflect the best-known version on the day the Guideline
> was sent to the Board for approval.  One added benefit of this
> approach is that people running the tests today may get a version
> of Tempest that includes a fix for a flagged test.  A flagged test
> isn’t required, but it does get run—and now will show a passing
> result, so we have data that says “this provider actually does
> support this capability (even though it’s flagged), and the test
> does indeed seem to be working."
> 
> 
> So, that’s actually not hugely different from the second scenario
> I think?  Or did I miss something there?

What I was proposing is that we keep the certification rules in
sync with OpenStack versions. If a vendor stays on an old version
of OpenStack, they can keep using the old (matching) version of
Tempest to certify.  It's not clear from your description if that's
allowed now.

We would want to expose the version of OpenStack, the version of
Tempest, any flagged tests, and any configuration settings that
change test behavior (such as the ones we're talking about in this
thread) in the results reported for the cloud.

If a vendor *wants* to use a newer version of Tempest with an older
version of OpenStack (using Mitaka Tempest to test a Liberty cloud),
that seems fine, and we should highlight that.

What effect would that change have on the DefCore program (or is
this what's allowed now)?

Doug

> 
> 
> [1] http://git.openstack.org/cgit/openstack/refstack-client/tree/setup_env#n3
> 
> At Your Service,
> 
> Mark T. Voelker