Open Stack

Thanks, Sam!

Huawei's issue is that the same release passes 2015.05 and not 2015.07.  This is exactly the problem Mark Voelker highlighted in the IRC log he included [0].  Reviewing Mark's references, a few questions/issues can be extracted.

1.  In the discussions, there are multiple times where Chris mentions that a vendor could just use an older version of the test.  This raises the questions:  
	- How would you do this with Refstack?
	- How would the Foundation know which version of the test was used if this could be done with Refstack?
	- How should test changes that change the effective API definition (as opposed to the documented definition) be handled?
2.  Refstack used labels for the 2015.04 and 2015.05 guidelines to fix the sha of the tests that were used to validate against a cloud, but switched to the idempotent tags for 2015.07.  With the label, you get tests that are fixed in time, but with idempotent tags, the test could change and the tag would stay the same, as long as the test "container" was the same.  So, even as we speak, 2015.07 and 2016.01 could be changing in ways that would cause a cloud that passed today to fail tomorrow.  Is this statement true?  If so, how do we deal with this?
3.  We need to be much clearer and more careful if we do not use point-in-time versions of tests to ensure that Clouds that have passed a guideline continue to pass that guideline as long as the OpenStack software has not been changed.
4.  We need to be much clearer and more careful about how undocumented aspects of the API should not be retroactively be considered "incorrect" unless documented as such.

It turns out that Huawei still has time to qualify its public cloud offering under the 2015.05 guidelines, which use the branch tag, if they can get the tests filed before the board meeting.  But, I want to stress that they *want* to test against the latest guidelines approved, yet because of the test change, their Juno distro would need to use some tests that are earlier versions of the current tests to pass.

Huawei tries to stay on the most current guidelines for the release have in their products and will continue to do so.  We just need to have tests stable enough to not disrupt products we have in maintenance mode.

--Rocky



[0] http://eavesdrop.openstack.org/irclogs/%23openstack-defcore/%23openstack-defcore.2015-06-24.log.html#t2015-06-24T23:16:06

> -----Original Message-----
> From: Sam Danes [mailto:sam.danes at RACKSPACE.COM]
> Sent: Thursday, January 07, 2016 10:53 AM
> To: Mark Voelker; Rochelle Grober
> Cc: Chenrui (A); Liyongle (Fred); defcore-
> committee at lists.openstack.org; Rick Lopez
> Subject: Re: [OpenStack-DefCore] Huawei Juno based distro passes
> 2015.05; same code fails 2015.07. Incompatible API test change from
> 2015.05 to 2015.05/07
> 
> Hey folks,
> 
> We actually have been impacted by this exact same issue. We have been
> working with our development teams to actually change some of the
> parameters in our API so that it will not be an issue for us and in
> fact
> we pass if we manually disable this change in Tempest (we did that
> analysis back when the change first landed to isolate the difference in
> a
> passing then suddenly failing run). I have mentioned in context of
> other
> DefCore discussions a few times that this is an example of a Tempest
> change, for something that is not technically covered by the DefCore
> specification, that actually modifies the outcome of DefCore
> certification.
> 
> That having been said, this specific change is something we are behind
> from an interop perspective, the question in my mind that remains is
> more
> what the process for the future is about Tempest changes that impact
> the
> DefCore Spec/Tests.
> 
> Thanks,
> 
> 
> Sam
> 
> Sam Danes, Architect, Software Development - Test
> sam.danes at rackspace.com
> mobile: 412.689.1532
>  <https://www.rackspacemarketing.com/signatyourEmail/>
> <https://www.linkedin.com/in/samueldanes>
> 
> 
> 
> 
> On 1/7/16, 9:14 AM, "Mark Voelker" <mvoelker at vmware.com> wrote:
> 
> >>
> >> On Jan 6, 2016, at 8:55 PM, Rochelle Grober
> >><rochelle.grober at huawei.com> wrote:
> >>
> >> Hi, folks.
> >>
> >> This is the email I sent to the interop at openstack.orb and the co-
> chairs
> >>of DefCore.  From today's DefCore meeting, I am posting this to the
> >>mailing list for discussion and hopefully resolution.  Mark Voelker
> said
> >>that other vendors had also had the same issue.
> >
> >Thanks for writing in, Rocky.  Actually what I said was that this
> >potential issue had been discussed before, but that in 6.5 months this
> is
> >the first time we’ve actually had someone report it. =)  Here are the
> >historical references I provided on the etherpad you originally posted
> >this to yesterday for discussion:
> >
> >===
> >
> >I brought this up on the DefCore mailing list in June:
> >
> >http://lists.openstack.org/pipermail/defcore-committee/2015-
> June/000849.ht
> >ml
> >
> >I believe we discussed it some at the midcycle as well after having it
> on
> >the weekly agenda for a couple of weeks:
> >
> >https://etherpad.openstack.org/p/DefCoreFlag.5
> >https://etherpad.openstack.org/p/DefCoreFlag.6
> >https://etherpad.openstack.org/p/DefCoreFlag.MidCycle
> >
> >And I believe it’s come up in-channel a couple of times too…one I
> could
> >put my finger on quickly is here; there are probably others:
> >
> >http://eavesdrop.openstack.org/irclogs/%23openstack-
> defcore/%23openstack-d
> >efcore.2015-06-24.log.html#t2015-06-24T23:16:06
> >
> >This originated with an announcement about the change on the
> >openstack-dev list in February:
> >
> >http://lists.openstack.org/pipermail/openstack-dev/2015-
> February/057613.ht
> >ml
> >
> >I’m not sure there was ever a formal statement about it, but I believe
> >generally the consensus from nova-core et al was that vendor
> modification
> >of API responses (even if they were adding additional info rather than
> >changing or removing info) was never really ok from an
> interoperability
> >standpoint anyway (and violated the API Change Guidelines per
> >openstack-dev discussion above), and the tempest change was deemed to
> >help make that more clear.  There was some discussion [from John
> Garbutt
> >and Ken’ichi Ohmichi whom I’ve CC'd here] around whether the change
> >should apply to the 2.1 API only (instead of 2.0 and 2.1), but AFAIK
> the
> >sentiment never really generated enough support for someone to bother
> >submitting a patch to change it in the gate.
> >
> >On the DefCore side, I recall some discussion that if there were any
> >affected vendors (in 6.5 months this is actually the first time we’ve
> >actually had someone report it) could simply use a version of Tempest
> >that predates the change since we do not currently require a specific
> >version of Tempest (though refstack-client does have a setting that
> uses
> >a specific Tempest version specified by a SHA by default).  I recall
> >warning at the time that this probably wouldn’t work in the long run
> >since bugfixes to Tempest would eventually mandate that a newer
> version
> >be used (see IRC link above), but again: sentiment seemed to be that
> >since modifying API responses was never going to be interoperable
> anyway,
> >that might simply be both a stopgap folks could use in the short term
> and
> >incentive to stop modifying API responses in the longer term.
> >
> >===
> >
> >So with that historical context provided: a couple of questions for
> you.
> >
> >1.)  Is the intent here to ask for a flag on the tests that are
> failing
> >for you?  If so, the most appropriate thing to do may be to open a
> >request via gerrit and we’ll discuss it there.
> >
> >2.)  Also, are you attempting to pass 2015.05 or 2015.07?  If the
> former,
> >have you tried simply rolling back to an earlier Tempest SHA that
> >predates the additionalProperties change as suggested above?  I
> believe
> >there’s a good chance you’d be able to pass 2015.05 by doing so.  The
> >additionalProperties change was f0c30bc, so 968f1b3 would be the
> commit
> >before that.
> >
> >3.)  How are you injecting additional properties into the API
> responses?
> >E.g. have you changed nova-api code to do so?  I ask because the API
> code
> >(other than extensions) is currently a designated section in both the
> >2015.05 and 2015.07 guidelines:
> >
> >https://github.com/openstack/defcore/blob/master/2015.05.json#L540
> >https://github.com/openstack/defcore/blob/master/2015.07.json#L1468
> >
> >At Your Service,
> >
> >Mark T. Voelker
> >
> >
> >> TL;DR  a set of Nova tests changed between the time 2015.05 build
> SHA
> >>was captured and the 2015.07 SHA was captured.  The changed tests
> were
> >>intended for the LIberty release, as they were not proposed/merged
> until
> >>after the Kilo release  Kilo release was April; changes to tests were
> >>merged June 19 (definitely in the Liberty cycle).
> >>
> >> Thanks,
> >> --Rocky
> >>
> >> Below, is more info on Huawei's situation:
> >>
> >>
> >> Huawei has a problem we hope you can help us with.  Our FusionSphere
> >>6.0, based on Juno, qualified for  the "interop" OpenStack Powered
> (tm)
> >>program on the 2015.05 DefCore tests.  We are now trying to qualify
> our
> >>Public cloud offering, running the exact same FusionSphere/OpenStack
> >>software but cannot pass a subset of the Nova tests.
> >>
> >> Huawei uses Nova api V2 (not V2.1).  As such, our implementation has
> >>"Additional Properties" enabled and used.  These are supposedly valid
> >>operations for the Juno release.
> >>
> >> The tests in question were changed via
> >>https://review.openstack.org/#/c/156130.  AdditionalProperties were
> >>changed to "False" in all of the tests included in the review.
> >>
> >> The problem is that Juno was released in October of 2014, with
> >>AdditionalProperties (extenstions) allowed in the v2 apis.  The tests
> >>were changed for the Kilo release cycle, but affect the Juno releases
> >>under the Interop/DefCore/Refstack test selection process because of
> the
> >>nonbranching nature of Tempest.
> >>
> >> We think a waiver is appropriate in this situation, as the Huawei
> code
> >>has not changed, yet it now fails API tests that were valid three
> months
> >>earlier.  We think that including the v2 api in the changes in
> >>theseDefCore  tests was a mistake, as the discussion below indicates,
> >>and that more care needs to be exercised when test changes are made
> to
> >>enforce new behaviors for later release cycles.  We would be happy to
> >>demonstrate that our release still passes the failed tests under the
> >>older version of those tests that exist in 2015.05.
> >>
> >> We have been working diligently to get the certification on our
> Public
> >>Cloud product, and had hoped to qualify/certify before the release of
> >>the 2016.01 standards, as our development resources are currently
> >>focused on moving our product to a more current  OpenStack release.
> >>
> >> Please advise us as to how we proceed from here.
> >>
> >>
> >> Respectfully,
> >> Rocky Grober and our FusionSphere team
> >>
> >> an excerpt from one of the changed files:
> >> <image001.gif>
> >>
> >> The problem of a backward incompatibility to an API being introduced
> by
> >>test changes was mentioned in one an exchange in comments:
> >>
> >> afazekas
> >> Mar 17, 2015
> >> ↩
> >>
> >> Patch Set 31: Code-Review-1
> >>
> >> Does not makes sense to prevent additional properties in tempest on
> >>something which allows extensions.
> >> Ghanshyam Mann
> >> Mar 18, 2015
> >> ↩
> >>
> >> Patch Set 31:
> >>
> >> @afazekas - Thanks for review.
> >>
> >> Actually all schema has been modified to work for all extension (as
> >>current tests runs for all extension enable).
> >>
> >> As v2 (v2.1) API are frozen, tempest should have
> additionalProperties
> >>False to capture any unwanted changes in those API.
> >> afazekas
> >> Mar 18, 2015
> >> ↩
> >>
> >> Patch Set 31:
> >>
> >> https://wiki.openstack.org/wiki/APIChangeGuidelines
> >>
> >> "Adding a property to a resource representation" is generally
> >>acceptable.
> >>
> >> "Changing or removing a property in a resource representation" OR
> >>"Changing the semantics of a property in a resource representation
> which
> >>may be supplied by clients" is generally not acceptable.
> >>
> >> We do not need to prevent additional properties.
> >> Ghanshyam Mann
> >> Mar 19, 2015
> >> ↩
> >>
> >> Patch Set 31:
> >>
> >> Yea, those are acceptable and it used to done by adding new
> extension
> >>etc. But as v2 nd v2.1 are frozen and no new extension for thosse.
> All
> >>new attributes etc can be added with microversion only.
> >>  As it is done for first microversion -
> >>
> >> https://review.openstack.org/#/c/140313/
> >>
> >> Please do let me know if m missing anything on this.
> >> Ken'ichi Ohmichi
> >> Mar 19, 2015
> >> ↩
> >>
> >> Patch Set 31: Code-Review+2
> >>
> >> Yeah, Ghanshyam is right.
> >>
> >> As
> >>
> >>http://lists.openstack.org/pipermail/openstack-dev/2015-
> February/057613.h
> >>tml
> >>  , new attributes will be added with microversions and v2(v2.1)
> should
> >>be frozen. So this change is necessary for blocking additional
> attributes
> >>  on v2 and v2.1 API.
> >>
> >>
> >> _______________________________________________
> >> Defcore-committee mailing list
> >> Defcore-committee at lists.openstack.org
> >> http://lists.openstack.org/cgi-bin/mailman/listinfo/defcore-
> committee
> >
> >_______________________________________________
> >Defcore-committee mailing list
> >Defcore-committee at lists.openstack.org
> >http://lists.openstack.org/cgi-bin/mailman/listinfo/defcore-committee