Open Stack

> 
> On Jan 6, 2016, at 8:55 PM, Rochelle Grober <rochelle.grober at huawei.com> wrote:
> 
> Hi, folks.
>  
> This is the email I sent to the interop at openstack.orb and the co-chairs of DefCore.  From today's DefCore meeting, I am posting this to the mailing list for discussion and hopefully resolution.  Mark Voelker said that other vendors had also had the same issue.  

Thanks for writing in, Rocky.  Actually what I said was that this potential issue had been discussed before, but that in 6.5 months this is the first time we’ve actually had someone report it. =)  Here are the historical references I provided on the etherpad you originally posted this to yesterday for discussion:

===

I brought this up on the DefCore mailing list in June:

http://lists.openstack.org/pipermail/defcore-committee/2015-June/000849.html

I believe we discussed it some at the midcycle as well after having it on the weekly agenda for a couple of weeks:

https://etherpad.openstack.org/p/DefCoreFlag.5
https://etherpad.openstack.org/p/DefCoreFlag.6
https://etherpad.openstack.org/p/DefCoreFlag.MidCycle

And I believe it’s come up in-channel a couple of times too…one I could put my finger on quickly is here; there are probably others:

http://eavesdrop.openstack.org/irclogs/%23openstack-defcore/%23openstack-defcore.2015-06-24.log.html#t2015-06-24T23:16:06

This originated with an announcement about the change on the openstack-dev list in February: 

http://lists.openstack.org/pipermail/openstack-dev/2015-February/057613.html

I’m not sure there was ever a formal statement about it, but I believe generally the consensus from nova-core et al was that vendor modification of API responses (even if they were adding additional info rather than changing or removing info) was never really ok from an interoperability standpoint anyway (and violated the API Change Guidelines per openstack-dev discussion above), and the tempest change was deemed to help make that more clear.  There was some discussion [from John Garbutt and Ken’ichi Ohmichi whom I’ve CC'd here] around whether the change should apply to the 2.1 API only (instead of 2.0 and 2.1), but AFAIK the sentiment never really generated enough support for someone to bother submitting a patch to change it in the gate.

On the DefCore side, I recall some discussion that if there were any affected vendors (in 6.5 months this is actually the first time we’ve actually had someone report it) could simply use a version of Tempest that predates the change since we do not currently require a specific version of Tempest (though refstack-client does have a setting that uses a specific Tempest version specified by a SHA by default).  I recall warning at the time that this probably wouldn’t work in the long run since bugfixes to Tempest would eventually mandate that a newer version be used (see IRC link above), but again: sentiment seemed to be that since modifying API responses was never going to be interoperable anyway, that might simply be both a stopgap folks could use in the short term and incentive to stop modifying API responses in the longer term.

===

So with that historical context provided: a couple of questions for you.

1.)  Is the intent here to ask for a flag on the tests that are failing for you?  If so, the most appropriate thing to do may be to open a request via gerrit and we’ll discuss it there.  

2.)  Also, are you attempting to pass 2015.05 or 2015.07?  If the former, have you tried simply rolling back to an earlier Tempest SHA that predates the additionalProperties change as suggested above?  I believe there’s a good chance you’d be able to pass 2015.05 by doing so.  The additionalProperties change was f0c30bc, so 968f1b3 would be the commit before that.  

3.)  How are you injecting additional properties into the API responses?  E.g. have you changed nova-api code to do so?  I ask because the API code (other than extensions) is currently a designated section in both the 2015.05 and 2015.07 guidelines:

https://github.com/openstack/defcore/blob/master/2015.05.json#L540
https://github.com/openstack/defcore/blob/master/2015.07.json#L1468

At Your Service,

Mark T. Voelker


> TL;DR  a set of Nova tests changed between the time 2015.05 build SHA was captured and the 2015.07 SHA was captured.  The changed tests were intended for the LIberty release, as they were not proposed/merged until after the Kilo release  Kilo release was April; changes to tests were merged June 19 (definitely in the Liberty cycle).
>  
> Thanks,
> --Rocky
>  
> Below, is more info on Huawei's situation:
>  
>  
> Huawei has a problem we hope you can help us with.  Our FusionSphere 6.0, based on Juno, qualified for  the "interop" OpenStack Powered (tm) program on the 2015.05 DefCore tests.  We are now trying to qualify our Public cloud offering, running the exact same FusionSphere/OpenStack software but cannot pass a subset of the Nova tests.
>  
> Huawei uses Nova api V2 (not V2.1).  As such, our implementation has "Additional Properties" enabled and used.  These are supposedly valid operations for the Juno release.
>  
> The tests in question were changed via https://review.openstack.org/#/c/156130.  AdditionalProperties were changed to "False" in all of the tests included in the review.
>  
> The problem is that Juno was released in October of 2014, with AdditionalProperties (extenstions) allowed in the v2 apis.  The tests were changed for the Kilo release cycle, but affect the Juno releases under the Interop/DefCore/Refstack test selection process because of the nonbranching nature of Tempest.  
>  
> We think a waiver is appropriate in this situation, as the Huawei code has not changed, yet it now fails API tests that were valid three months earlier.  We think that including the v2 api in the changes in theseDefCore  tests was a mistake, as the discussion below indicates, and that more care needs to be exercised when test changes are made to enforce new behaviors for later release cycles.  We would be happy to demonstrate that our release still passes the failed tests under the older version of those tests that exist in 2015.05.  
>  
> We have been working diligently to get the certification on our Public Cloud product, and had hoped to qualify/certify before the release of the 2016.01 standards, as our development resources are currently focused on moving our product to a more current  OpenStack release.
>  
> Please advise us as to how we proceed from here.
>  
>  
> Respectfully,
> Rocky Grober and our FusionSphere team
>  
> an excerpt from one of the changed files:
> <image001.gif>
>  
> The problem of a backward incompatibility to an API being introduced by test changes was mentioned in one an exchange in comments:
>  
> afazekas
> Mar 17, 2015
> ↩
> 
> Patch Set 31: Code-Review-1
> 
> Does not makes sense to prevent additional properties in tempest on something which allows extensions.
> Ghanshyam Mann
> Mar 18, 2015
> ↩
> 
> Patch Set 31:
> 
> @afazekas - Thanks for review.
> 
> Actually all schema has been modified to work for all extension (as current tests runs for all extension enable).
> 
> As v2 (v2.1) API are frozen, tempest should have additionalProperties False to capture any unwanted changes in those API.
> afazekas
> Mar 18, 2015
> ↩
> 
> Patch Set 31:
> 
> https://wiki.openstack.org/wiki/APIChangeGuidelines
> 
> "Adding a property to a resource representation" is generally acceptable.
> 
> "Changing or removing a property in a resource representation" OR "Changing the semantics of a property in a resource representation which may be supplied by clients" is generally not acceptable.
> 
> We do not need to prevent additional properties.
> Ghanshyam Mann
> Mar 19, 2015
> ↩
> 
> Patch Set 31:
> 
> Yea, those are acceptable and it used to done by adding new extension etc. But as v2 nd v2.1 are frozen and no new extension for thosse. All new attributes etc can be added with microversion only.
>  As it is done for first microversion - 
> 
> https://review.openstack.org/#/c/140313/
> 
> Please do let me know if m missing anything on this.
> Ken'ichi Ohmichi
> Mar 19, 2015
> ↩
> 
> Patch Set 31: Code-Review+2
> 
> Yeah, Ghanshyam is right.
> 
> As 
> http://lists.openstack.org/pipermail/openstack-dev/2015-February/057613.html
>  , new attributes will be added with microversions and v2(v2.1) should be frozen. So this change is necessary for blocking additional attributes
>  on v2 and v2.1 API.
> 
>  
> _______________________________________________
> Defcore-committee mailing list
> Defcore-committee at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/defcore-committee