[openstack-dev] [tempest][nova][defcore] Add option to disable some strict response checking for interop testing

Mike Perez thingee at gmail.com
Fri Jun 17 23:26:49 UTC 2016

On 15:12 Jun 14, Matthew Treinish wrote:
> On Tue, Jun 14, 2016 at 02:41:10PM -0400, Doug Hellmann wrote:
> > Excerpts from Matthew Treinish's message of 2016-06-14 14:21:27 -0400:
> > > On Tue, Jun 14, 2016 at 10:57:05AM -0700, Chris Hoge wrote:


> > We have basically three options.
> > 
> > 1. Tell deployers who are trying to do the right for their immediate
> >    users that they can't use the trademark.
> > 
> > 2. Flag the related tests or remove them from the DefCore enforcement
> >    suite entirely.
> > 
> > 3. Be flexible about giving consumers of Tempest time to meet the
> >    new requirement by providing a way to disable the checks.
> > 
> > Option 1 goes against our own backwards compatibility policies.
> I don't think backwards compatibility policies really apply to what what define
> as the set of tests that as a community we are saying a vendor has to pass to
> say they're OpenStack. From my perspective as a community we either take a hard
> stance on this and say to be considered an interoperable cloud (and to get the
> trademark) you have to actually have an interoperable product. We slowly ratchet
> up the requirements every 6 months, there isn't any implied backwards
> compatibility in doing that. You passed in the past but not in the newer stricter
> guidelines.
> Also, even if I did think it applied, we're not talking about a change which
> would fall into breaking that. The change was introduced a year and half ago
> during kilo and landed a year ago during liberty:
> https://review.openstack.org/#/c/156130/
> That's way longer than our normal deprecation period of 3 months and a release
> boundary.


What kind of communication happens today for these changes? There are so many
channels/high volume mailing lists a downstream deployer is expected by the
community to listening in. Some disruptive change being introduced a year or
longer ago can still be communicated poorly.

Just like we've done with Reno in communicating better about disruptive changes
in release notes, what tells teams like DefCore about changes with Tempest?
(I looked in release.o.o for tempest release notes, although maybe I missed

Since some members of DefCore have interest in making the market place healthy,
what is DefCore doing today to communicate these disruptive changes early to
deployers? Did it not happen in this particular case because:

* DefCore has no one working closely in the Tempest project to flag things?
* Defcore does work closely with Tempest, but somehow the communication for
  this was missed?
* Not having clear deprecation notices because release notes in the Tempest
  don't exist (see above)?

This all just sounds like a communication problem, and it makes me sad to
interpret this thread as people being angry with deployers as a result. How
about we not think the worse of people that are trying to prove our project
being successful and start working with them?

With that said I agree with this strict checking in tests. Deployments need to
stop defining the community defined APIs.

Mike Perez

More information about the OpenStack-dev mailing list