Open Stack

Adrian,

Capabilities are defined by existing Tempest API tests. Currently we have about 127 tests across a range
of capabilities. The vendor is expected to run those tests against their product (or an appropriate testing
platform) and report the test results back to Foundation staff.

As we’re just beginning this process, it’s not out of the scope of possibility that some tests will produce
false negatives. Indeed, this is the case right now, and I’m in New York with the QA team this week
to fix known instances of false negatives in preparation for testing. While our expectation is that a vendor
will pass all tests, at the stage it might not be possible in practice*. Patches to Tempest may be considered
with the expectation that those patches are submitted upstream to fix legitimate bugs or deficiencies.

I will be actively working with vendors to help run the tests, to understand failures, and help correct
any false negatives they encounter.

-Chris

* It’s hard to fully understand how Tempest will run on production clouds. I’m running in a non-devstack
environment which is helping to catch the edge cases, but until we’re running against a wide range
of products we can’t really know. The sooner we start testing the sooner we begin to understand any
biases that Tempest might have towards Devstack.

> On Mar 26, 2015, at 4:30 PM, Adrian Otto <adrian.otto at rackspace.com> wrote:
> 
> Rob,
> 
> I’m a little foggy on how we actually test capabilities. The tests for capabilities would need to be standard, right? I think that we are using the Refstack tests for this. Does that mean that someone has a cloud that has a previous version of service XYZ that exhibits all the required capabilities, but they would need to run a nonstandard test to demonstrate it, right? How would that work? Do we just trust participants to create and run their own tests, and just publish results with links to what they used? Would we allow patches to tools like Tempest to allow interoperability with the variants?
> 
> Thanks,
> 
> Adrian
> 
> On Mar 26, 2015, at 1:00 PM, Rob Hirschfeld <rob at rackn.com> wrote:
> 
>> 
>> On 03/26/2015 12:15 PM, Adrian Otto wrote:
>>> We can adjust that expected compliance value to a higher one when we have confidence that OpenStack Cloud operators can actually meet the desired requirement value. Using a simple OR is less attractive because every requirement wold need to be re-evaluated every time there is a new version of a capability. Using a lower boundary is a common solution to this problem. Adrian 
>> 
>> I agree OR is very expensive to users.  For something like API versions, the DefCore work is likely to drive that new API versions to support previous versions to the extent that we've required parts of the API.  The point would be that required Capabilities are testable so we can make sure that APIs endure as much as possible.
>> 
>> I think we are all in agreement - when we require APIs by making them required Capabilities, they should become long term stable. That means new versions should preserve old calls so they can pass the Capabilities tests.
>> 
>> -- 
>> 
>> Rob
>> ____________________________
>> Rob Hirschfeld, 512-773-7522
>> RackN CEO, Founder
>> 
>> I am in CENTRAL (-6) time
>> http://robhirschfeld.com
>> twitter: @zehicle, github: cloudedge & ravolt
>> 
> 
> 
> _______________________________________________
> Defcore-committee mailing list
> Defcore-committee at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/defcore-committee