Open Stack

Agreed that the lack of clarity and clear steps/expectations in both the spec and often the tests themselves does lend to anywhere from minor to major confusion when things fail.

I believe that the test_delete_saving_image was exactly designed to detect this sort of failure. In other words, this is potentially a legit failure of the cloud as the cloud under test has implemented a different behavior than the intent of the actual Tempest test case.

--Sam

________________________________________
From: Meyer, Jim <jim.meyer at hpe.com>
Sent: Thursday, March 10, 2016 11:43 AM
To: Daryl Walleck
Cc: defcore-committee at lists.openstack.org
Subject: Re: [OpenStack-DefCore] Fw:  Discussing tests

I see your point and don’t disagree in principle.

However, for the specific example given I’d say it’s a broken API when the software can know the state of the image upload; knows it can’t delete during upload; and returns success anyway.

—j


On Mar 10, 2016, at 11:23 AM, Daryl Walleck <daryl.walleck at RACKSPACE.COM<mailto:daryl.walleck at rackspace.com>> wrote:

I think this also brings up a good point from yesterday's discussion about "what is an interim test?". We all vigorously agree that compliance to the API contract falls into interop testing. We also agreed that non-happy path (negative responses) are also in the scope of interop. I think this test brings up another aspect important to interop, which for lack of a better term I will call application behavior.

It would be easy for the API responses to the create image/delete image to meet the correct response codes and schemas, but the behavior of the application to not be correct (the delete request is ignored, the image goes into the error state). Given that we consider non-happy path API tests to be in consideration for interop, I think non-happy path application behavior would also fall into scope. I do think we can be more clear about what aspects of the OpenStack projects fall into scope (the artifacts created by an application like VMs, volumes, etc are others that we include in our tests now) so that it is more apparent what tests should fall into scope.

Daryl
________________________________
From: Gema Gomez-Solano <gema.gomez-solano at canonical.com<mailto:gema.gomez-solano at canonical.com>>
Sent: Thursday, March 10, 2016 9:59 AM
To: Catherine Cuong Diep
Cc: Daryl Walleck; defcore-committee at lists.openstack.org<mailto:defcore-committee at lists.openstack.org>
Subject: Re: [OpenStack-DefCore] Fw: Discussing tests

In my view the call to delete image should work regardless of the status of the image (from interoperability viewpoint). This is an async call, right? so it should be queued and dealt with in due time (i.e. whenever the save operation is finished). This is the behavior I would expect as an end user, havent looked at the test yet, will do when i get home!

Cheers,
Gema

Sent from my iPhone

On 10 Mar 2016, at 09:42, Catherine Cuong Diep <cdiep at us.ibm.com<mailto:cdiep at us.ibm.com>> wrote:

I think the question for DefCore to review is:

Is "being able to delete an image that is not done saving " an important capability for interoperability ?

Catherine Diep
IBM Silicon Valley Laboratory, San Jose, California 95141
cdiep at us.ibm.com<mailto:cdiep at us.ibm.com>, Tel: (408) 463-4352 T/L: 543-4352
----- Forwarded by Catherine Cuong Diep/San Jose/IBM on 03/10/2016 07:37 AM -----

From: Daryl Walleck <daryl.walleck at RACKSPACE.COM<mailto:daryl.walleck at rackspace.com>>
To: Jean-Daniel Bonnetot <jean-daniel.bonnetot at corp.ovh.com<mailto:jean-daniel.bonnetot at corp.ovh.com>>, "defcore-committee at lists.openstack.org<mailto:defcore-committee at lists.openstack.org>" <defcore-committee at lists.openstack.org<mailto:defcore-committee at lists.openstack.org>>
Date: 03/10/2016 07:25 AM
Subject: Re: [OpenStack-DefCore] Discussing tests
________________________________



My understanding of that test is that it is validating the system behavior being able to delete an image that is not done saving.

As to your question of atomicity, one of the outcomes of the DefCore midcycle is an audit of the existing tests to provide a precise list of what API calls each test makes and what assertions are really being made. The outcome of that audit should help with these types of questions about individual tests.

Daryl

From: Jean-Daniel Bonnetot<mailto:jean-daniel.bonnetot at corp.ovh.com>
Sent: Thursday, March 10, 2016 9:05 AM
To: defcore-committee at lists.openstack.org<mailto:defcore-committee at lists.openstack.org>
Subject: [OpenStack-DefCore] Discussing tests
Hi defcore,

Not sure it’s the right place to discuss about it but I try ;)

I’m using refstack to test our Public Cloud at OVH and I have question about a test which make trouble on our solution.

I run the 2015.07 guidelines.
The test is tempest.api.compute.images.test_images.ImagesTestJSON.test_delete_saving_image.

I saw this test corresponds to the compute-images-delete capability.
I understand that image deletion is required, but why the test is deletion on a saving image?

In this particular case, at OVH we had to patch Glance to forbid deletion if the image is in saving state because of some Ceph issue.
I saw that during the last meetup you talked about "Atomicity of tests », I think that’s what I’m talking about too and I agree that it’s a problem we need to solve.

compute-images-delete is marked as atomic but it’s not from my point of view.


--
Jean-Daniel Bonnetot
http://www.ovh.com<http://www.ovh.com/>
@pilgrimstack




_______________________________________________
Defcore-committee mailing list
Defcore-committee at lists.openstack.org<mailto:Defcore-committee at lists.openstack.org>
http://lists.openstack.org/cgi-bin/mailman/listinfo/defcore-committee_______________________________________________
Defcore-committee mailing list
Defcore-committee at lists.openstack.org<mailto:Defcore-committee at lists.openstack.org>
http://lists.openstack.org/cgi-bin/mailman/listinfo/defcore-committee

_______________________________________________
Defcore-committee mailing list
Defcore-committee at lists.openstack.org<mailto:Defcore-committee at lists.openstack.org>
http://lists.openstack.org/cgi-bin/mailman/listinfo/defcore-committee
_______________________________________________
Defcore-committee mailing list
Defcore-committee at lists.openstack.org<mailto:Defcore-committee at lists.openstack.org>
http://lists.openstack.org/cgi-bin/mailman/listinfo/defcore-committee


_______________________________________________
Defcore-committee mailing list
Defcore-committee at lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/defcore-committee