<html><body><p>Hi Ken,<br><br>DefCore test list is using full test path name and not idempotent_id due to the issue reported in <a href="https://bugs.launchpad.net/tempest/+bug/1433700">https://bugs.launchpad.net/tempest/+bug/1433700</a> . As such, any change in the path name or test name will affect the test list. Currently, DefCore uses "alias" to identify tests with same idempotent_id . Please see alias description in <a href="http://git.openstack.org/cgit/openstack/defcore/tree/doc/source/schema/1.4.rst#n82" target="_blank"><u><font size="4" color="#0000FF">http://git.openstack.org/cgit/openstack/defcore/tree/doc/source/schema/1.4.rst#n82</font></u></a><font size="4"> </font> <br><br>Catherine Diep<br>IBM Silicon Valley Laboratory, San Jose, California 95141<br>cdiep@us.ibm.com, Tel: (408) 463-4352 T/L: 543-4352<br><font size="2" color="#800080">----- Forwarded by Catherine Cuong Diep/San Jose/IBM</font><font size="2" color="#800080"> on 03/17/2016 09:25 PM</font><font size="2" color="#800080"> -----</font><br><br><font size="2" color="#5F5F5F">From: </font><font size="2">"Ken'ichi Ohmichi" <ken1ohmichi@gmail.com></font><br><font size="2" color="#5F5F5F">To: </font><font size="2">Mark Voelker <mvoelker@vmware.com></font><br><font size="2" color="#5F5F5F">Cc: </font><font size="2">"defcore-committee@lists.openstack.org" <defcore-committee@lists.openstack.org></font><br><font size="2" color="#5F5F5F">Date: </font><font size="2">03/17/2016 07:25 PM</font><br><font size="2" color="#5F5F5F">Subject: </font><font size="2">[OpenStack-DefCore] Is it fine to change test paths of Tempest? (Re: [openstack-dev] [QA][all] Propose to remove negative tests from Tempest)</font><br><hr width="100%" size="2" align="left" noshade style="color:#8091A5; "><br><br><br><tt>Hi Mark,<br><br>Thanks for sharing it on Defcore committee :-)<br><br>That is off-topic from the original discussion, but I'd like to ask a<br>question between Tempest and Defcore.<br><br>The link[1] as you pointed contains test paths of Tempest like<br>tempest.api.compute.test_authorization.AuthorizationTestJSON.test_create_server_fails_when_tenant_incorrect<br><br>Is it acceptable to change these paths from Defcore viewpoint?<br><br>[1] contains idempotent_id also which is tagged for each test and<br>Tempest consumers would be able to use the id instead of the path.<br>So it seems possible to change the paths in general and sometimes<br>people propose changing paths for correct ones.<br>For example, AuthorizationTestJSON could be changed to<br>AuthorizationTest by removing JSON because now opposite(XML) tests<br>don't exist in Tempest.<br><br>If such change is fine from Defcore viewpoint also, I can agree<br>confidently with these changes.<br><br>Thanks<br><br>---<br>[1] </tt><tt><a href="http://git.openstack.org/cgit/openstack/defcore/tree/2016.01.json">http://git.openstack.org/cgit/openstack/defcore/tree/2016.01.json</a></tt><tt><br><br>2016-03-17 9:20 GMT-07:00 Mark Voelker <mvoelker@vmware.com>:<br>> FYI for DefCore folks that might wish to contribute to the discussion as there are ~118 negative tests included in the most recent Board-approved DefCore Guideline. [1]<br>><br>> We discussed negative tests included in the DefCore Guidelines a bit in the past and general consensus has been that it’s reasonable to include negative tests for a Capability since it validates that the “unhappy path” works in an interoperable way just as the “happy path” for an action does. Note that the suggestion here is to move negative tests to the Tempest plugin interface. As we’ve discussed, DefCore can accept in-project-tree tests via the Tempest plugin interface and RefStack will able to run them. Therefore I think if this plan is implemented, it should be mostly non-impacting for us: we can continue including the negative tests we have today, though we may need to do some housekeeping as tests move around (so we should probably keep an eye on the review queues…it would be super helpful to have a heads-up from reviewers on those patches if possible, but we’ll probably find them either way =).<br>><br>> [1] </tt><tt><a href="http://git.openstack.org/cgit/openstack/defcore/tree/2016.01.json">http://git.openstack.org/cgit/openstack/defcore/tree/2016.01.json</a></tt><tt><br>><br>> At Your Service,<br>><br>> Mark T. Voelker<br>><br>><br>><br>><br>>> Begin forwarded message:<br>>><br>>> From: "Ken'ichi Ohmichi" <ken1ohmichi@gmail.com><br>>> Subject: [openstack-dev] [QA][all] Propose to remove negative tests from Tempest<br>>> Date: March 16, 2016 at 9:20:11 PM EDT<br>>> To: OpenStack Development Mailing List <openstack-dev@lists.openstack.org><br>>> Reply-To: "OpenStack Development Mailing List (not for usage questions)" <openstack-dev@lists.openstack.org><br>>><br>>> Hi<br>>><br>>> I have one proposal[1] related to negative tests in Tempest, and<br>>> hoping opinions before doing that.<br>>><br>>> Now Tempest contains negative tests and sometimes patches are being<br>>> posted for adding more negative tests, but I'd like to propose<br>>> removing them from Tempest instead.<br>>><br>>> Negative tests verify surfaces of REST APIs for each component without<br>>> any integrations between components. That doesn't seem integration<br>>> tests which are scope of Tempest.<br>>> In addition, we need to spend the test operating time on different<br>>> component's gate if adding negative tests into Tempest. For example,<br>>> we are operating negative tests of Keystone and more<br>>> components on the gate of Nova. That is meaningless, so we need to<br>>> avoid more negative tests into Tempest now.<br>>><br>>> If wanting to add negative tests, it is a nice option to implement<br>>> these tests on each component repo with Tempest plugin interface. We<br>>> can avoid operating negative tests on different component gates and<br>>> each component team can decide what negative tests are valuable on the<br>>> gate.<br>>><br>>> In long term, all negative tests will be migrated into each component<br>>> repo with Tempest plugin interface. We will be able to operate<br>>> valuable negative tests only on each gate.<br>>><br>>> Any thoughts?<br>>><br>>> Thanks<br>>> Ken Ohmichi<br>>><br>>> ---<br>>> [1]: </tt><tt><a href="https://review.openstack.org/#/c/293197/">https://review.openstack.org/#/c/293197/</a></tt><tt><br>>><br>>> __________________________________________________________________________<br>>> OpenStack Development Mailing List (not for usage questions)<br>>> Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe<br>>> </tt><tt><a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a></tt><tt><br>><br><br>_______________________________________________<br>Defcore-committee mailing list<br>Defcore-committee@lists.openstack.org<br></tt><tt><a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/defcore-committee">http://lists.openstack.org/cgi-bin/mailman/listinfo/defcore-committee</a></tt><tt><br></tt><BR>
</body></html>