Towards the end of merging the regression test for the nova volume-attach bug - https://review.openstack.org/#/c/157959/ there was a discussion around what style the functional tests should take. Especially as that had a mix of CLI and API calls in it. Here are my thoughts for why that test ended up that way: 1) All resource setup that is table stakes for the test should be done via the API, regardless if it's a CLI or API test. The reason for this is that structured data is returned, which removes one possible error in the tests by parsing incorrectly. The API objects returned also include things like .delete methods in most cases, which means that addCleanup is a little more clean. 2) Main logic should touch which ever interface you are trying to test. This was demonstrating a CLI regression, so the volume-attach call was important to be done over the CLI. Now... here's where theory runs into issues. #1 - nova boot is table stakes. Under the above guidelines it should be called via API. However --poll is a CLI construct and so saved a custom wait loop here. We should implement that custom wait loop down the road and make that an API call https://github.com/openstack/python-novaclient/blob/master/novaclient/tests/functional/test_instances.py#L116 #2 - the volume create command is table stakes. It should be an API call. However, it can't be because the service catalog redirection only works at the CLI layer. This is actually also the crux of bug #1423695. The same reason the completion cache code failed is the reason we can't use the API for that. https://github.com/openstack/python-novaclient/blob/master/novaclient/tests/functional/test_instances.py#L129 #3 - the cleanup of the volume should have been API call. See reason for #2. https://github.com/openstack/python-novaclient/blob/master/novaclient/tests/functional/test_instances.py#L131 #4 - the cleanup of the attachment should be an addCleanup via the API. See reason for #2 why it's not. https://github.com/openstack/python-novaclient/blob/master/novaclient/tests/functional/test_instances.py#L155 I'm happy if there are other theories about how we do these things, being the first functional test in the python-novaclient tree that creates and destroys real resources, there isn't an established pattern yet. But I think doing all CLI calls in CLI tests is actually really cumbersome, especially in the amount of output parsing code needed if you are going to setup any complicated resource structure. -Sean -- Sean Dague http://dague.net