Open Stack

Hi Chris,


I've added some comments and feedback in line. I'm still learning the ropes so if I'm asking questions that may have already been discussed and resolved, my appologies:


>>DefCore has two technical conditions for a test to be graded as a required capability >>against the DefCore criteria. It needs to access public endpoints, and >>not require administrator credentials.

At least for Compute, the "admin"-ness of a capability is set in the policy configuration of the service (http://docs.openstack.org/juno/config-reference/content/section_nova-policy.json.html). Given this flexibility, who then becomes the person to judge what functionality is expected to be admin only? Would this be the responsiblity of the individual PTLs or another group?

>>While this helps to ensure that the required APIs and capabilities are actually callable by >>any user, it can implicitly place a burden on the end user to have a number of resources >>that may not actually be available to >>them.
>>For example, API tests that check for tenant isolation require users >>across
>>multiple tenants[1].

The test in question is part of the authorization suite of tests. Does security testing fall under the umbrella of interoperability?

>>Tests also may require the implicit existence >>of
>>resources such at tenant networks and machine images if administrator
>>access is not available. Currently the DefCore tests can be run against a public cloud >>without administrator access, but it implicitly requires the existence >>of 1. An OpenStack endpoint. 2. Two guests in seperate tenants. 3. Two independent >>machine images. 4. One public network, or two isolated tenant networks.

The networks portion concerns me because these are all assumptions that are built into the tests, which may or may not reflect realistic environments. If the capability being tested requires a network, should it matter whether it is a public or isolated network? If the type of network has no bearing on the outcome of test, wouldn't a test that auto-configures itself based on discovery be more reliable?

>>The goal of this proposal is to explicitly define a maximum set of resources that can be >>used to test >>interoperability as part of the DefCore standard. Only tests that use at most the defined >>resources would be graded against the DefCore criteria. Ideally, this maximum set of >>resources would be an OpenStack cloud endpoint and non-admin user credentials to >>access it. However, there are resources that are required to have an operating cloud, but >>may need to be set up either by the provider if admin credentials are needed, or by the >>user beforehand. As previously mentioned, two critical resources are a network >>and a machine image. My list of proposed resources is: 1. OpenStack Endpoint: This >>public API endpoint to test against. 2. Guest user credentials: Login credentials for the >>endpoint. 3. Network ID: The ID or name of a network available to the user to attach >>virtual machines to. 4. Image ID: The ID or name of a bootable machine image.

I really like the idea of coming up with a minimum set of data required for any DefCore selected test as it provides a guideline for what assumptions the test can make.
The one part that I would add is that if you already have a valid endpoint and user credentials, the rest of the data is auto-discoverable by querying the public API for the related services as that user. I keep coming back to the idea of auto-discoverability because it has the possibility of greatly reducing friction and pushback due to fustration configuring the test suite.

>>While a DefCore capability tells you what functionlity exists in a cloud, it provides no >>guidance on how to access that functionality. Focusing on tests rather than APIs gave us >>an easy way to bootstrap >>the testing process, but at the expense of obfuscating the path for application developers >>to know which APIs match to capabilities for
>>building portable applications. My proposal is to define these resources as a future >>tandard and begin by identifying existing tests that meet the standard. >>Then begin to phase out tests that don't meet the standard by working with the QA team >>to write new tests to match the capabilities, and drop required capabilites that don't meet >>the standard.

My challenge with this portion is that we're trusting the tests, which exist outside of DefCore, to be atomic in nature and that they stay that way. Without constant manual reviews, it would seem like some type of DefCore gating job would be more efficient to ensure only the expected application paths are being traversed. This also creates a burden on functional testers to work within the boundaries of both application testing and compatibility testing, which I could see leading to duplicate test development to meet the requirements both for their project team and compatibility testing as well.

My other concern is that unit test based tests are not always the best tool to describe an expected behavior or experience. Their strong suite is validate very specific outcomes, which do not always translate well back to a user experience.

If the goal is to have a set of tests that clearly define expectations of behavior, I could see behavior driven development style of testing being much more explicit about what functionality is being tested and what the expected outcomes should be ( http://heynemann.github.io/pyvows/ and http://pythonhosted.org/behave/ are runners I've worked with in the past). I've spent a lot of time over the past several years trying to shoehorn rich, descriptive tests into unittest-style frameworks, and at the best it works just good enough. I'm not suggesting BDD as a solution, but more to better understand the problem that is trying to be solved.

Thanks,

Daryl
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/defcore-committee/attachments/20150617/4b5e0bfc/attachment-0001.html>