Open Stack

DefCore has two technical conditions for a test to be graded as a
required capability against the DefCore criteria. It needs to access
public endpoints, and not require administrator credentials. While this
helps to ensure that the required APIs and capabilities are actually
callable by any user, it can implicitly place a burden on the end user to
have a number of resources that may not actually be available to them.
For example, API tests that check for tenant isolation require users across
multiple tenants[1]. Tests also may require the implicit existence of 
resources such at tenant networks and machine images if administrator
access is not available. Currently the DefCore tests can be run against a
public cloud without administrator access, but it implicitly requires the
existence of
    
1. An OpenStack endpoint.
2. Two guests in seperate tenants.
3. Two independent machine images.
4. One public network, or two isolated tenant networks.

The goal of this proposal is to explicitly define a maximum set of
resources that can be used to test interoperability as part of the
DefCore standard. Only tests that use at most the defined resources would
be graded against the DefCore criteria. Ideally, this maximum set of
resources would be an OpenStack cloud endpoint and non-admin user
credentials to access it. However, there are resources that are required
to have an operating cloud, but may need to be set up either by the
provider if admin credentials are needed, or by the user beforehand. As
previously mentioned, two critical resources are a network and a machine
image.

My list of proposed resources is:
1. OpenStack Endpoint: This public API endpoint to test against.
2. Guest user credentials: Login credentials for the endpoint.
3. Network ID: The ID or name of a network available to the user to
   attach virtual machines to.
4. Image ID: The ID or name of a bootable machine image.
    
That list is smaller than the implicit list required by Tempest,  and
represents the most basic resources needed for launching generic and
portable applications across OpenStack clouds. By testing APIs against
this standard, we can help to establish a set of common calls that will
be used as a foundation for portable applications that run on top of
OpenStack. One of the benefits of this approach would allow users to
quickly configure Tempest to test their clouds using a tool like the now
abandoned TCup, or even a web service that can automatically test clouds
remotely.

The maximum resources aren't intended to fully test the correctness of an
OpenStack cloud. Indeed, one might want to ensure that a cloud is
providing tenant isolation and resource protection. Nothing precludes
this testing, and DefCore should continue to encourage collection of and
reporting on all API test results to identify widely deployed
capabilities.

In support of interoperability, DefCore and Tempest should also map tests
to API calls. While a DefCore capability tells you what functionlity
exists in a cloud, it provides no guidance on how to access that
functionality. Focusing on tests rather than APIs gave us an easy way to
bootstrap the testing process, but at the expense of obfuscating the path
for application developers to know which APIs match to capabilities for
building portable applications.

My proposal is to define these resources as a future standard and begin
by identifying existing tests that meet the standard. Then begin to phase
out tests that don't meet the standard by working with the QA team to
write new tests to match the capabilities, and drop required capabilites
that don't meet the standard. One year should be sufficient for a
complete and non-disruptive transition.

-Chris

[1] For example, test_create_keypair_in_analt_user_tenant

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/defcore-committee/attachments/20150616/61dced3a/attachment.html>