<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
<style type="text/css" style="display:none"><!-- p { margin-top: 0px; margin-bottom: 0px; }--></style>
</head>
<body dir="ltr" style="font-size:12pt;color:#000000;background-color:#FFFFFF;font-family:Calibri,Arial,Helvetica,sans-serif;">
<p>Hi Chris,<br>
</p>
<p><br>
</p>
<p>I've added some comments and feedback in line. I'm still learning the ropes so if I'm asking questions that may have already been discussed and resolved, my appologies:<span style="font-size: 12pt;"> </span></p>
<div style="word-wrap:break-word">
<div id="divRplyFwdMsg" dir="ltr">
<div> </div>
</div>
<div>
<div id="magicdomid407" class="ace-line" style="margin:0px; padding:0px 1px 0px 0px">
<font face="Arial, sans-serif" class=""><span class="" style="line-height:16px; white-space:pre-wrap">>>DefCore has two technical conditions for a test to be graded as a required capability >>against the DefCore criteria. It needs to access public endpoints,
and >>not require administrator credentials.</span></font></div>
<div id="magicdomid407" class="ace-line" style="margin:0px; padding:0px 1px 0px 0px">
<font face="Arial, sans-serif" class=""><span class="" style="line-height:16px; white-space:pre-wrap"><br>
</span></font></div>
<div id="magicdomid407" class="ace-line" style="margin:0px; padding:0px 1px 0px 0px">
<font face="Arial, sans-serif" class=""><span class="" style="line-height:16px; white-space:pre-wrap">At least for Compute, the "admin"-ness of a capability is set in the policy configuration of the service (http://docs.openstack.org/juno/config-reference/content/section_nova-policy.json.html).
Given this flexibility, who then becomes the person to judge what functionality is expected to be admin only? Would this be the responsiblity of the individual PTLs or another group?</span></font></div>
<div id="magicdomid407" class="ace-line" style="margin:0px; padding:0px 1px 0px 0px">
<font face="Arial, sans-serif" class=""><span class="" style="line-height:16px; white-space:pre-wrap"><br>
</span></font></div>
<div id="magicdomid407" class="ace-line" style="margin:0px; padding:0px 1px 0px 0px">
<font face="Arial, sans-serif" class=""><span class="" style="line-height:16px; white-space:pre-wrap">>>While this helps to ensure that the required APIs and capabilities are actually callable by >>any user, it can implicitly place a burden on the end user
to have a number of resources >>that may not actually be available to >>them.</span></font></div>
<div id="magicdomid407" class="ace-line" style="margin:0px; padding:0px 1px 0px 0px">
<font face="Arial, sans-serif" class=""><span class="" style="line-height:16px; white-space:pre-wrap">>>For example, API tests that check for tenant isolation require users
</span></font><span class="" style="line-height:16px; white-space:pre-wrap; font-family:Arial,sans-serif">>>across</span></div>
<div id="magicdomid407" class="ace-line" style="margin:0px; padding:0px 1px 0px 0px">
<span class="" style="line-height:16px; white-space:pre-wrap; font-family:Arial,sans-serif">>>multiple tenants[1].</span></div>
<div id="magicdomid407" class="ace-line" style="margin:0px; padding:0px 1px 0px 0px">
<span class="" style="line-height:16px; white-space:pre-wrap; font-family:Arial,sans-serif"><br>
</span></div>
<div id="magicdomid407" class="ace-line" style="margin:0px; padding:0px 1px 0px 0px">
<span class="" style="line-height:16px; white-space:pre-wrap; font-family:Arial,sans-serif">The test in question is part of the authorization suite of tests. Does security testing fall under the umbrella of interoperability?</span></div>
<div id="magicdomid407" class="ace-line" style="margin:0px; padding:0px 1px 0px 0px">
<span class="" style="line-height:16px; white-space:pre-wrap; font-family:Arial,sans-serif"><br>
</span></div>
<div id="magicdomid407" class="ace-line" style="margin:0px; padding:0px 1px 0px 0px">
<span class="" style="line-height:16px; white-space:pre-wrap; font-family:Arial,sans-serif">>>Tests also may require the implicit existence
</span><span class="" style="line-height:16px; white-space:pre-wrap; font-family:Arial,sans-serif">>>of </span></div>
<div id="magicdomid407" class="ace-line" style="margin:0px; padding:0px 1px 0px 0px">
<span class="" style="line-height:16px; white-space:pre-wrap; font-family:Arial,sans-serif">>>resources such at tenant networks and machine images if administrator</span></div>
<div id="magicdomid407" class="ace-line" style="margin:0px; padding:0px 1px 0px 0px">
<font face="Arial, sans-serif" class=""><span class="" style="line-height:16px; white-space:pre-wrap">>>access is not available. Currently the DefCore tests can be run against a public cloud >>without administrator access, but it implicitly requires the existence
>>of 1. An OpenStack endpoint. 2. Two guests in seperate tenants. 3. Two independent >>machine images. 4. One public network, or two isolated tenant networks.</span></font></div>
<div id="magicdomid407" class="ace-line" style="margin:0px; padding:0px 1px 0px 0px">
<font face="Arial, sans-serif" class=""><span class="" style="line-height:16px; white-space:pre-wrap"><br>
</span></font></div>
<div id="magicdomid407" class="ace-line" style="margin:0px; padding:0px 1px 0px 0px">
<font face="Arial, sans-serif" class=""><span class="" style="line-height:16px; white-space:pre-wrap">The networks portion concerns me because these are all assumptions that are built into the tests, which may or may not reflect realistic environments. If the
capability being tested requires a network, should it matter whether it is a public or isolated network? If the type of network has no bearing on the outcome of test, wouldn't a test that auto-configures itself based on discovery be more reliable?</span></font></div>
<div id="magicdomid407" class="ace-line" style="margin:0px; padding:0px 1px 0px 0px">
<font face="Arial, sans-serif" class=""><span class="" style="line-height:16px; white-space:pre-wrap"><br>
</span></font></div>
<div id="magicdomid407" class="ace-line" style="margin:0px; padding:0px 1px 0px 0px">
<font face="Arial, sans-serif" class=""><span class="" style="line-height:16px; white-space:pre-wrap">>>The goal of this proposal is to explicitly define a maximum set of resources that can be >>used to test >>interoperability as part of the DefCore standard.
Only tests that use at most the defined >>resources would be graded against the DefCore criteria. Ideally, this maximum set of >>resources would be an OpenStack cloud endpoint and non-admin user credentials to >>access it. However, there are resources that
are required to have an operating cloud, but >>may need to be set up either by the provider if admin credentials are needed, or by the >>user beforehand. As previously mentioned, two critical resources are a network >>and a machine image. My list of proposed
resources is: 1. OpenStack Endpoint: This >>public API endpoint to test against. 2. Guest user credentials: Login credentials for the >>endpoint. 3. Network ID: The ID or name of a network available to the user to attach >>virtual machines to. 4. Image ID:
The ID or name of a bootable machine image.</span></font></div>
<div id="magicdomid407" class="ace-line" style="margin:0px; padding:0px 1px 0px 0px">
<font face="Arial, sans-serif" class=""><span class="" style="line-height:16px; white-space:pre-wrap"><br>
</span></font></div>
<div id="magicdomid407" class="ace-line" style="margin:0px; padding:0px 1px 0px 0px">
<font face="Arial, sans-serif" class=""><span class="" style="line-height:16px; white-space:pre-wrap">I really like the idea of coming up with a minimum set of data required for any DefCore selected test as it provides a guideline for what assumptions the test
can make.</span></font></div>
<div id="magicdomid407" class="ace-line" style="margin:0px; padding:0px 1px 0px 0px">
<font face="Arial, sans-serif" class=""><span class="" style="line-height:16px; white-space:pre-wrap">The one part that I would add is that if you already have a valid endpoint and user credentials, the rest of the data is auto-discoverable by querying the
public API for the related services as that user. I keep coming back to the idea of auto-discoverability because it has the possibility of greatly reducing friction and pushback due to fustration configuring the test suite.</span></font></div>
<div id="magicdomid407" class="ace-line" style="margin:0px; padding:0px 1px 0px 0px">
<font face="Arial, sans-serif" class=""><span class="" style="line-height:16px; white-space:pre-wrap"><br>
</span></font></div>
<div id="magicdomid407" class="ace-line" style="margin:0px; padding:0px 1px 0px 0px">
<font face="Arial, sans-serif" class=""><span class="" style="line-height:16px; white-space:pre-wrap">>>While a DefCore capability tells you what functionlity exists in a cloud, it provides no >>guidance on how to access that functionality. Focusing on tests
rather than APIs gave us >>an easy way to bootstrap >>the testing process, but at the expense of obfuscating the path for application developers >>to know which APIs match to capabilities for</span></font></div>
<div id="magicdomid407" class="ace-line" style="margin:0px; padding:0px 1px 0px 0px">
<font face="Arial, sans-serif" class=""><span class="" style="line-height:16px; white-space:pre-wrap">>>building portable applications. My proposal is to define these resources as a future >>tandard and begin by identifying existing tests that meet the standard.
>>Then begin to phase out tests that don't meet the standard by working with the QA team >>to write new tests to match the capabilities, and drop required capabilites that don't meet >>the standard.</span></font></div>
<div id="magicdomid407" class="ace-line" style="margin:0px; padding:0px 1px 0px 0px">
<font face="Arial, sans-serif" class=""><span class="" style="line-height:16px; white-space:pre-wrap"><br>
</span></font></div>
<div id="magicdomid407" class="ace-line" style="margin:0px; padding:0px 1px 0px 0px">
<font face="Arial, sans-serif" class=""><span class="" style="line-height:16px; white-space:pre-wrap">My challenge with this portion is that we're trusting the tests, which exist outside of DefCore, to be atomic in nature and that they stay that way. Without
constant manual reviews, it would seem like some type of DefCore gating job would be more efficient to ensure only the expected application paths are being traversed. This also creates a burden on functional testers to work within the boundaries of both application
testing and compatibility testing, which I could see leading to duplicate test development to meet the requirements both for their project team and compatibility testing as well.</span></font></div>
<div id="magicdomid407" class="ace-line" style="margin:0px; padding:0px 1px 0px 0px">
<font face="Arial, sans-serif" class=""><span class="" style="line-height:16px; white-space:pre-wrap"><br>
</span></font></div>
<div id="magicdomid407" class="ace-line" style="margin:0px; padding:0px 1px 0px 0px">
<font face="Arial, sans-serif" class=""><span class="" style="line-height:16px; white-space:pre-wrap">My other concern is that unit test based tests are not always the best tool to describe an expected behavior or experience. Their strong suite is validate
very specific outcomes, which do not always translate well back to a user experience.
</span></font></div>
<div id="magicdomid407" class="ace-line" style="margin:0px; padding:0px 1px 0px 0px">
<font face="Arial, sans-serif" class=""><span class="" style="line-height:16px; white-space:pre-wrap"><br>
</span></font></div>
<div id="magicdomid407" class="ace-line" style="margin:0px; padding:0px 1px 0px 0px">
<font face="Arial, sans-serif" class=""><span class="" style="line-height:16px; white-space:pre-wrap">If the goal is to have a set of tests that clearly define expectations of behavior, I could see behavior driven development style of testing being much more
explicit about what functionality is being tested and what the expected outcomes should be (
<a href="http://heynemann.github.io/pyvows/">http://heynemann.github.io/pyvows/</a> and
<a href="http://pythonhosted.org/behave/">http://pythonhosted.org/behave/</a> are runners I've worked with in the past). I've spent a lot of time over the past several years trying to shoehorn rich, descriptive tests into unittest-style frameworks, and at the
best it works just good enough. I'm not suggesting BDD as a solution, but more to better understand the problem that is trying to be solved.
</span></font></div>
<div id="magicdomid407" class="ace-line" style="margin:0px; padding:0px 1px 0px 0px">
<font face="Arial, sans-serif" class=""><span class="" style="line-height:16px; white-space:pre-wrap"><br>
</span></font></div>
<div id="magicdomid407" class="ace-line" style="margin:0px; padding:0px 1px 0px 0px">
<font face="Arial, sans-serif" class=""><span class="" style="line-height:16px; white-space:pre-wrap">Thanks,</span></font></div>
<div id="magicdomid407" class="ace-line" style="margin:0px; padding:0px 1px 0px 0px">
<font face="Arial, sans-serif" class=""><span class="" style="line-height:16px; white-space:pre-wrap"><br>
</span></font></div>
<div id="magicdomid407" class="ace-line" style="margin:0px; padding:0px 1px 0px 0px">
<font face="Arial, sans-serif" class=""><span class="" style="line-height:16px; white-space:pre-wrap">Daryl</span></font></div>
</div>
</div>
</body>
</html>