Open Stack

All,

Just an update, we had a very successful DefCore Criteria meeting yesterday (notes and recording are posted).  In short, we've got a set of workable consensus criteria (14) and an approach to scoring the tests that we can use to start culling the list.  Next steps will be to actually start scoring.

Rob

Complete History > https://etherpad.openstack.org/p/DefCoreTestCriteria

MEETING 2:Interactive Discussion Scheduled for 12/30 @ 1:30pm PST


 *   Attendees: Lew Tucker, Rob HIrschfeld, Sean Roberts, Mark Radcliff, Troy Toman (joined late)

 *   Summary:  We agreed to use a TBD weighting system for criteria and locked down to 14 initial criteria
 *   Recording: http://youtu.be/VJCuaUajF90

Agenda:
    * DONE> weighting approach (15 mins)
    * DONE> re-open/review discussion existing 7+2 critiera (15 minutes)
    * DONE> additional criteria (30 mins)
    * DONE> ugly contentious items that we identified
    * DONE> exit meeting w/ usable criteria!

MEETING 3:

 *
Date TBD

Agenda:
    * Discussion of the RefStack Use Cases
    * What is our process for changing the criteria in the future
    * Begin applying criteria to tests


## Critieria

AGREED-UPON CRITERIA:
----------------------

Criteria for evaluating proposed required criteria/passing tests for the first revision/iteration
 These tests are meant to define a core set of capabilities that application developers can count on to be present in OpenStack distributions and deployments

1. Test is required stable for >2 releases (because things leaving Core are bad)

 *   the least number/amount of must pass tests as possible (due to above)

 *   but noting that the number will increase over time

 *   least amount of change from current requirements as possible (nova, swift 2 versions)

 *   (Acknowledge that deprecation is punted for now, but can be executed by TC)
2. Where the code being tested has an designed area of alternate implementation (extension framework) as per the Core Principles, there should be parity in capability tested across extensionimplementations

 *   Test is not configuration specific (test cannot meet criteria if it requires a specific configuration)

 *   Test does not require an non-open extension to pass (only the OpenStack code)

 *   Extension
3. Capability being tested is Service Discoverable (can be found in Keystone and via service introspection) - MONTY TO FIX WORDING around REST/DOCS, etc.

 *   Nearly core or "compatible" clouds need to be introspected to see what's missing

 *   Not clear at this point if it's project or capability level enforced.  Perhaps for Elephant it's project but it should move to capability for later
4. Candidates are widely used capabilities

 *   4A favor capabilities that are supported by multiple public cloud providers and private cloud products

 *   Allow the committee to use expert judgement to promote capabilities that need to resolve the "chicken-and-egg"

 *   Goals are both diversity and quantity of users

 *   (there's a follow-up about how to weight this input)

 *   4B. Should be included if supported by common tools

 *   4C. Should be included if part of common libraries (Fog, JCloud,e tc)
5. Test capabilities that are required by other must-pass tests and/or depended on by many other capabilities
6. Should reflect future technical direction (from the project technical teams and the TC)

 *   Deprecated capabilities would be excluded (or phased out)

 *   This could potentially become a "stick" if used incorrectly because we could force capabilities
7. Should be well documented, particularly the expected behavior.

 *   includes the technical references for others in the project as well as documentation for the users and or developers accessing the feature or functionality
8. A test that is a must-pass test should stay a must-pass test (makes must-pass tests sticky release per release)
9. A test for a Capability with must-pass tests is more likely to be considered must-pass
10 Capabilities is unique and cannot be build out of other must-pass capabilities

 *   Candidates favor capabilities that users cannot implement if given the presence of other capabilities

 *   consider the pain to users if a cloud doesn't have the capability - not so much pain if they can run it themselves

 *   "Unique capabilities that cannot be build out of other must-pass capabilities should not be considered as strongly"
11. Tests do not require administrative right to execute

At this time, the list is considered sufficient for validation against the actual tests and will be revisited later in the cycle.


From: Hirschfeld, Rob
Sent: Monday, December 30, 2013 9:11 AM
To: 'Defcore-committee at lists.openstack.org'
Cc: foundation-board at lists.openstack.org; 'openstack-tc at lists.openstack.org'
Subject: REMINDER: DefCore Test Selection Criteria Meeting Today 12/30 1:30 PST

OpenStack Board & TC,

I'm sending this to a broad distribution because of the interest level in core; however, this is really a DefCore subcommittee discussion meeting so attendance is welcome but not expected.  If you are new to the topics, please make sure to do a review before joining - we keep notes and do not historical reviews.

We are taking good notes and our progress can be followed on https://etherpad.openstack.org/p/DefCoreTestCriteria


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/defcore-committee/attachments/20131231/51a8d26c/attachment-0001.html>