Open Stack

In advance of the meeting I have some notes and observations regarding testing Havana advisory capabilities against my test cloud. I encourage anyone else  to contribute their own observations and suggestions.

https://etherpad.openstack.org/p/havana-advisory-testing-notes <https://etherpad.openstack.org/p/havana-advisory-testing-notes>

Chris Hoge
Interop Engineer
OpenStack Foundation

> On Feb 3, 2015, at 10:21 PM, Rob Hirschfeld <rob at zehicle.com> wrote:
> 
> All,
> 
> 
> 
> Sorry for the late updates, I'm pulling together background material for tomorrow's meeting (It's still tomorrow in PT) and I wanted to drop some links and material that we will review in the meeting.
> 
> 
> 
> Sources of text copied below:
> 
> Rob's Blog about process & flow chart: http://robhirschfeld.com/2014/09/02/defcore-process-flow/ <http://robhirschfeld.com/2014/09/02/defcore-process-flow/>
> Detailed Process: https://etherpad.openstack.org/p/DefCoreLighthouse.F2F <https://etherpad.openstack.org/p/DefCoreLighthouse.F2F> <https://etherpad.openstack.org/p/DefCoreLighthouse.F2F>
> 
> Presented to the Board in December 2014:
> 
> 
> 
> The draft process flow chart was provided to the Board at our OSCON meeting without additional review. It below boils down to a few key points:
> 
> We are using the documents in the Gerrit review process to ensure that we work within the community processes.
> Going forward, we want to rely on the technical leadership to create, cluster and describe capabilities. DefCore bootstrapped this process for Havana. Further, Capabilities are defined by tests in Tempest so test coverage gaps (like Keystone v2) translate into Core gaps.
> We are investing in data driven and community involved feedback (via Refstack) to engage the largest possible base for core decisions.
> There is a “safety valve” for vendors to deal with test scenarios that are difficult to recreate in the field.
> The Board is responsible for approving the final artifacts based on the recommendations. By having a transparent process, community input is expected in advance of that approval.
> The process is time sensitive. There’s a need for the Board to produce Core definition in a timely way after each release and then feed that into the next one. Ideally, the definitions will be approved at the Board meeting immediately following the release.
> 
> 
> Drafted at the June 2014 DefCore Face to Face:
> 
> 
> 
> ## PROCESS FOR DEFINITION OF CORE
> 
> 
> 
> Today's core definitions are tightly coupled to individual projects.Tomorrow's core definitions will be individual capabilities of different projects.
> 
> 
> 
> DEFINITION: Certification to use the OpenStack mark requires passing tests of capabilities, and including designated sections of code.
> 
> 
> 
> A. TESTS
> 
> All tempest tests for OpenStack must be grouped into capabilities.
> Tests that don't pass in trunk (or that can only be passed by certain configurations) can be “flagged” by DefCore, and will be skipped.
> These capabilities are scored by the DefCore committee during each release cycle, using board-approved criteria.
> The board defines the criteria for this scoring and may adjust relative importance of criteria with each release.
> The board defines the cut-off score for determining that a capability is core
> Capabilities that surpass the cutoff score are considered “core capabilities”.
> In each core capability, all non-flagged tests must pass in any OpenStack product or service that uses the mark
> Capabilities will not be depricated without with at least 1 release notice
> 
> B. DESIGNATED SECTIONS
> 
> The PTL of each project will identify sections of code to be “designated” which must be used in OpenStack products and services.
> The TC will ratify these designated sections.
> Designated sections only apply to projects that have some core capabilities.
> Designated sections will not be increased without at least 1 release notice
> 
> C. ACTIVITIES(Work Flow?)
> 
> Criteria changes are managed by the DefCore committee and ratified by Board vote and posted on the OpenStack Wiki
> Designated Sections changes are managed by the TC as Gerrit proposals
> Capabilities changes are managed via Gerrit reviews
> all tests are categorized into a capability by the PTL through Gerrit patch of the JSON file
> tests are flagged-to-skip by approval of patches to the refstack json (by the Board as a slate vote)
> Tests are unflagged through the same mechanism (by action of the DefCore committee)
> DefCore changes to flagged to skip tests, criteria are to be ratified by the Board
> 
> D. GRIEVANCES
> 
> Test Issues > via Gerrit to Capabilities list
> Critiera Issues > via DefCore mailing list & meetings
> Designated Sections > via TC processes
> Compliance Violations > same as trademark issues ACTION> SETUP trademark at openstack.org
> Capabilities scores -> Community forums and roundtables during matrix review / also through Gerrit
> 
> E. PUBLISHING
> 
> The DefCore committee will manage content of a community website (currently at http://RefStack.org) <http://refstack.org%29/> and publish:
> The DefCore principles
> The current and historical scoring criteria
> The categorization of tests into capabilities
> The current and historical scored matrixes of capabilities
> The scorecards of licensed products and services
> Only “pass” tests will be reported in the process
> OpenStack retains the right to be the official source of the results, and will copyright the RefStack and DefCore materials to support this.
> 
> F. TIMELINES:
> 
> Vendors will anonymously self-certify their products and services under the DefCore program for the Juno release.
> In the “K” release, the Vendor's scorecards will be made public on the RefStack website.
> Process modifications will be applied to future releases.
> 
> _______________________________________________
> Defcore-committee mailing list
> Defcore-committee at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/defcore-committee

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/defcore-committee/attachments/20150204/b1c379c9/attachment.html>