Open Stack

Before trying to create new group I suggest we review what we already have.
We have project Zun and Murano that do have Tempest plugin.
 

-----Original Message-----
From: Ghanshyam Mann <gmann at ghanshyammann.com> 
Sent: Sunday, March 15, 2020 12:19 PM
To: pramchan at yahoo.com
Cc: Voelker, Mark (VMware); Thierry Carrez; Kanevsky, Arkady; openstack-discuss at lists.openstack.org
Subject: Re: [Tempest] OpenSack Powered * vs OpeStack Compatible


[EXTERNAL EMAIL] 

 ---- On Sun, 15 Mar 2020 02:57:02 -0500 prakash RAMCHANDRAN <pramchan at yahoo.com> wrote ----  > Mark,  > it took me some time to get up to speed and a thank you for your thoroughful brief on the subject.
 > Like to add few more questions before we get on call to see where we go from here.
 > 5th  Area for Interoperability: Should we add latest kubernetes API testing to Tempest or adapt third party Open Source  tool to achieve K8S API compliance.

Tempest provides the plugin framework where you can write the k8s API testing. I remember that there was a proposal for a container testing tempest plugin. Or I will say check the existing tooling if that works for example: e2e testing[1].

As mentioned earlier, If you want to tests OpenStack APIs then how it is deployed ( either on k8s cluster) does not matter and Tempest existing test will work as it is.


 > Motivation :
 > A. Would Like Open Infrastructure project to align in OpenSrack to   kubernetes Cluster API for Containers and containerized workloads.
 > B. The k8s API must have tests to support kubernetes infra on Baremetal and VM.
 > C. Offer or ask each of the Core  services using Container (Docker latest) to an  internal  test for CRI, CNI and CSI certifications.
 > D. Level of Branding will be OpenStack and Kubernetes Dual-Stack certification based o. each of the service modules like Nova, Neutron, Cinder, Ceph or any other of projects within or third party k8s tools that will use to get OpebStack trusted interoperability for kubernetes Infrastructure under Open Infrastructure initiative.
 > This us just the beginning to improve Branding efforts which other community's trust us to be the guides. Let's see if this out of box thinking needs shooting down or should we revive the trailing indicators for adoption and interoperability.
 > Please advice if thusdditiin should be at end user level or can include kubeadmin APIs too?Should we support RBAC options of keystone or strictly start with K8s RBAC support in Keystone to switch between keystone v3 for VMs vs RBAC v? for Containers?
 > Think through as a team and let's see if this helps our concept if Dual-Stack support.
 > Sure we have a bunch of Container projects and they may suggest brownfield options, but stick to Greenfield first before addressing them to fall in line with commo. interoperability area 5 - dual-stack using k8s.
 > How do we setup a Call for Tempest or DefCore team  to review this. Let me have your reply to see if we can get TC members to bring this as response to ask from Thiery for rebooting DefCore Branding efforts.

You can reach out to the Tempest team on #openstack-qa IRC channel or let me know if any other communication channel you would like to discuss. 


[1] https://github.com/kubernetes/community/blob/master/contributors/devel/sig-testing/e2e-tests.md

-gmann

 > ThanksPrakash
 >
 >
 >
 >
 > Sent from Yahoo Mail on Android 
 >    On Wed, Feb 26, 2020 at 6:32 AM, Mark Voelker<mvoelker at vmware.com> wrote:   
 > On Feb 25, 2020, at 10:00 AM, Ghanshyam Mann <gmann at ghanshyammann.com> wrote:
 > ---- On Tue, 25 Feb 2020 01:16:13 -0600 prakash RAMCHANDRAN <pramchan at yahoo.com> wrote ----  > Mark,  > Glad you pointed to right code. Reviewed and stand corrected. My mis-underdtanding was I considered 20919.01 as first release and 2019.02 as second release of the year. However based on your comments and reference I understand that it is year.month of release , thus 2019.11 includes 'usuri' and previous 3 releases as listed pointed by you.
 >    "os_trademark_approval": {
 >      "target_approval": "2019.11",
 >      "replaces": "2019.06",
 >      "releases": ["rocky", "stein", "train", "ussuri"],
 >      "status": "approved"
 >    }
 >  },
 >
 >
 > That clears that I should have asked for 2019.11.
 > Few more questions on Tempedt tests.
 > I read some where that we have about 1500 Tempest tests overall. Is that correct?
 >
 > Yeah, it might be little more or less but around 1500 in-tree tests in Tempest.
 >
 > The interop code lines have gone down from 3836 lines to 3232 in train to usuri.
 > Looks contrary to growth, any comments?
 >
 > As Thierry pointed out, a chunk of this was due to the removal of the volumes-v2 API from the required list.  A few other tests have been removed over time for various other reasons (changes to or removal of tests in Tempest, etc).  Since the test lists are kept in git, you can actually walk through the complete history yourself to see why some tests were added or removed if you’d like:
 > https://opendev.org/openstack/interop/commits/branch/master
 > It’s also important to note that just because we’ve added more tests over time to Tempest, more projects over time to OpenStack, or more API’s to existing projects, that doesn’t mean there will be a corresponding increase in the number of tests required for the trademark programs.  The OpenStack Powered program is more a trailing indicator of adoption than an attempt to force commercial products to support any and all capabilities.  Each API is considered for inclusion in the program against a set of criteria detailed here:
 > https://opendev.org/openstack/interop/src/branch/master/doc/source/process/CoreCriteria.rst
 > So, for example: if a project introduced a new API in Usuri, it’s highly unlikely that it would appear in the very next Guideline because it would fail to meet several criteria.  It would be unlikely to be “widely deployed” since many deployments would still be using Stein or Train (also covered in the same Guideline).  It might not yet be supported in many third-party SDK’s or tools, so it might not yet meet the “used by tools” criteria.  It might be only supported by one or two plugins/drivers in it’s first release.  Etc, etc, etc.  Over time that API might gain adoption, meet sufficient criteria, and be added to the required list--or it might not. 
 > If you’re curious about the history of all this and the process, you might have a look at this slightly old but still mostly relevant deck:
 > https://www.slideshare.net/markvoelker/interopwg-intro-vertical-programs-jan-2017
 >
 > Question then is  60 compute and 40 storage lines I see in test cases, do we have stats for Tempest tests what's the distribution of 1500 tests across Platform, compute, storage etc. Where and how can I get that. information as documented report.
 >
 > Those should be counted from interop guidelines where you have mapping of capabilities with test cases.
 > Few or most of the capabilities have one test to verifying it or a few more than one test.
 > For example, "compute-flavors-list" capability is verified by two tests[2]. 
 > This way you can count and identify the exact number of tests per compute, storage etc.
 >
 > If you would like to know about the Tempest test categorization, you can find it from the directory  > structure. We have structured the tests service wise directory, for example, all compute tests under  > tempest/api/compute [2].
 >
 >
 > Based on above should we expect decrease or increase for say 2020.05 Vancouver release?
 >
 > Anecdotally, the Guidelines haven’t been changing very much in recent times as the “core” capabilities that have met with a lot of adoption seem fairly well established (though there have been a few new ones added and a few removed).  I wouldn’t expect dramatic changes.
 > How does one certify a kubernetes cluster running openstak modules, one module per docker container in a kubrrnetes  cluster using tempest say like in Airship Control Plane on k8s worker node, which is a OpenStack over kubernetes cluster.
 > Is this possible and if so what test we need to modify to test and certify a Containerized OpenStack in Airship as OpenStack Platform?
 >
 > I should be verified same way via Tempest. Tempest does not reply on how OpenStack is deployed it interacts via the public  > interface (where interop needs only user-facing API excluding admin API) of each service which should be accessible on  > k8s cluster also. Tests are not required to modify in this case.
 >
 > NOTE: we can always extend (writing new tests for current 6 services tests) Tempest for tests required for  > interop capabilities either that are new or improved in the verification of existing ones. I remember  > about the discussion on covering the API microversion. We keep adding new tests to cover the  > new microverison where we need integration tests but we can add API tests also if interop requires those.
 >
 > Right on.  In fact, the capabilities in the interop programs are intended to be usable independent of how a cloud is deployed (including whether it’s containerized or on bare metal, whether it uses KVM/OVS/Ceph or vSphere/NSX/vSAN, whether it’s highly available or a single-box appliance, etc).
 >
 > Can we even certify if for say 2019.11?
 >
 > The OpenStack Foundation’s interoperability programs allow you to use either of two most recently approved Guidelines for your testing (which as of right now are 2019.11 and 2019.06).  Once the Board of Directors approves the next guideline, 2019.06 will rotate out.  I should note though: the Foundation’s interoperability programs are primarily targeted at commercial products (distributions, public clouds, appliances, managed services, etc).  There’s no real reason an open source product couldn’t use these same tests of course!
 > At Your Service,
 > Mark T. Voelker
 > This should open up exciting possibilities if practical to extend OpeStack powered platform to Airship.
 > Like to hear anyone who has insight to educate us on that.
 > ThanksPrakash
 > Sent from Yahoo Mail on Android 
 >   On Mon, Feb 24, 2020 at 6:09 AM, Mark Voelker<mvoelker at vmware.com> wrote:   Hi Prakash,
 >
 > I am curious to find out if any Distribution or Products based on Openstack  Train or Usuri are seeking the latest certifications based on 2019.02.
 > Hm, there actually isn’t a 2019.02 guideline--were you perhaps referring to 2019.06 or 2019.11?  2019.06 does cover Train but not Usuri [1], 2019.11 covers both [2].  As an FYI, the OpenStack Marketplace does list which guideline a particular product was most recently tested against (refer to https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.openstack.org%2Fmarketplace%2Fdistros%2F&data=02%7C01%7Cmvoelker%40vmware.com%7C5567a591df5a45322f6808d7ba037f79%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637182396561657063&sdata=K9zbzRvFzs3cxVnCocij2Sjy43MNyejDKuRN7ivQkMk%3D&reserved=0 for example, and look for the green “TESTED” checkmark and accompanying guideline version), though this obviously doesn’t tell you what testing might be currently in flight.
 > [1] https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fopendev.org%2Fopenstack%2Finterop%2Fsrc%2Fbranch%2Fmaster%2F2019.06.json%23L75%5B2&data=02%7C01%7Cmvoelker%40vmware.com%7C5567a591df5a45322f6808d7ba037f79%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637182396561657063&sdata=Vcy8iL%2F8L93m0%2B3PIwikD5vN%2BaREJigrncz7cjd3Zyc%3D&reserved=0] https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fopendev.org%2Fopenstack%2Finterop%2Fsrc%2Fbranch%2Fmaster%2F2019.11.json%23L75&data=02%7C01%7Cmvoelker%40vmware.com%7C5567a591df5a45322f6808d7ba037f79%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637182396561667051&sdata=pTrl8xMypZJAPFu2vzQwFFOWnJu51k44oxn2PrrQI2Y%3D&reserved=0
 > At Your Service,
 > Mark T. Voelker
 >
 >
 > On Feb 22, 2020, at 5:04 AM, prakash RAMCHANDRAN <pramchan at yahoo.com> wrote:
 > Hi all,
 > I am curious to find out if any Distribution or Products based on Openstack  Train or Usuri are seeking the latest certifications based on 2019.02.
 > Similarly does any Hardware Driver of Software application seeking OpenStack compatibility Logo?
 > Finally does anyone think that Open Infra Distro like Airship or StarlingX should promote Open Infra Airship or Open Infra  StarlingX powered as a new way to promote eco system surrounding them similar to OpenStack compatible drivers and software.
 > Will then Argo, customize,Metal3.io or Ironic be qualified as Open Infra Airship compatible?
 > If so how tempest can help in testing the above comments?
 > Refer to this market place below as how Distos and Products leverage OpenStack logos and branding programs.
 > https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.openstack.org%2Fmarketplace%2Fdistros%2F&data=02%7C01%7Cmvoelker%40vmware.com%7C5567a591df5a45322f6808d7ba037f79%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637182396561667051&sdata=rABzR1tWXUHinSG4jpe46ayuELsini4fbKcJhsxrN9g%3D&reserved=0
 > Discussions and feedback are welcome. A healthy debate as how k8s modules used in Open Infra can be certified will be a good start.
 > ThanksPrakash
 >
 > Sent from Yahoo Mail on Android
 >
 > [1] https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fopendev.org%2Fopenstack%2Finterop%2Fsrc%2Fcommit%2F8f2e82b7db54cfff9315e5647bd2ba3dd6aacaad%2F2019.11.json%23L260-L281&data=02%7C01%7Cmvoelker%40vmware.com%7C5567a591df5a45322f6808d7ba037f79%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637182396561667051&sdata=wncg32ql1Gq%2B%2F7fZPUDdPzSv2Fbay2zCAXPT%2F9%2BQIo4%3D&reserved=0
 > [2] https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fopendev.org%2Fopenstack%2Ftempest%2Fsrc%2Fbranch%2Fmaster%2Ftempest%2Fapi%2Fcompute&data=02%7C01%7Cmvoelker%40vmware.com%7C5567a591df5a45322f6808d7ba037f79%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637182396561667051&sdata=mgDV%2Bng445lIrj3CjMwdTeifyGMdXyogQhnVdXk4FS0%3D&reserved=0
 >
 > -gmann
 >