Open Stack

I like os-faults library which can provide the abstraction over different destructive actions like reboot/poweroff node etc.

But not much clear about Stepler that what all tests it will contain. Tempest do have scenario tests which can hit the production env with significant way of testing scenario.
It can cover the end user scenario also which involves the interaction of public OpenStack APIs and always in enhancement state by adding more and more scenario tests.

Few query over Stepler as separate project:

1.       Is Stepler will cover only tests which hits the node level action(reboot, HA etc)?

2.       This should not mix the scenario tests which are in Tempest scope because that can make confusion for developers (where to add scenario tests) as well as for tester(from where to run what scenario tests).

3.       How we make sure those tests run fine or at least run while adding.

a.       I think devstack enhancement for multi-node etc can do this as mentioned by you also.

b.      If so then why not adding those tests in Tempest only using os-faults lib ?

Overall I feel os-faults  as lib is really nice idea but tests scope can go in Tempest under HW_scenario  (or something else) till it does not break basic principle of Tempest.

Thanks
gmann

From: Timur Nurlygayanov [mailto:tnurlygayanov at mirantis.com]
Sent: 06 October 2016 20:09
To: OpenStack Development Mailing List (not for usage questions) <openstack-dev at lists.openstack.org>
Subject: Re: [openstack-dev] [QA] The end-user test suite for OpenStack clusters

Ken, it is a good idea!

The plan is to develop os-faults as a library which will be able
to manage cluster nodes and OpenStack services on these nodes.
It is a good idea to add some Tempest tests which will use
os-faults library as well for some API tests.

The Stepler framework [1] will use os-faults to perform all destructive
actions in the clouds (the reboot of nodes, restart of OpenStack services,
enable/disable network interfaces or some firewall rules and etc).

We need to get +1 from you in [1] to create the repository with
advanced end-user scenario tests.

Thank you!

[1] https://review.openstack.org/#/c/374667/

On Tue, Oct 4, 2016 at 8:53 PM, Yaroslav Lobankov <ylobankov at mirantis.com<mailto:ylobankov at mirantis.com>> wrote:
Hi Ken,

OS-Faults doesn't have any scenarios in the tree yet (the project is two months old), but you can find some examples of the use in the os-faults/examples directory.

Regards,
Yaroslav Lobankov.

On Tue, Oct 4, 2016 at 8:02 PM, Ken'ichi Ohmichi <ken1ohmichi at gmail.com<mailto:ken1ohmichi at gmail.com>> wrote:
Hi Timur,

Thanks for your explanation.

2016-09-29 6:22 GMT-07:00 Timur Nurlygayanov <tnurlygayanov at mirantis.com<mailto:tnurlygayanov at mirantis.com>>:
>
>> I am guessing the above "restart nodes" is for verifying each
>> OpenStack service restarts successfully, right?
>
> Yes, this is right. And we also will check that HA logic for these
> services works correctly (for example, rescheduling of L3 Neutron
> agents for networks).
>
>> But these service scripts are provided by distributors, and Devstack
>> itself doesn't contain service scripts IIUC.
>> So I'd like to know how to verify it on Devstack clouds.
>
> Yes, DevStack doesn't support many scenarios which are actual
> and should be supported on the production clouds.
> It will be not possible to run all advanced test scenarios for DevStack
> clouds,
> just because DevStack can't deploy OpenStack cloud with 3 controllers
> now (so, probably it will be possible in the future).
>
> Of course, some advanced scenarios will support DevStack clouds,
> for example, some test scenarios which are based on customer-found
> issues from the real production clouds, like upload of the large images
> (100+ Gb)
> to Glance with Swift backend. Such cases are important for verification of
> pre-production environments, but not very important for CI gate jobs.
>
> It is also important to note that in these advanced cases we are targeting
> to check not only the logic of Python code, but also the correct
> configuration
> of all OpenStack components on some pre-production OpenStack clusters.
I guessed some part of os-faults can be moved to Tempest if os-faults
contains API tests for enabling/disabling OpenStack services.
Then, os-faults would be able to concentrate on more destructive tests
like rebooting physical nodes, etc.
However, I could not find any actual scenarios on current os-faults
(https://github.com/openstack/os-faults).
That seems to just contain some abstraction layers and unit tests. Can
we see actual test scenarios of os-faults ?
Maybe I missed something.

Thanks
Ken Ohmichi

__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe<http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe>
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe<http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe>
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



--

Timur,
Senior QA Manager
OpenStack Projects
Mirantis Inc
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20161014/c41396a5/attachment.html>