Open Stack

Hi stackers,

Intro (Gabbi)
-------------------------

Gabbi is amazing tool that allows you to describe in human readable way
what API requests to execute and what you are expecting as a result. It
Simplifies a lot API testing.

It's based on unittest so it can be easily run using tox/tester/nose and so
on..


Intro (Functional in-tree tests)
---------------------------------------

Keeping all tests in one project like Tempest, that is maintained by one
team, was not enough scalable approach. To scale things, projects started
maintaining their own functional tests in their own tree. This resolves
scale issues and now new features can be merged with functional tests.


The Problem
-----------------

As far as you know there are a lot of OpenStack projects with their own
functional tests / gabbi tests in tree. It becomes hard for developers,
devops and operators to work with them. (Like it's hard to install
OpenStack by hands without DevStack. )

Usually, end users are choosing 2 approach:
1) Make own tests
2) Make scripts that runs somehow all these tests


Small Intro (Rally)
------------------------

Rally idea is to make tool that simplifies all kinds of testing of multiple
OpenStack clouds.
It should be for human and as well simple to integrated in CI/CD process.

Rally automates all testing process (managing testing systems / running
tests / storing results / working with results)

At this moment there are 3 major parts:
*) deployment  - manages OpenStack deployments (creates or uses existing)
*) verify - manages fully tempest (installs/configurtion/running/parsing
output/storing results/working with results)
*) task - own rally testing framework that allows you to do all kinds of
testing functional/load/performance/scale/load/volume and others.

I can say that "rally verify" command that automates work with Tempest is
very popular. More details here:
https://www.mirantis.com/blog/rally-openstack-tempest-testing-made-simpler/


Proposal to make the life better
------------------------------------------

Recently Yair Fried and Prasanth Anbalagan proposed a great idea to extend
"rally verify" command to add ability to run in-tree functional tests in
the same way as tempest.

In other words to have next syntax:  "rally verify <project> <command>"

Something like this:

  rally verify swift start   # 1. Check is installed swift for active rally
deployment.
                                     # IF NO:
                                     #   Downloads from default (our
specified place) swift
                                     #   Switch to master or specified tag
                                     #   Installs in venv swift
                                     #   Configure swift functional test
config for active deployment
                                     # 2. Run swift functional test
                                     # 3. Parse subunit output and store to
Rally DB (for future work)

  rally verify swift list                                  # List all swift
verification runs
  rally verify swift show UUID                    # Shows results
  rally verify swift compare UUID1 UUID2 # Compare results of two runs


Why it makes sense?
----------------------------

1) Unification of testing process.

There is a simple to learn set of command "rally verify <project> <cmd>"
that works for all projects in the same way.  End users like such things=)

2) Simplification of testing process.

"rally verify <project> start" - will automate all steps, so you won't need
to install project manually, and configure functional test, collect and
somewhere store results.

3) Avoiding duplication of effort

We don't need to implement part of "rally verify" functionality in every
project.
It is better to implement it in one place with plugin support. Adding new
project means implementing new plugin (in most case it will be just
functional test conf generation)

4) Reusing already existing code

Most of the code that we need is already implemented in Rally,
it requires just small refactoring and generalization.


Thoughts?


Best regards,
Boris Pavlovic
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20150307/b06475b3/attachment.html>