Open Stack

On 7/5/2017 4:16 AM, Ferenc Horváth wrote:
> I looked at how novaclient's functional test suite works.
> As far as I understand there is a functional environment in tox
> in which all the test cases are executed in a common dsvm job.
> Obviously, the test cases are using the CLI to communicate
> with devstack.
> 
> My current idea is that we could break down the problem to
> at least three work items. So, from a (very) high-level view, we
> will need the following things to implement the aforementioned
> nova-integration environment:
> 1. A new tox environment, which will run test from a directory
> (which might be /nova/tests/functional/integration for example).

If it's a new tox env, I'd put them in nova/tests/integration, and leave 
the 'functional' subdirectory to the "tox -e functional" tests.

> 2. A new dsvm job, which will execute only the new integration
> environment.

Yes.

> 3. Some way to communicate with devstack. In case of tempest
> or novaclient this layer is already implemented, but my guess is
> that we will have to implement new clients/fixtures in nova.

It probably depends on what needs to be tested. If you're going through 
the REST API, then a simple REST client using keystoneauth1 is probably 
good enough. We can use os-client-config for auth credentials like the 
novaclient functional job uses.

> 
> I think the most critical is #3, but if we can discuss this and the idea
> is mature enough then I'd like to start with proposing tests for evacuate.

Note that for evacuate testing, we'd need a multi-node job (minimum of 2 
nodes), and we'd also need to run the tests serially since evacuate 
requires that the nova-compute service is 'down' before you can evacuate 
instances from it.

The novaclient functional job runs it's tests serially, but is single node.

If we wanted to test evacuate cheaply without an entirely new test / job 
infrastructure, we could add a multi-node novaclient functional job 
which runs serially and tests evacuate for us - we have all of the CLIs 
we'd need to do that and the majority of the infra and test framework 
plumbing is already in place to do this.

It seems a bit odd to build that into novaclient functional testing 
rather than in something that runs against nova changes though, i.e. we 
can't verify fixes to evacuate in nova itself, and if we break something 
in evacuate because we aren't testing it in nova, then novaclient 
changes will all be blocked until that's fixed.

We'd have to weigh the pros/cons to having some testing in the 
short-term with novaclient vs the effort it would take to get a new job 
framework setup for nova.

Another alternative is we should be able to test evacuate with the 
in-tree functional tests using fixtures. That kind of testing only gives 
us so much coverage since we have to stub some things out, but it would 
at least test the mechanics of an evacuate through all of the services.

-- 

Thanks,

Matt