Open Stack

> On Thu, Apr 27, 2017 at 11:56 PM, Matt Riedemann<mriedemos at gmail.com>  
> wrote:
>> On 4/21/2017 8:36 AM, Ferenc Horváth wrote:
>>> Dear OpenStackerz,
>>>
>>> I'd like to improve the coverage of the current test suite over some
>>> special code parts in Nova.
>>> My main target is to add a few scenarios [1] that would exercise the
>>> AggregateMultiTenancyIsolation scheduler filter.
>>> I'm also planning on adding new test cases for other filters and for
>>> some libvirt related features [2] as well.
>>>
>>> Unfortunately, [1] and [2] could not be merged into Tempest for various
>>> reasons, hence I started working on functional tests in Nova.
>>> However, since functional tests cannot be used to verify that a deployed
>>> system behaves correctly, we still need end to end tests.
>>> Therefore, I'm proposing a new Tempest test plug-in [3,4] that would be
>>> the home of currently out of tree tests.
>>> The idea is that these tests would run separately on a weekly basis to
>>> not use too much resources, but the rest of the questions are still open.
>>>
>>> Therefore, I'd appreciate any advice or review on this topic.
>>>
>>> Thank You all in advance.
>>>
>>> Best regards,
>>> Ferenc Horváth
>>>
>>> [1]https://review.openstack.org/#/c/374887/
>>> [2]https://review.openstack.org/#/c/315786/
>>> [3]https://review.openstack.org/#/c/448482/
>>> [4]https://review.openstack.org/#/c/451227/
>>>
>>>
>> __________________________________________________________________________
>>> OpenStack Development Mailing List (not for usage questions)
>>> Unsubscribe:
>> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>> As discussed in the nova meeting today [1] I think some of this could
>> make sense as a Tempest plugin, or an in-tree functional test using
>> fixtures, or an in-tree dsvm integration style job that's not Tempest
>> but runs real tests against a live devstack configuration.
>>
>> For the scheduler filter testing, I think that's something that is
>> totally doable with the in-tree functional tests using our fixtures.
>> You
>> don't need devstack for that since it's just testing the logic of the
>> scheduler filters. We have running services, database, and a live API
>> in
>> those tests, and external services like cinder/glance/neutron are
>> stubbed out. An example of a test like this is here [2]. I understand
>> you have some internal requirements for how these tests are performed
>> though, so I can't really help you there. Keep in mind if you do it
>> with
>> an in-tree functional test, it gets tested on every change and is
>> voting, whereas a periodic job is not and you only find out it's
>> broken
>> after we break it.
> I think the best of both words would be to set up a 3pp CI to run the
> extra tempest tests. This way the upstream CI doesn't have to spend
> extra resources to run the tests but we could have e2e test coverage.
> At the moment I don't think this will be realized in the near future.
>
> So as a compromise we will provide filter tests in nova functional test
> environment. I think hferenc already started adding such coverage.
You are correct, I added a scheduler filter test [1] to Nova's
functional test suite some time ago. It is ready for core review.
I'd appreciate any comments.
>
>> For testing the libvirt watchdog action we obviously need a real
>> deployment with running libvirt. I think you could do that as a
>> tempest
>> plugin or as an in-tree dsvm integration job, much like how the
>> novaclient functional tests work (those aren't tempest but they run
>> against a live devstack and execute real APIs via the nova CLI). The
>> downside is we don't have any dsvm-integration infrastructure setup in
>> Nova today, so you'd have to blaze that trail. But we've talked about
>> this as a need for a long time, so it'd be great if someone worked on
>> it. Alternatively it could be an in-tree Tempest plugin...but then we
>> can't test things like the libvirt image cache which is something we
>> could do with a dsvm-integration job I think, or the evacuate API
>> (we'd
>> have to run that in serial so it doesn't break other concurrently
>> running tests). Actually testing evacuate would be awesome though.
> I think dsvm-integration-libvirt is a nice idea so we will look the
> novaclient env to learn how that works and we will try to implement
> something similar in nova tree.
I looked at how novaclient's functional test suite works.
As far as I understand there is a functional environment in tox
in which all the test cases are executed in a common dsvm job.
Obviously, the test cases are using the CLI to communicate
with devstack.

My current idea is that we could break down the problem to
at least three work items. So, from a (very) high-level view, we
will need the following things to implement the aforementioned
nova-integration environment:
1. A new tox environment, which will run test from a directory
(which might be /nova/tests/functional/integration for example).
2. A new dsvm job, which will execute only the new integration
environment.
3. Some way to communicate with devstack. In case of tempest
or novaclient this layer is already implemented, but my guess is
that we will have to implement new clients/fixtures in nova.

I think the most critical is #3, but if we can discuss this and the idea
is mature enough then I'd like to start with proposing tests for evacuate.
> Thanks for the feedback!
> Cheers,
> gibi
>
>> In general I feel like writing a CI job for a very specific scheduler
>> filter configuration is overdoing it, unless you also made that job
>> re-configurable on the fly, like how the live migration job works [3].
>>
>> [1]
>> http://eavesdrop.openstack.org/meetings/nova/2017/nova.2017-04-27-21.00.log.html#l-111
>> [2]
>> https://github.com/openstack/nova/blob/master/nova/tests/functional/regressions/test_bug_1671648.py
>> [3]
>> https://github.com/openstack/nova/blob/master/nova/tests/live_migration/hooks/run_tests.sh
>>
>> --
>>
>> Thanks,
>>
>> Matt
>>
>> __________________________________________________________________________
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe:
>> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[1] https://review.openstack.org/#/c/464002/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20170705/726e0665/attachment.html>