Open Stack

On 4/21/2017 8:36 AM, Ferenc Horváth wrote:
> Dear OpenStackerz,
>
> I'd like to improve the coverage of the current test suite over some
> special code parts in Nova.
> My main target is to add a few scenarios [1] that would exercise the
> AggregateMultiTenancyIsolation scheduler filter.
> I'm also planning on adding new test cases for other filters and for
> some libvirt related features [2] as well.
>
> Unfortunately, [1] and [2] could not be merged into Tempest for various
> reasons, hence I started working on functional tests in Nova.
> However, since functional tests cannot be used to verify that a deployed
> system behaves correctly, we still need end to end tests.
> Therefore, I'm proposing a new Tempest test plug-in [3,4] that would be
> the home of currently out of tree tests.
> The idea is that these tests would run separately on a weekly basis to
> not use too much resources, but the rest of the questions are still open.
>
> Therefore, I'd appreciate any advice or review on this topic.
>
> Thank You all in advance.
>
> Best regards,
> Ferenc Horváth
>
> [1] https://review.openstack.org/#/c/374887/
> [2] https://review.openstack.org/#/c/315786/
> [3] https://review.openstack.org/#/c/448482/
> [4] https://review.openstack.org/#/c/451227/
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

As discussed in the nova meeting today [1] I think some of this could 
make sense as a Tempest plugin, or an in-tree functional test using 
fixtures, or an in-tree dsvm integration style job that's not Tempest 
but runs real tests against a live devstack configuration.

For the scheduler filter testing, I think that's something that is 
totally doable with the in-tree functional tests using our fixtures. You 
don't need devstack for that since it's just testing the logic of the 
scheduler filters. We have running services, database, and a live API in 
those tests, and external services like cinder/glance/neutron are 
stubbed out. An example of a test like this is here [2]. I understand 
you have some internal requirements for how these tests are performed 
though, so I can't really help you there. Keep in mind if you do it with 
an in-tree functional test, it gets tested on every change and is 
voting, whereas a periodic job is not and you only find out it's broken 
after we break it.

For testing the libvirt watchdog action we obviously need a real 
deployment with running libvirt. I think you could do that as a tempest 
plugin or as an in-tree dsvm integration job, much like how the 
novaclient functional tests work (those aren't tempest but they run 
against a live devstack and execute real APIs via the nova CLI). The 
downside is we don't have any dsvm-integration infrastructure setup in 
Nova today, so you'd have to blaze that trail. But we've talked about 
this as a need for a long time, so it'd be great if someone worked on 
it. Alternatively it could be an in-tree Tempest plugin...but then we 
can't test things like the libvirt image cache which is something we 
could do with a dsvm-integration job I think, or the evacuate API (we'd 
have to run that in serial so it doesn't break other concurrently 
running tests). Actually testing evacuate would be awesome though.

In general I feel like writing a CI job for a very specific scheduler 
filter configuration is overdoing it, unless you also made that job 
re-configurable on the fly, like how the live migration job works [3].

[1] 
http://eavesdrop.openstack.org/meetings/nova/2017/nova.2017-04-27-21.00.log.html#l-111
[2] 
https://github.com/openstack/nova/blob/master/nova/tests/functional/regressions/test_bug_1671648.py
[3] 
https://github.com/openstack/nova/blob/master/nova/tests/live_migration/hooks/run_tests.sh

-- 

Thanks,

Matt