Open Stack

On Thu, Apr 27, 2017 at 11:56 PM, Matt Riedemann <mriedemos at gmail.com> 
wrote:
> On 4/21/2017 8:36 AM, Ferenc Horváth wrote:
> > Dear OpenStackerz,
> >
> > I'd like to improve the coverage of the current test suite over some
> > special code parts in Nova.
> > My main target is to add a few scenarios [1] that would exercise the
> > AggregateMultiTenancyIsolation scheduler filter.
> > I'm also planning on adding new test cases for other filters and for
> > some libvirt related features [2] as well.
> >
> > Unfortunately, [1] and [2] could not be merged into Tempest for 
> various
> > reasons, hence I started working on functional tests in Nova.
> > However, since functional tests cannot be used to verify that a 
> deployed
> > system behaves correctly, we still need end to end tests.
> > Therefore, I'm proposing a new Tempest test plug-in [3,4] that 
> would be
> > the home of currently out of tree tests.
> > The idea is that these tests would run separately on a weekly basis 
> to
> > not use too much resources, but the rest of the questions are still 
> open.
> >
> > Therefore, I'd appreciate any advice or review on this topic.
> >
> > Thank You all in advance.
> >
> > Best regards,
> > Ferenc Horváth
> >
> > [1] https://review.openstack.org/#/c/374887/
> > [2] https://review.openstack.org/#/c/315786/
> > [3] https://review.openstack.org/#/c/448482/
> > [4] https://review.openstack.org/#/c/451227/
> >
> > 
> __________________________________________________________________________
> > OpenStack Development Mailing List (not for usage questions)
> > Unsubscribe: 
> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 
> As discussed in the nova meeting today [1] I think some of this could
> make sense as a Tempest plugin, or an in-tree functional test using
> fixtures, or an in-tree dsvm integration style job that's not Tempest
> but runs real tests against a live devstack configuration.
> 
> For the scheduler filter testing, I think that's something that is
> totally doable with the in-tree functional tests using our fixtures. 
> You
> don't need devstack for that since it's just testing the logic of the
> scheduler filters. We have running services, database, and a live API 
> in
> those tests, and external services like cinder/glance/neutron are
> stubbed out. An example of a test like this is here [2]. I understand
> you have some internal requirements for how these tests are performed
> though, so I can't really help you there. Keep in mind if you do it 
> with
> an in-tree functional test, it gets tested on every change and is
> voting, whereas a periodic job is not and you only find out it's 
> broken
> after we break it.

I think the best of both words would be to set up a 3pp CI to run the 
extra tempest tests. This way the upstream CI doesn't have to spend 
extra resources to run the tests but we could have e2e test coverage. 
At the moment I don't think this will be realized in the near future.

So as a compromise we will provide filter tests in nova functional test 
environment. I think hferenc already started adding such coverage.

> 
> 
> For testing the libvirt watchdog action we obviously need a real
> deployment with running libvirt. I think you could do that as a 
> tempest
> plugin or as an in-tree dsvm integration job, much like how the
> novaclient functional tests work (those aren't tempest but they run
> against a live devstack and execute real APIs via the nova CLI). The
> downside is we don't have any dsvm-integration infrastructure setup in
> Nova today, so you'd have to blaze that trail. But we've talked about
> this as a need for a long time, so it'd be great if someone worked on
> it. Alternatively it could be an in-tree Tempest plugin...but then we
> can't test things like the libvirt image cache which is something we
> could do with a dsvm-integration job I think, or the evacuate API 
> (we'd
> have to run that in serial so it doesn't break other concurrently
> running tests). Actually testing evacuate would be awesome though.

I think dsvm-integration-libvirt is a nice idea so we will look the 
novaclient env to learn how that works and we will try to implement 
something similar in nova tree.

Thanks for the feedback!
Cheers,
gibi

> 
> 
> In general I feel like writing a CI job for a very specific scheduler
> filter configuration is overdoing it, unless you also made that job
> re-configurable on the fly, like how the live migration job works [3].
> 
> [1]
> http://eavesdrop.openstack.org/meetings/nova/2017/nova.2017-04-27-21.00.log.html#l-111
> [2]
> https://github.com/openstack/nova/blob/master/nova/tests/functional/regressions/test_bug_1671648.py
> [3]
> https://github.com/openstack/nova/blob/master/nova/tests/live_migration/hooks/run_tests.sh
> 
> --
> 
> Thanks,
> 
> Matt
> 
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: 
> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev