<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body text="#000000" bgcolor="#FFFFFF">
<blockquote type="cite">
<pre wrap="">On Thu, Apr 27, 2017 at 11:56 PM, Matt Riedemann <a class="moz-txt-link-rfc2396E" href="mailto:mriedemos@gmail.com"><mriedemos@gmail.com></a>
wrote:
</pre>
<blockquote type="cite" style="color: #000000;">
<pre wrap="">On 4/21/2017 8:36 AM, Ferenc Horváth wrote:
</pre>
<blockquote type="cite" style="color: #000000;">
<pre wrap="">Dear OpenStackerz,
I'd like to improve the coverage of the current test suite over some
special code parts in Nova.
My main target is to add a few scenarios [1] that would exercise the
AggregateMultiTenancyIsolation scheduler filter.
I'm also planning on adding new test cases for other filters and for
some libvirt related features [2] as well.
Unfortunately, [1] and [2] could not be merged into Tempest for various
reasons, hence I started working on functional tests in Nova.
However, since functional tests cannot be used to verify that a deployed
system behaves correctly, we still need end to end tests.
Therefore, I'm proposing a new Tempest test plug-in [3,4] that would be
the home of currently out of tree tests.
The idea is that these tests would run separately on a weekly basis to
not use too much resources, but the rest of the questions are still open.
Therefore, I'd appreciate any advice or review on this topic.
Thank You all in advance.
Best regards,
Ferenc Horváth
[1] <a class="moz-txt-link-freetext" href="https://review.openstack.org/#/c/374887/">https://review.openstack.org/#/c/374887/</a>
[2] <a class="moz-txt-link-freetext" href="https://review.openstack.org/#/c/315786/">https://review.openstack.org/#/c/315786/</a>
[3] <a class="moz-txt-link-freetext" href="https://review.openstack.org/#/c/448482/">https://review.openstack.org/#/c/448482/</a>
[4] <a class="moz-txt-link-freetext" href="https://review.openstack.org/#/c/451227/">https://review.openstack.org/#/c/451227/</a>
</pre>
</blockquote>
<pre wrap="">__________________________________________________________________________
</pre>
<blockquote type="cite" style="color: #000000;">
<pre wrap="">OpenStack Development Mailing List (not for usage questions)
Unsubscribe:
</pre>
</blockquote>
<pre wrap=""><a class="moz-txt-link-abbreviated" href="mailto:OpenStack-dev-request@lists.openstack.org?subject:unsubscribe">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a>
</pre>
<blockquote type="cite" style="color: #000000;">
<pre wrap=""><a class="moz-txt-link-freetext" href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a>
</pre>
</blockquote>
<pre wrap="">As discussed in the nova meeting today [1] I think some of this could
make sense as a Tempest plugin, or an in-tree functional test using
fixtures, or an in-tree dsvm integration style job that's not Tempest
but runs real tests against a live devstack configuration.
For the scheduler filter testing, I think that's something that is
totally doable with the in-tree functional tests using our fixtures.
You
don't need devstack for that since it's just testing the logic of the
scheduler filters. We have running services, database, and a live API
in
those tests, and external services like cinder/glance/neutron are
stubbed out. An example of a test like this is here [2]. I understand
you have some internal requirements for how these tests are performed
though, so I can't really help you there. Keep in mind if you do it
with
an in-tree functional test, it gets tested on every change and is
voting, whereas a periodic job is not and you only find out it's
broken
after we break it.
</pre>
</blockquote>
<pre wrap="">I think the best of both words would be to set up a 3pp CI to run the
extra tempest tests. This way the upstream CI doesn't have to spend
extra resources to run the tests but we could have e2e test coverage.
At the moment I don't think this will be realized in the near future.
So as a compromise we will provide filter tests in nova functional test
environment. I think hferenc already started adding such coverage.</pre>
</blockquote>
You are correct, I added a scheduler filter test [1] to Nova's<br>
functional test suite some time ago. It is ready for core review.<br>
I'd appreciate any comments.<br>
<blockquote type="cite"><br>
<blockquote type="cite" style="color: #000000;">
<pre wrap="">For testing the libvirt watchdog action we obviously need a real
deployment with running libvirt. I think you could do that as a
tempest
plugin or as an in-tree dsvm integration job, much like how the
novaclient functional tests work (those aren't tempest but they run
against a live devstack and execute real APIs via the nova CLI). The
downside is we don't have any dsvm-integration infrastructure setup in
Nova today, so you'd have to blaze that trail. But we've talked about
this as a need for a long time, so it'd be great if someone worked on
it. Alternatively it could be an in-tree Tempest plugin...but then we
can't test things like the libvirt image cache which is something we
could do with a dsvm-integration job I think, or the evacuate API
(we'd
have to run that in serial so it doesn't break other concurrently
running tests). Actually testing evacuate would be awesome though.
</pre>
</blockquote>
<pre wrap="">I think dsvm-integration-libvirt is a nice idea so we will look the
novaclient env to learn how that works and we will try to implement
something similar in nova tree.</pre>
</blockquote>
I looked at how novaclient's functional test suite works.<br>
As far as I understand there is a functional environment in tox<br>
in which all the test cases are executed in a common dsvm job.<br>
Obviously, the test cases are using the CLI to communicate<br>
with devstack.<br>
<br>
My current idea is that we could break down the problem to<br>
at least three work items. So, from a (very) high-level view, we<br>
will need the following things to implement the aforementioned<br>
nova-integration environment:<br>
1. A new tox environment, which will run test from a directory<br>
(which might be /nova/tests/functional/integration for example).<br>
2. A new dsvm job, which will execute only the new integration<br>
environment.<br>
3. Some way to communicate with devstack. In case of tempest<br>
or novaclient this layer is already implemented, but my guess is<br>
that we will have to implement new clients/fixtures in nova.<br>
<br>
I think the most critical is #3, but if we can discuss this and the
idea<br>
is mature enough then I'd like to start with proposing tests for
evacuate.<br>
<blockquote type="cite">
<pre wrap="">Thanks for the feedback!
Cheers,
gibi
</pre>
<blockquote type="cite" style="color: #000000;">
<pre wrap="">
In general I feel like writing a CI job for a very specific scheduler
filter configuration is overdoing it, unless you also made that job
re-configurable on the fly, like how the live migration job works [3].
[1]
<a class="moz-txt-link-freetext" href="http://eavesdrop.openstack.org/meetings/nova/2017/nova.2017-04-27-21.00.log.html#l-111">http://eavesdrop.openstack.org/meetings/nova/2017/nova.2017-04-27-21.00.log.html#l-111</a>
[2]
<a class="moz-txt-link-freetext" href="https://github.com/openstack/nova/blob/master/nova/tests/functional/regressions/test_bug_1671648.py">https://github.com/openstack/nova/blob/master/nova/tests/functional/regressions/test_bug_1671648.py</a>
[3]
<a class="moz-txt-link-freetext" href="https://github.com/openstack/nova/blob/master/nova/tests/live_migration/hooks/run_tests.sh">https://github.com/openstack/nova/blob/master/nova/tests/live_migration/hooks/run_tests.sh</a>
--
Thanks,
Matt
__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe:
<a class="moz-txt-link-abbreviated" href="mailto:OpenStack-dev-request@lists.openstack.org?subject:unsubscribe">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a>
<a class="moz-txt-link-freetext" href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a>
</pre>
</blockquote>
</blockquote>
[1] <a class="moz-txt-link-freetext" href="https://review.openstack.org/#/c/464002/">https://review.openstack.org/#/c/464002/</a><br>
</body>
</html>