<html>
<head>
<meta content="text/html; charset=utf-8" http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
<p><br>
</p>
<br>
<div class="moz-cite-prefix">On 04/05/17 15:01, Ghanshyam Mann
wrote:<br>
</div>
<blockquote
cite="mid:CACE3TKWcoppSeXP3QtD7CTa-hEDWZt=kParoavgntZ0R5iKPWw@mail.gmail.com"
type="cite">
<div><br>
<div class="gmail_quote">
<div>On Wed, May 3, 2017 at 21:57 Andrea Frittoli <<a
moz-do-not-send="true"
href="mailto:andrea.frittoli@gmail.com">andrea.frittoli@gmail.com</a>>
wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
<div>
<div class="gmail_quote">
<div>On Tue, May 2, 2017 at 2:41 PM Jordan Pittier <<a
moz-do-not-send="true"
href="mailto:jordan.pittier@scality.com"
target="_blank">jordan.pittier@scality.com</a>>
wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
<div>
<div class="gmail_extra">
<div class="gmail_quote">On Tue, May 2, 2017 at
7:42 AM, Ghanshyam Mann <span><<a
moz-do-not-send="true"
href="mailto:ghanshyammann@gmail.com"
target="_blank">ghanshyammann@gmail.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote"
style="margin:0px 0px 0px
0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">In Cinder,
there are many features/APIs which are backend
specific and<br>
will return 405 or 501 if same is not
implemented on any backend [1].<br>
If such tests are implemented in Tempest, then
it will break some gate<br>
where that backend job is voting. like ceph
job in glance_store gate.<br>
<br>
There been many such cases recently where ceph
jobs were broken due to<br>
such tests and recently it is for force-delete
backup feature[2].<br>
Reverting force-delete tests in [3]. To
resolve such cases at some<br>
extend, Jon is going to add a white/black list
of tests which can run<br>
on ceph job [4] depends on what all feature
ceph implemented. But this<br>
does not resolve it completely due to many
reason like<br>
1. External use of Tempest become difficult
where user needs to know<br>
what all tests to skip for which backend<br>
2. Tempest tests become too specific to
backend.<br>
<br>
Now there are few options to resolve this:<br>
1. Tempest should not tests such API/feature
which are backend<br>
specific like mentioned by api-ref like[1].<br>
</blockquote>
</div>
</div>
</div>
<div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>So basically, if one of the 50 Cinder
driver doesn't support a feature, we should
never test that feature ? What about the 49
other drivers ? If a feature exists and can be
tested in the Gate (with whatever default
config/driver is shipped) then I think we
should test it.</div>
</div>
</div>
</div>
<div>
<div class="gmail_extra">
<div class="gmail_quote">
<div> </div>
<blockquote class="gmail_quote"
style="margin:0px 0px 0px
0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
2. Tempest test can be disabled/skip based on
backend. - This is not<br>
good idea as it increase config options and
overhead of setting those.<br>
</blockquote>
</div>
</div>
</div>
<div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>Using regex and blacklist, any 3rd party CI
can skip any test based on the test ID.
Without introducing a config flag. See: <a
moz-do-not-send="true"
href="https://github.com/openstack-infra/project-config/blob/1cea31f402b6b0cccc47cde203c12184b5392c90/jenkins/jobs/devstack-gate.yaml#L1871"
target="_blank">https://github.com/openstack-infra/project-config/blob/1cea31f402b6b0cccc47cde203c12184b5392c90/jenkins/jobs/devstack-gate.yaml#L1871</a></div>
</div>
</div>
</div>
</blockquote>
<div><br>
</div>
</div>
</div>
<div>
<div class="gmail_quote">
<div>This way each 3rd party system has to maintain its
own list, which has the advantage that</div>
<div>different teams maintain their own list (which is
nice from an ownership and scale pov).</div>
<div><br>
</div>
<div>However I think such list of tests are not as
re-usable as having a devstack plugin (or an</div>
<div>ansible or puppet module) changing a few tempest
config options. </div>
</div>
</div>
</blockquote>
<div><br>
</div>
<div>Humm, I am little bit hesitate to go that way. For gate
and CI it might be good solution but for production cloud
testing it makes Tenpest difficult to use.</div>
<div><br>
</div>
<div>If I use Tempest to test my cloud, few tests going to
fail as those were not supported by cinder driver my cloud
has or going to have. </div>
<div>I do not have any way to configure something so that test
can be disabled. Instead I need to maintain list of tests to
run or skip. And that list is not static, it grows
dynamically. </div>
<div>This makes Tempest difficult to use. </div>
<div><br>
</div>
<div><br>
</div>
</div>
</div>
</blockquote>
<br>
Agree. We (Catalyst Cloud based in NZ) are using Tempest as the
monitoring tool and CI/CD gate for our cloud. But we do have to
maintain a white list of test cases because there are some cases are
not fitting our cloud.<br>
<br>
<blockquote
cite="mid:CACE3TKWcoppSeXP3QtD7CTa-hEDWZt=kParoavgntZ0R5iKPWw@mail.gmail.com"
type="cite">
<div>
<div class="gmail_quote">
<div><br>
</div>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
<div>
<div class="gmail_quote">
<div><br>
</div>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
<div>
<div class="gmail_extra">
<div class="gmail_quote">
<div> </div>
<blockquote class="gmail_quote"
style="margin:0px 0px 0px
0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
3. Tempest test can verify behavior with if
else condition as per<br>
backend. This is bad idea and lose the test
strength.<br>
</blockquote>
</div>
</div>
</div>
<div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>Yeah, that's bad. </div>
</div>
</div>
</div>
<div>
<div class="gmail_extra">
<div class="gmail_quote">
<blockquote class="gmail_quote"
style="margin:0px 0px 0px
0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
<br>
IMO options 1 is better options. More feedback
are welcome. </blockquote>
<blockquote class="gmail_quote"
style="margin:0px 0px 0px
0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
<br>
..1 <a moz-do-not-send="true"
href="https://developer.openstack.org/api-ref/block-storage/v3/?expanded=force-delete-a-backup-detail#force-delete-a-backup"
rel="noreferrer" target="_blank">https://developer.openstack.org/api-ref/block-storage/v3/?expanded=force-delete-a-backup-detail#force-delete-a-backup</a><br>
..2 <a moz-do-not-send="true"
href="https://bugs.launchpad.net/glance/+bug/1687538"
rel="noreferrer" target="_blank">https://bugs.launchpad.net/glance/+bug/1687538</a><br>
..3 <a moz-do-not-send="true"
href="https://review.openstack.org/#/c/461625/"
rel="noreferrer" target="_blank">https://review.openstack.org/#/c/461625/</a><br>
..4 <a moz-do-not-send="true"
href="http://lists.openstack.org/pipermail/openstack-dev/2017-April/115229.html"
rel="noreferrer" target="_blank">http://lists.openstack.org/pipermail/openstack-dev/2017-April/115229.html</a><br>
<br>
-gmann<br>
<br>
__________________________________________________________________________<br>
OpenStack Development Mailing List (not for
usage questions)<br>
Unsubscribe: <a moz-do-not-send="true"
href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe"
rel="noreferrer" target="_blank">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
<a moz-do-not-send="true"
href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev"
rel="noreferrer" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
</blockquote>
</div>
</div>
</div>
__________________________________________________________________________<br>
OpenStack Development Mailing List (not for usage
questions)<br>
Unsubscribe: <a moz-do-not-send="true"
href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe"
rel="noreferrer" target="_blank">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
<a moz-do-not-send="true"
href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev"
rel="noreferrer" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
</blockquote>
</div>
</div>
__________________________________________________________________________<br>
OpenStack Development Mailing List (not for usage questions)<br>
Unsubscribe: <a moz-do-not-send="true"
href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe"
rel="noreferrer" target="_blank">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
<a moz-do-not-send="true"
href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev"
rel="noreferrer" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
</blockquote>
</div>
</div>
<div dir="ltr">-- <br>
</div>
<div data-smartmail="gmail_signature">
<div dir="ltr">
<div>-gmann</div>
</div>
</div>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<br>
<pre wrap="">__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: <a class="moz-txt-link-abbreviated" href="mailto:OpenStack-dev-request@lists.openstack.org?subject:unsubscribe">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a>
<a class="moz-txt-link-freetext" href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a>
</pre>
</blockquote>
<br>
<pre class="moz-signature" cols="72">--
Cheers & Best regards,
Feilong Wang (王飞龙)
--------------------------------------------------------------------------
Senior Cloud Software Engineer
Tel: +64-48032246
Email: <a class="moz-txt-link-abbreviated" href="mailto:flwang@catalyst.net.nz">flwang@catalyst.net.nz</a>
Catalyst IT Limited
Level 6, Catalyst House, 150 Willis Street, Wellington
-------------------------------------------------------------------------- </pre>
</body>
</html>