Open Stack

On Mon, Jul 13, 2015 at 07:50:59AM -0400, Emilien Macchi wrote:
> 
> 
> On 07/12/2015 11:29 PM, Matt Fischer wrote:
> > We used Tempest for a time against our production environment. It was a
> > pain to clean up but ephemeral test jobs solves that for you. A few
> > questions: 
> > 
> > What version of tempest will be using?
> 
> master for now. We will see if new tags are created later.
> AFIK, tags mean Tempest does not support an old version of OpenStack
> (ie, tempest 5 only support juno / kilo / current (liberty)).

We push a tempest tag for basically 3 reasons:

1. Support for a new release, for example tempest-4 was used to mark the start
of supporting the kilo release (we pushed it at the ~kilo release) prior to this
point in the git history it wasn't guaranteed that tempest will work with kilo

2. The EOL of a stable branch support. This is what tempest-5 indicates, that
after the tag tempest isn't testing against icehouse code so it may or may
not work.

3. A major milestone in tempest development. An example for this one is
tempest-3 which we used to mark the removal of all the xml tests from tempest.

> 
> > Will we maintain a blacklist if there are known failures? (although this
> > is a pain to keep updated)
> 
> I don't think we'll have to, since tempest already provide some Python
> decorators to skip some bugs. If that happens, yes we'll have to find a
> way to exclude some tests or disable the run at all. That's why having
> it running without affecting job return code does not break anything now.

Honestly, you should be really reluctant to skip tests, and it should only be a
last resort. As for the skip decorators that's really only useful for skipping
from inside tempest. We use a skip decorator if there is a bug with a failure
rate is sufficiently high that it's blocking anything from landing and the cause
is unknown or the fix is far enough away that we need a faster way around. But
this is not normal operating procedure as is only done in very specific cases.

That being said, if you really need to do this the only selection logic you
have through testr is to leverage a regex to exclude tests.

-Matt Treinish


> 
> > On Sun, Jul 12, 2015 at 8:44 PM, Emilien Macchi
> > <emilien.macchi at gmail.com <mailto:emilien.macchi at gmail.com>> wrote:
> > 
> >     Hi,
> > 
> >     I would like to propose to run Tempest at the end of the beaker
> >     jobs, for testing purpose now.
> > 
> >     As a start, we would accept 0 & 1 as return code, because this is
> >     really experimental.
> >     Though I think it will be interesting to see how it behaves,
> >     specially when we implement new configurations or plugins in our
> >     modules.
> > 
> >     I already did a PoC for puppet-keystone:
> >     https://review.openstack.org/198561
> >     (failing because it needs more work to pass keystone v3 tests but v2
> >     tests are successful).
> >     As you can see the code is really light, since we use puppet-tempest.
> > 
> >     Any suggestion is welcome !
> > 
> >     -- 
> >     Emilien Macchi
> > 
> >     __________________________________________________________________________
> >     OpenStack Development Mailing List (not for usage questions)
> >     Unsubscribe:
> >     OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> >     <http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe>
> >     http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> > 
> > 
> > 
> > 
> > __________________________________________________________________________
> > OpenStack Development Mailing List (not for usage questions)
> > Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> > 
> 
> -- 
> Emilien Macchi
> 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 819 bytes
Desc: not available
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20150713/477a8e2b/attachment.pgp>