[openstack-dev] [qa] [tc] [all] more tempest plugins (was Re: [tc] [all] TC Report 22)
cdent+os at anticdent.org
Fri Jun 2 09:14:06 UTC 2017
On Thu, 1 Jun 2017, Matthew Treinish wrote:
> On Thu, Jun 01, 2017 at 11:09:56AM +0100, Chris Dent wrote:
>> A lot of this results, in part, from there being no single guiding
>> pattern and principle for how (and where) the tests are to be
> It sounds like you want to write a general testing guide for openstack.
> Have you started this effort anywhere? I don't think anyone would be opposed
> to starting a document for that, it seems like a reasonable thing to have.
> But, I think you'll find there is not a one size fits all solution though,
> because every project has their own requirements and needs for testing.
No, I haven't made any decisions about what ought to happen. I'm
still trying to figure out if there is a problem, a suite of
problems, or everything is great. Knowing what the problems are
tends to be a reasonable thing to do before proposing or
implementing solutions, especially if we want those solutions to be
> So have you read the documentation:
> https://docs.openstack.org/developer/tempest/ (or any of the other relevant
> and filed bugs about where you think there are gaps? This is something that
> really bugs me sometimes (yes the pun is intended) just like anything else this
> is all about iterative improvements. These broad trends are things tempest
> and (every project hopefully) have been working on. But improvements don't
> just magically occur overnight it takes time to implement them.
This is a huge part of the colllaboration issues I was identifying
in my previous message. Somebody says "there seems to be some
confusion here" and somebody else comes along and asks "have you
filed bugs?" or "have you proposed a solution?".
Well, "no" because like I said above I don't know what (or even _if_)
there's something to fix or the relevant foundations of the confusion.
I have some suspicions or concerns that the implicit hierarchy of
some tempests tests being in plugins and some not creates issues
with discovery, management and identification of responsible parties
and _may_ imply a lack of "level playing field".
* if other people don't have those concerns it's not worth
* until we reach some kind of shared understanding and agreement
about the concerns, speculating about solutions is premature
> Just compare the state of the documentation and tooling from 2 years ago (when
> tempest started adding the plugin interface) to today. Things have steadily
> improved over time and the situation now is much better. This will continue and
> in the future things will get even better.
Yes, it's great. If you feel like I was suggesting otherwise, then
my apologies for not being clear. As a general rule tempest and
other QA tools have consistently done great work in terms of
documentation and tooling. That there are plugins at all is
fantastic; that we are having discussions about how to make the most
effective and fair use of them is a sign that they work.
Chris Dent ┬──┬◡ﾉ(° -°ﾉ) https://anticdent.org/
freenode: cdent tw: @anticdent
More information about the OpenStack-dev