Open Stack

On Thu, 1 Jun 2017, Matthew Treinish wrote:

> On Thu, Jun 01, 2017 at 11:09:56AM +0100, Chris Dent wrote:
>> A lot of this results, in part, from there being no single guiding
>> pattern and principle for how (and where) the tests are to be
>> managed.
>
> It sounds like you want to write a general testing guide for openstack.
> Have you started this effort anywhere? I don't think anyone would be opposed
> to starting a document for that, it seems like a reasonable thing to have.
> But, I think you'll find there is not a one size fits all solution though,
> because every project has their own requirements and needs for testing.

No, I haven't made any decisions about what ought to happen. I'm
still trying to figure out if there is a problem, a suite of
problems, or everything is great. Knowing what the problems are
tends to be a reasonable thing to do before proposing or
implementing solutions, especially if we want those solutions to be
most correct.

> So have you read the documentation:
>
> https://docs.openstack.org/developer/tempest/ (or any of the other relevant
> documentation
>
> and filed bugs about where you think there are gaps? This is something that
> really bugs me sometimes (yes the pun is intended) just like anything else this
> is all about iterative improvements. These broad trends are things tempest
> and (every project hopefully) have been working on. But improvements don't
> just magically occur overnight it takes time to implement them.

This is a huge part of the colllaboration issues I was identifying
in my previous message. Somebody says "there seems to be some
confusion here" and somebody else comes along and asks "have you
filed bugs?" or "have you proposed a solution?".

Well, "no" because like I said above I don't know what (or even _if_)
there's something to fix or the relevant foundations of the confusion.

I have some suspicions or concerns that the implicit hierarchy of
some tempests tests being in plugins and some not creates issues
with discovery, management and identification of responsible parties
and _may_ imply a lack of "level playing field".

But:

* if other people don't have those concerns it's not worth
   pursuing
* until we reach some kind of shared understanding and agreement
   about the concerns, speculating about solutions is premature

> Just compare the state of the documentation and tooling from 2 years ago (when
> tempest started adding the plugin interface) to today. Things have steadily
> improved over time and the situation now is much better. This will continue and
> in the future things will get even better.

Yes, it's great. If you feel like I was suggesting otherwise, then
my apologies for not being clear. As a general rule tempest and
other QA tools have consistently done great work in terms of
documentation and tooling. That there are plugins at all is
fantastic; that we are having discussions about how to make the most
effective and fair use of them is a sign that they work.

-- 
Chris Dent                  ┬──┬◡ﾉ(° -°ﾉ)       https://anticdent.org/
freenode: cdent                                         tw: @anticdent