Open Stack

On Thu, Jun 01, 2017 at 11:09:56AM +0100, Chris Dent wrote:
> A lot of this results, in part, from there being no single guiding
> pattern and principle for how (and where) the tests are to be
> managed. 

It sounds like you want to write a general testing guide for openstack.
Have you started this effort anywhere? I don't think anyone would be opposed
to starting a document for that, it seems like a reasonable thing to have.
But, I think you'll find there is not a one size fits all solution though,
because every project has their own requirements and needs for testing. 

> When there's a choice between one, some and all, "some" is
> almost always the wrong way to manage something. "some" is how we do
> tempest (and fair few other OpenStack things).
> 
> If it is the case that we want some projects to not put their tests
> in the main tempest repo then the only conceivable pattern from a
> memorability, discoverability, and equality standpoint is actually
> for all the tests to be in plugins.
> 
> If that isn't possible (and it is clear there are many reasons why
> that may be the case) then we need to be extra sure that we explore
> and uncover the issues that the "some" approach presents and provide
> sufficient documentation, tooling, and guidance to help people get
> around them. And that we recognize and acknowledge the impact it has.

So have you read the documentation:

https://docs.openstack.org/developer/tempest/ (or any of the other relevant
documentation

and filed bugs about where you think there are gaps? This is something that
really bugs me sometimes (yes the pun is intended) just like anything else this
is all about iterative improvements. These broad trends are things tempest
and (every project hopefully) have been working on. But improvements don't
just magically occur overnight it takes time to implement them.

Just compare the state of the documentation and tooling from 2 years ago (when
tempest started adding the plugin interface) to today. Things have steadily
improved over time and the situation now is much better. This will continue and
in the future things will get even better.

The thing is this is open source collaborative development and there is an
expectation that people who have issues with something in the project will
report them or contribute a fix and communicate with the maintainers. The users
of tempest's plugin interface tend to be other openstack projects (but not
exclusively) and if there are something that's not clear we need to work
together to fix them.

Based on this paragraph I feel like you think the decision to add a tempest
plugin interface and decrease it's scope was taken lightly without forethought
or careful consideration. But, it's the exact opposite there was extensive
debate and exploration of the problem space and took a long time to reach a
consensus.

> 
> If the answer to that is "who is going to do that?" or "who has the
> time?" then I ask you to ask yourself why we think the "non-core"
> projects have time to fiddle about with tempest plugins?

I think this unfair simplification, no one is required to write a tempest
plugin it's a choice the projects made. While I won't say the interface
is perfect, things are always improving. If a project chooses to write
a plugin, the expectation is that we'll all work together to help fix
issues as they are encountered. No individual can do everything by themselves and
it's a shared group effort. But, even so there is no shortage of work for
anyone, it's all about prioritization of effort.

> 
> And finally, I actually don't have too strong of a position in the
> case of tempest and tempest plugins. What I take issue with is the
> process whereby we discuss and decide these things and characterize
> the various projects
> 
> If I have any position on tempest at all it is that we should limit
> it to gross cloud validation and maybe interop testing, and projects
> should manage their own integration testing in tree using whatever
> tooling they feel is most appropriate. If that turns out to be
> tempest, cool.

I fail to see how this is any different than how things work today. No one is
required to use a tempest plugin and they can write tests however they want.
Tempest itself has a well defined scope (which does evolve over time like any
other project) and doesn't try to be all the testing everywhere. Almost every
other project has it's own in tree testing outside of tempest or tempest
plugins. Also, projects which have in-tree tempest tests also have tempest
plugins to expand on that set of functionality.

-Matt Treinish
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: not available
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20170601/3b8dea95/attachment.sig>