Open Stack

On 03/01/2017 08:03 AM, Andrea Frittoli wrote:
> 
> 
> On Wed, Mar 1, 2017 at 12:54 PM Sean Dague <sean at dague.net
> <mailto:sean at dague.net>> wrote:
> 
>     On 03/01/2017 07:35 AM, Jordan Pittier wrote:
>     >
>     >
>     > On Wed, Mar 1, 2017 at 3:57 AM, Ghanshyam Mann
>     <ghanshyammann at gmail.com <mailto:ghanshyammann at gmail.com>
>     > <mailto:ghanshyammann at gmail.com <mailto:ghanshyammann at gmail.com>>>
>     wrote:
>     >
>     >     Doing gradual refactoring and fixing plugins time to time
>     needs lot
>     >     of wait and sync.
>     >
>     >     That needs:
>     >     1. Plugins to switch from current method usage. Plugins to
>     have some
>     >     other function or same copy paste code what current scenario base
>     >     class has.
>     >     2. Tempest patch to wait for plugin fix.
>     >     3. Plugins to switch back to stable interface once Tempest
>     going to
>     >     provide those.
>     >
>     >     This needs lot of sync between tempest and plugins and we have to
>     >     wait for tempest refactoring patch for long.
>     >
>     >     To make it more efficient, how about this:
>     >     1. Keep the scenario manger copy in tempest as it is. for plugins
>     >     usage only.
>     >
>     > Given that the refactoring effort "started" a year ago, at the current
>     > speed it may take 2 years to complete. In the mean time we will have a
>     > massive code duplication and a maintenance burden.
>     >
>     >     2. Start refactoring the scenario framework by adding more and
>     more
>     >     helper function under /common or lib.
>     >
>     > Starting a "framework" (each time I see that word, I have a bad
>     feeling)
>     > from scratch without users and usage is very very difficult. How do we
>     > know what we need in that framework and what will be actually used in
>     > tests ?
> 
>     I'm with Jordan, honestly, common functions and inheritance aren't going
>     to help anyway. My feeling is that the only way to sanely take things
>     forward is to start pivoting towards fixtures. And basically error if
>     inheritance is used in tests. The functions on base classes is what is
>     causing all the pain of rewind, because it's not a contract, but it's
>     also magically there, so people try to use it.
> 
>     This would mean that every test would have more setUp with a bunch of
>     explicit fixtures, but that's fine, it's a little more time thinking
>     through the writing of the tests, and a lot easier to read them to know
>     what is going on, and to delete them later. Basically get rid of all
>     class/superclass magic.
> 
>     > The effort was called scenario refactoring and I think that's what we
>     > should do. We should not do "start from scratch scenarios" or
>     "copy all
>     > the code and see what happens".
>     >
>     > There's no problem with plugins. We committed to have a stable
>     interface
>     > which is documented and agreed upon. It's clearly written
>     > here
>     https://docs.openstack.org/developer/tempest/plugin.html#stable-tempest-apis-plugins-may-use
>     > The rest is private, for Tempest internal use. If Tempest cores
>     disagree
>     > with that, then we should first of all put a spec and rewrite that
>     > document.
> 
>     It does seem like there is a tough love moment coming about the fact
>     that APIs were consumed that were specifically out of bounds. Either the
>     team commits to them inbounds, and there is the contract, or moves
>     forward under the existing contract.
> 
>     Though, I expect the issue is that because all the Tempest tests work in
>     this super class way, this is the only stuff people understood and copy
>     pasted. So it does feel like there has to be at least a couple of
>     instances in the Tempest tree of how the team *wants* other projects to
>     write tests that they can point people to.
> 
> 
> +1 on this. Which is why we need to refactor our scenario test to a sane
> code
> base that people can use as examples to copy paste.
> 
> The basic credentials provider services that comes with test.py is something
> that is useful to have I think - and we are working on make them a
> contract people
> can rely on. 
> 
> We had a discussion at the PTG about how to deal with those, and the
> conclusion
> was that we are going to move credentials providers to tempest.lib and keep 
> test.py more or less as it is and declare it a stable interface for plugins.
> 
> Alternatives like dropping class fixtures for good, and move everything
> to be test
> fixtures, involve too much code churn / refactor for an advantage which 
> is too little - and we wouldn't have the bandwidth to do that anyways.

So, as someone who is only an infrequence Tempest contributor now, I
personally have a pretty hard time with the magic credentials, because
it's super non obvious to me what a new test would give me off a blank
page, or how I would expose that.

More or less I had to just find Matt and ask him what to do, and then
have Jordan propose a different fix once he saw the patch. That's really
people intensive, and I think at least as big a scalability problem as
anything else.

...

That being said, there are only 46 scenario tests. They totally could be
rewritten in an ideal way as examples and deprecate / rot out the
existing scenario tests (including base classes).

	-Sean

-- 
Sean Dague
http://dague.net