[openstack-dev] [QA]Refactoring Scenarios manager.py
sean at dague.net
Wed Mar 1 13:20:36 UTC 2017
On 03/01/2017 08:03 AM, Andrea Frittoli wrote:
> On Wed, Mar 1, 2017 at 12:54 PM Sean Dague <sean at dague.net
> <mailto:sean at dague.net>> wrote:
> On 03/01/2017 07:35 AM, Jordan Pittier wrote:
> > On Wed, Mar 1, 2017 at 3:57 AM, Ghanshyam Mann
> <ghanshyammann at gmail.com <mailto:ghanshyammann at gmail.com>
> > <mailto:ghanshyammann at gmail.com <mailto:ghanshyammann at gmail.com>>>
> > Doing gradual refactoring and fixing plugins time to time
> needs lot
> > of wait and sync.
> > That needs:
> > 1. Plugins to switch from current method usage. Plugins to
> have some
> > other function or same copy paste code what current scenario base
> > class has.
> > 2. Tempest patch to wait for plugin fix.
> > 3. Plugins to switch back to stable interface once Tempest
> going to
> > provide those.
> > This needs lot of sync between tempest and plugins and we have to
> > wait for tempest refactoring patch for long.
> > To make it more efficient, how about this:
> > 1. Keep the scenario manger copy in tempest as it is. for plugins
> > usage only.
> > Given that the refactoring effort "started" a year ago, at the current
> > speed it may take 2 years to complete. In the mean time we will have a
> > massive code duplication and a maintenance burden.
> > 2. Start refactoring the scenario framework by adding more and
> > helper function under /common or lib.
> > Starting a "framework" (each time I see that word, I have a bad
> > from scratch without users and usage is very very difficult. How do we
> > know what we need in that framework and what will be actually used in
> > tests ?
> I'm with Jordan, honestly, common functions and inheritance aren't going
> to help anyway. My feeling is that the only way to sanely take things
> forward is to start pivoting towards fixtures. And basically error if
> inheritance is used in tests. The functions on base classes is what is
> causing all the pain of rewind, because it's not a contract, but it's
> also magically there, so people try to use it.
> This would mean that every test would have more setUp with a bunch of
> explicit fixtures, but that's fine, it's a little more time thinking
> through the writing of the tests, and a lot easier to read them to know
> what is going on, and to delete them later. Basically get rid of all
> class/superclass magic.
> > The effort was called scenario refactoring and I think that's what we
> > should do. We should not do "start from scratch scenarios" or
> "copy all
> > the code and see what happens".
> > There's no problem with plugins. We committed to have a stable
> > which is documented and agreed upon. It's clearly written
> > here
> > The rest is private, for Tempest internal use. If Tempest cores
> > with that, then we should first of all put a spec and rewrite that
> > document.
> It does seem like there is a tough love moment coming about the fact
> that APIs were consumed that were specifically out of bounds. Either the
> team commits to them inbounds, and there is the contract, or moves
> forward under the existing contract.
> Though, I expect the issue is that because all the Tempest tests work in
> this super class way, this is the only stuff people understood and copy
> pasted. So it does feel like there has to be at least a couple of
> instances in the Tempest tree of how the team *wants* other projects to
> write tests that they can point people to.
> +1 on this. Which is why we need to refactor our scenario test to a sane
> base that people can use as examples to copy paste.
> The basic credentials provider services that comes with test.py is something
> that is useful to have I think - and we are working on make them a
> contract people
> can rely on.
> We had a discussion at the PTG about how to deal with those, and the
> was that we are going to move credentials providers to tempest.lib and keep
> test.py more or less as it is and declare it a stable interface for plugins.
> Alternatives like dropping class fixtures for good, and move everything
> to be test
> fixtures, involve too much code churn / refactor for an advantage which
> is too little - and we wouldn't have the bandwidth to do that anyways.
So, as someone who is only an infrequence Tempest contributor now, I
personally have a pretty hard time with the magic credentials, because
it's super non obvious to me what a new test would give me off a blank
page, or how I would expose that.
More or less I had to just find Matt and ask him what to do, and then
have Jordan propose a different fix once he saw the patch. That's really
people intensive, and I think at least as big a scalability problem as
That being said, there are only 46 scenario tests. They totally could be
rewritten in an ideal way as examples and deprecate / rot out the
existing scenario tests (including base classes).
More information about the OpenStack-dev