Open Stack

Hi Pavlo,

thank you for your suggestion.
See my comments inline.

andrea
On Thu, Mar 2, 2017 at 9:19 AM Pavlo Shchelokovskyy <
pshchelokovskyy at mirantis.com> wrote:

> Hi Andrea,
>
> I am not sure why the new tempest.scenario.manager has to be developed
> that way. May I humbly suggest another path? Roughly goes like:
>
> 1) Rename the present class to "OldScenarioTest", set the original name to
> point to it (ScenarioTest=OldScenarioTest) (in a single commit)
>

We considered a similar option [0], but I fear that maintaining the old
class in Tempest, and even gating on it, would just postpone the issue for
plugins, and would probably have the negative effect of making even more
plugin rely on that.


> 2) Create a new class NewScenarioTest (initially a copy of the old one?),
> development/refactoring/rewrite happens there, do not merge any new
> features in OldScenarioTest
>
> 3) add a tempest config option "USE_NEW_SCENARIO_MANAGER", False by default
>
> 4) conditionally on the value of USE_NEW_SCENARIO_MANAGER set the
> ScenarioTest to point to OldScenarioTest or NewScenarioTest
>
> 5) add a gate job(s) to tempest that forces USE_NEW_SCENARIO_MANAGER to
> True to test the new code, non-voting from the start, gate on it later
>

Having a gate job like this would make it really hard for us to get rid of
the it in future.


> This way the projects using tempest plugins and other tempest consumers
> will not be affected at all. Later each project on its jobs using tempest
> can try to enforce USE_NEW_SCENARIO_MANAGER to True if wishing to test the
> new one or switch to it when it is ready. Eventually when new one is
> stable, the config option might be removed and new class renamed to the
> name of the original class.
>
> The only downside of such approach I see might be a small? explosion of
> number of jobs in the tempest project (running on both new and old
> manager), but I'd think at least in the beginning of this refactoring
> effort the tempest team would like to have the number of jobs testing the
> new manager limited any way.
>
> I might be completely wrong with this advice, as I presume tempest team
> have put a lot of thinking into this problem already. But still, could you
> consider such approach?
>
>
Another option would be to:
- create a new base class in Tempest
- incrementally move scenario tests across to the new base class, along
with the helpers they need
However this would have the drawback of risking to break plugins again and
again, and to slow down the overall process.


[0] https://review.openstack.org/#/c/439291/1


> Cheers,
>
> Dr. Pavlo Shchelokovskyy
> Senior Software Engineer
> Mirantis Inc
> www.mirantis.com
>
> On Mon, Feb 27, 2017 at 1:34 PM, Andrea Frittoli <
> andrea.frittoli at gmail.com> wrote:
>
> Hello folks,
>
> TL;DR: if today you import manager,py from tempest.scenario please
> maintain a copy of [0] in tree until further notice.
>
> Full message:
> ------------------
>
> One of the priorities for the QA team in the Pike cycle is to refactor
> scenario tests to a sane code base [1].
>
> As they are now, changes to scenario tests are difficult to develop and
> review, and failures in those tests are hard to debug, which is in many
> directions far away from where we need to be.
>
> The issue we face is that, even though tempest.scenario.manager is not
> advertised as a stable interface in Tempest, many project use it today for
> convenience in writing their own tests. We don't know about dependencies
> outside of the OpenStack ecosystem, but we want to try to make this
> refactor a smooth experience for our uses in OpenStack, and avoid painful
> gate breakages as much as possible.
>
> The process we're proposing is as follows:
> - hold a copy of [0] in tree - in most cases you won't even have to change
> your imports as a lot of projects use tempest/scenario in their code base.
> You may decide to include the bare minimum you need from that module
> instead of all of it. It's a bit more work to make the patch, but less
> un-used code lying around afterwards.
> - the QA team will refactor scenario tests, and make more interfaces
> stable (test.py, credential providers). We won't advertise every single
> change in this process, only when we start and once we're done.
> - you may decide to discard your local copy of manager.py and consume
> Tempest stable interfaces directly. We will help with any question you may
> have on the process and on Tempest interfaces.
>
> Repositories affected by the refactor are (based on [2]):
>
>
> blazar,ceilometer,congress,intel-nfv-ci-tests,ironic,manila,networking-bgpvpn,networking-fortinet,networking-sfc,neutron-fwaas,neutron-lbaas,nova-lxd,octavia,sahara-tests,tap-as-a-service,tempest-horizon,vmware-nsx,watcher
>
> If we don't hear from a team at all in the next two weeks, we will assume
> that the corresponding Tempest plugin / bunch of tests is not in use
> anymore, and ignore it. If you use tempest.scenario.manager.py today and
> your repo is not on the list, please let us know!
>
> I'm happy to propose an initial patch for any team that may require it -
> just ping me on IRC (andreaf).
> I won't have the bandwidth myself to babysit each patch through review and
> gate though.
>
> Thank you for your cooperation and patience!
>
> Andrea
>
> [0]
> http://git.openstack.org/cgit/openstack/tempest/tree/tempest/scenario/manager.py
>
> [1] https://etherpad.openstack.org/p/pike-qa-priorities
> [2]
> https://github.com/andreafrittoli/tempest_stable_interfaces/blob/master/data/get_deps.sh
>
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20170307/872f87cd/attachment.html>