Open Stack

On Wed, May 03, 2017 at 11:56:56PM -0400, Matthew Treinish wrote:
> On Wed, May 03, 2017 at 11:51:13AM +0000, Andrea Frittoli wrote:
> > On Tue, May 2, 2017 at 5:33 PM Matthew Treinish <mtreinish at kortar.org>
> > wrote:
> > 
> > > On Tue, May 02, 2017 at 09:49:14AM +0530, Rabi Mishra wrote:
> > > > On Fri, Apr 28, 2017 at 2:17 PM, Andrea Frittoli <
> > > andrea.frittoli at gmail.com>
> > > > wrote:
> > > >
> > > > >
> > > > >
> > > > > On Fri, Apr 28, 2017 at 10:29 AM Rabi Mishra <ramishra at redhat.com>
> > > wrote:
> > > > >
> > > > >> On Thu, Apr 27, 2017 at 3:55 PM, Andrea Frittoli <
> > > > >> andrea.frittoli at gmail.com> wrote:
> > > > >>
> > > > >>> Dear stackers,
> > > > >>>
> > > > >>> starting in the Liberty cycle Tempest has defined a set of projects
> > > > >>> which are in scope for direct
> > > > >>> testing in Tempest [0]. The current list includes keystone, nova,
> > > > >>> glance, swift, cinder and neutron.
> > > > >>> All other projects can use the same Tempest testing infrastructure
> > > (or
> > > > >>> parts of it) by taking advantage
> > > > >>> the Tempest plugin and stable interfaces.
> > > > >>>
> > > > >>> Tempest currently hosts a set of API tests as well as a service
> > > client
> > > > >>> for the Heat project.
> > > > >>> The Heat service client is used by the tests in Tempest, which run in
> > > > >>> Heat gate as part of the grenade
> > > > >>> job, as well as in the Tempest gate (check pipeline) as part of the
> > > > >>> layer4 job.
> > > > >>> According to code search [3] the Heat service client is also used by
> > > > >>> Murano and Daisycore.
> > > > >>>
> > > > >>
> > > > >> For the heat grenade job, I've proposed two patches.
> > > > >>
> > > > >> 1. To run heat tree gabbi api tests as part of grenade 'post-upgrade'
> > > > >> phase
> > > > >>
> > > > >> https://review.openstack.org/#/c/460542/
> > > > >>
> > > > >> 2. To remove tempest tests from the grenade job
> > > > >>
> > > > >> https://review.openstack.org/#/c/460810/
> > > > >>
> > > > >>
> > > > >>
> > > > >>> I proposed a patch to Tempest to start the deprecation counter for
> > > Heat
> > > > >>> / orchestration related
> > > > >>> configuration items in Tempest [4], and I would like to make sure
> > > that
> > > > >>> all tests and the service client
> > > > >>> either find a new home outside of Tempest, or are removed, by the end
> > > > >>> the Pike cycle at the latest.
> > > > >>>
> > > > >>> Heat has in-tree integration tests and Gabbi based API tests, but I
> > > > >>> don't know if those provide
> > > > >>> enough coverage to replace the tests on Tempest side.
> > > > >>>
> > > > >>>
> > > > >> Yes, the heat gabbi api tests do not yet have the same coverage as the
> > > > >> tempest tree api tests (lacks tests using nova, neutron and swift
> > > > >> resources),  but I think that should not stop us from *not* running
> > > the
> > > > >> tempest tests in the grenade job.
> > > > >>
> > > > >> I also don't know if the tempest tree heat tests are used by any other
> > > > >> upstream/downstream jobs. We could surely add more tests to bridge
> > > the gap.
> > > > >>
> > > > >> Also, It's possible to run the heat integration tests (we've enough
> > > > >> coverage there) with tempest plugin after doing some initial setup,
> > > as we
> > > > >> do in all our dsvm gate jobs.
> > > > >>
> > > > >> It would propose to move tests and client to a Tempest plugin owned /
> > > > >>> maintained by
> > > > >>> the Heat team, so that the Heat team can have full flexibility in
> > > > >>> consolidating their integration
> > > > >>> tests. For Murano and Daisycloud - and any other team that may want
> > > to
> > > > >>> use the Heat service
> > > > >>> client in their tests, even if the client is removed from Tempest, it
> > > > >>> would still be available via
> > > > >>> the Heat Tempest plugin. As long as the plugin implements the service
> > > > >>> client interface,
> > > > >>> the Heat service client will register automatically in the service
> > > > >>> client manager and be available
> > > > >>> for use as today.
> > > > >>>
> > > > >>>
> > > > >> if I understand correctly, you're proposing moving the existing
> > > tempest
> > > > >> tests and service clients to a separate repo managed by heat team.
> > > Though
> > > > >> that would be collective decision, I'm not sure that's something I
> > > would
> > > > >> like to do. To start with we may look at adding some of the missing
> > > pieces
> > > > >> in heat tree itself.
> > > > >>
> > > > >
> > > > > I'm proposing to move tests and the service client outside of tempest
> > > to a
> > > > > new home.
> > > > >
> > > > > I also suggested that the new home could be a dedicate repo, since that
> > > > > would allow you to maintain the
> > > > > current branchless nature of those tests. A more detailed discussion
> > > about
> > > > > the topic can be found
> > > > > in the corresponding proposed queens goal [5],
> > > > >
> > > > > Using a dedicated repo *is not* a precondition for moving tests and
> > > > > service client out of Tempest.
> > > > >
> > > > >
> > > > We probably are mixing two different things here.
> > > >
> > > > 1. Moving in-tree heat templest plugn and tests to a dedicated repo
> > > >
> > > > Though we don't have any plans for it now, we may have to do it when/if
> > > > it's accepted as a community goal.
> > > >
> > > > 2.  Moving tempest tree heat tests and heat service client to a new home
> > > > and owner.
> > > >
> > > > I don't think that's something heat team would like to do given that we
> > > > don't use these tests anywhere and would probably spend time improving
> > > the
> > > > coverage of the gabbi api tests we already have.
> > > >
> > >
> > 
> > The test coverage provided by those tests is not available anywhere else at
> > the moment, however the tests are not really used, so I'm confused about
> > this.
> > 
> > You could run the existing Tempest tests with very little extra effort
> > until the
> > corresponding coverage is available in Gabbi format.
> > 
> > 
> > >
> > > Ok, well if the heat team has no interest in maintaining these tests there
> > > is
> > > no point in keeping them around anymore. I've pushed up:
> > >
> > > https://review.openstack.org/461841
> > 
> > 
> > Thanks for this.
> > We will need Heat PTL / QA Liaison +1 on the test removal before it goes
> > through.
> > 
> > 
> > >
> > > to remove the tests. As for the clients we can just move those to
> > > tempest.lib
> > > to not break the plugins that are using them.
> > >
> > 
> > We could, but other projects maintain their own service clients, I'm not
> > convinced we should
> > make a special case for Heat.
> > 
> 
> I'm not viewing it as a special case for heat, more treating it like any other
> tempest interface that has a number of external consumers already but isn't in
> lib yet. I don't think it's at all unreasonable to maintain the 321 lines of
> heat client code in tempest.lib so that the plugins identified on this this
> thread (and any others out there) don't lose an interface they rely on. The
> maintenance burden for the clients is not high at all, especially since the
> Rest API is supposed to be stable interface nothing should really need to
> change in the client code often. We won't be adding new features just
> preserving the existing interfaces. We just will need to document that they're
> not self tested anymore because the tests no longer exist.
> 
> Also, like it or not it this already is a special case. This is the first and
> only time a project team didn't take ownership of their project's tempest tests
> since we made the decision to migrate higher level services into plugins back
> at the Liberty summit. (it's also the last project left as part of that
> migration)

FWIW I think it's unfair to say we didn't take ownership of tempest tests
and I would have phrased the current status differently, basically the
stuff that landed pre-dates the decision to migrate new services to a
plugin based model, and that is also why our in-tree integration tests
weren't based on that tempest framework.

Thanks!

Steve