Open Stack

On Thu, May 4, 2017 at 3:56 PM, Matthew Treinish <mtreinish at kortar.org>
wrote:

> On Wed, May 03, 2017 at 11:51:13AM +0000, Andrea Frittoli wrote:
> > On Tue, May 2, 2017 at 5:33 PM Matthew Treinish <mtreinish at kortar.org>
> > wrote:
> >
> > > On Tue, May 02, 2017 at 09:49:14AM +0530, Rabi Mishra wrote:
> > > > On Fri, Apr 28, 2017 at 2:17 PM, Andrea Frittoli <
> > > andrea.frittoli at gmail.com>
> > > > wrote:
> > > >
> > > > >
> > > > >
> > > > > On Fri, Apr 28, 2017 at 10:29 AM Rabi Mishra <ramishra at redhat.com>
> > > wrote:
> > > > >
> > > > >> On Thu, Apr 27, 2017 at 3:55 PM, Andrea Frittoli <
> > > > >> andrea.frittoli at gmail.com> wrote:
> > > > >>
> > > > >>> Dear stackers,
> > > > >>>
> > > > >>> starting in the Liberty cycle Tempest has defined a set of
> projects
> > > > >>> which are in scope for direct
> > > > >>> testing in Tempest [0]. The current list includes keystone, nova,
> > > > >>> glance, swift, cinder and neutron.
> > > > >>> All other projects can use the same Tempest testing
> infrastructure
> > > (or
> > > > >>> parts of it) by taking advantage
> > > > >>> the Tempest plugin and stable interfaces.
> > > > >>>
> > > > >>> Tempest currently hosts a set of API tests as well as a service
> > > client
> > > > >>> for the Heat project.
> > > > >>> The Heat service client is used by the tests in Tempest, which
> run in
> > > > >>> Heat gate as part of the grenade
> > > > >>> job, as well as in the Tempest gate (check pipeline) as part of
> the
> > > > >>> layer4 job.
> > > > >>> According to code search [3] the Heat service client is also
> used by
> > > > >>> Murano and Daisycore.
> > > > >>>
> > > > >>
> > > > >> For the heat grenade job, I've proposed two patches.
> > > > >>
> > > > >> 1. To run heat tree gabbi api tests as part of grenade
> 'post-upgrade'
> > > > >> phase
> > > > >>
> > > > >> https://review.openstack.org/#/c/460542/
> > > > >>
> > > > >> 2. To remove tempest tests from the grenade job
> > > > >>
> > > > >> https://review.openstack.org/#/c/460810/
> > > > >>
> > > > >>
> > > > >>
> > > > >>> I proposed a patch to Tempest to start the deprecation counter
> for
> > > Heat
> > > > >>> / orchestration related
> > > > >>> configuration items in Tempest [4], and I would like to make sure
> > > that
> > > > >>> all tests and the service client
> > > > >>> either find a new home outside of Tempest, or are removed, by
> the end
> > > > >>> the Pike cycle at the latest.
> > > > >>>
> > > > >>> Heat has in-tree integration tests and Gabbi based API tests,
> but I
> > > > >>> don't know if those provide
> > > > >>> enough coverage to replace the tests on Tempest side.
> > > > >>>
> > > > >>>
> > > > >> Yes, the heat gabbi api tests do not yet have the same coverage
> as the
> > > > >> tempest tree api tests (lacks tests using nova, neutron and swift
> > > > >> resources),  but I think that should not stop us from *not*
> running
> > > the
> > > > >> tempest tests in the grenade job.
> > > > >>
> > > > >> I also don't know if the tempest tree heat tests are used by any
> other
> > > > >> upstream/downstream jobs. We could surely add more tests to bridge
> > > the gap.
> > > > >>
> > > > >> Also, It's possible to run the heat integration tests (we've
> enough
> > > > >> coverage there) with tempest plugin after doing some initial
> setup,
> > > as we
> > > > >> do in all our dsvm gate jobs.
> > > > >>
> > > > >> It would propose to move tests and client to a Tempest plugin
> owned /
> > > > >>> maintained by
> > > > >>> the Heat team, so that the Heat team can have full flexibility in
> > > > >>> consolidating their integration
> > > > >>> tests. For Murano and Daisycloud - and any other team that may
> want
> > > to
> > > > >>> use the Heat service
> > > > >>> client in their tests, even if the client is removed from
> Tempest, it
> > > > >>> would still be available via
> > > > >>> the Heat Tempest plugin. As long as the plugin implements the
> service
> > > > >>> client interface,
> > > > >>> the Heat service client will register automatically in the
> service
> > > > >>> client manager and be available
> > > > >>> for use as today.
> > > > >>>
> > > > >>>
> > > > >> if I understand correctly, you're proposing moving the existing
> > > tempest
> > > > >> tests and service clients to a separate repo managed by heat team.
> > > Though
> > > > >> that would be collective decision, I'm not sure that's something I
> > > would
> > > > >> like to do. To start with we may look at adding some of the
> missing
> > > pieces
> > > > >> in heat tree itself.
> > > > >>
> > > > >
> > > > > I'm proposing to move tests and the service client outside of
> tempest
> > > to a
> > > > > new home.
> > > > >
> > > > > I also suggested that the new home could be a dedicate repo, since
> that
> > > > > would allow you to maintain the
> > > > > current branchless nature of those tests. A more detailed
> discussion
> > > about
> > > > > the topic can be found
> > > > > in the corresponding proposed queens goal [5],
> > > > >
> > > > > Using a dedicated repo *is not* a precondition for moving tests and
> > > > > service client out of Tempest.
> > > > >
> > > > >
> > > > We probably are mixing two different things here.
> > > >
> > > > 1. Moving in-tree heat templest plugn and tests to a dedicated repo
> > > >
> > > > Though we don't have any plans for it now, we may have to do it
> when/if
> > > > it's accepted as a community goal.
> > > >
> > > > 2.  Moving tempest tree heat tests and heat service client to a new
> home
> > > > and owner.
> > > >
> > > > I don't think that's something heat team would like to do given that
> we
> > > > don't use these tests anywhere and would probably spend time
> improving
> > > the
> > > > coverage of the gabbi api tests we already have.
> > > >
> > >
> >
> > The test coverage provided by those tests is not available anywhere else
> at
> > the moment, however the tests are not really used, so I'm confused about
> > this.
> >
> > You could run the existing Tempest tests with very little extra effort
> > until the
> > corresponding coverage is available in Gabbi format.
> >
> >
> > >
> > > Ok, well if the heat team has no interest in maintaining these tests
> there
> > > is
> > > no point in keeping them around anymore. I've pushed up:
> > >
> > > https://review.openstack.org/461841
> >
> >
> > Thanks for this.
> > We will need Heat PTL / QA Liaison +1 on the test removal before it goes
> > through.
> >
> >
> > >
> > > to remove the tests. As for the clients we can just move those to
> > > tempest.lib
> > > to not break the plugins that are using them.
> > >
> >
> > We could, but other projects maintain their own service clients, I'm not
> > convinced we should
> > make a special case for Heat.
> >
>
> I'm not viewing it as a special case for heat, more treating it like any
> other
> tempest interface that has a number of external consumers already but
> isn't in
> lib yet. I don't think it's at all unreasonable to maintain the 321 lines
> of
> heat client code in tempest.lib so that the plugins identified on this this
> thread (and any others out there) don't lose an interface they rely on. The
> maintenance burden for the clients is not high at all, especially since the
> Rest API is supposed to be stable interface nothing should really need to
> change in the client code often. We won't be adding new features just
> preserving the existing interfaces. We just will need to document that
> they're
> not self tested anymore because the tests no longer exist.
>
> Also, like it or not it this already is a special case. This is the first
> and
> only time a project team didn't take ownership of their project's tempest
> tests
> since we made the decision to migrate higher level services into plugins
> back
> at the Liberty summit. (it's also the last project left as part of that
> migration)
>

The heat tree has 215 tests exposed as a tempest plugin, including one
which runs a gabbi suite. So I *really* struggle to respond to the
suggestion that we "didn't take ownership" without feelings of
defensiveness and anger.

What we didn't do is use the tempest client, so moving existing tests over
isn't a simple copy. Also it took a while to get the gabbi stuff in place
to be able to rewrite the api coverage that the heat tempest tests have.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20170505/a54b5591/attachment-0001.html>