Open Stack

On Fri, May 05, 2017 at 09:29:40AM +1200, Steve Baker wrote:
> On Thu, May 4, 2017 at 3:56 PM, Matthew Treinish <mtreinish at kortar.org>
> wrote:
> 
> > On Wed, May 03, 2017 at 11:51:13AM +0000, Andrea Frittoli wrote:
> > > On Tue, May 2, 2017 at 5:33 PM Matthew Treinish <mtreinish at kortar.org>
> > > wrote:
> > >
> > > > On Tue, May 02, 2017 at 09:49:14AM +0530, Rabi Mishra wrote:
> > > > > On Fri, Apr 28, 2017 at 2:17 PM, Andrea Frittoli <
> > > > andrea.frittoli at gmail.com>
> > > > > wrote:
> > > > >
> > > > > >
> > > > > >
> > > > > > On Fri, Apr 28, 2017 at 10:29 AM Rabi Mishra <ramishra at redhat.com>
> > > > wrote:
> > > > > >
> > > > > >> On Thu, Apr 27, 2017 at 3:55 PM, Andrea Frittoli <
> > > > > >> andrea.frittoli at gmail.com> wrote:
> > > > > >>
> > > > > >>> Dear stackers,
> > > > > >>>
> > > > > >>> starting in the Liberty cycle Tempest has defined a set of
> > projects
> > > > > >>> which are in scope for direct
> > > > > >>> testing in Tempest [0]. The current list includes keystone, nova,
> > > > > >>> glance, swift, cinder and neutron.
> > > > > >>> All other projects can use the same Tempest testing
> > infrastructure
> > > > (or
> > > > > >>> parts of it) by taking advantage
> > > > > >>> the Tempest plugin and stable interfaces.
> > > > > >>>
> > > > > >>> Tempest currently hosts a set of API tests as well as a service
> > > > client
> > > > > >>> for the Heat project.
> > > > > >>> The Heat service client is used by the tests in Tempest, which
> > run in
> > > > > >>> Heat gate as part of the grenade
> > > > > >>> job, as well as in the Tempest gate (check pipeline) as part of
> > the
> > > > > >>> layer4 job.
> > > > > >>> According to code search [3] the Heat service client is also
> > used by
> > > > > >>> Murano and Daisycore.
> > > > > >>>
> > > > > >>
> > > > > >> For the heat grenade job, I've proposed two patches.
> > > > > >>
> > > > > >> 1. To run heat tree gabbi api tests as part of grenade
> > 'post-upgrade'
> > > > > >> phase
> > > > > >>
> > > > > >> https://review.openstack.org/#/c/460542/
> > > > > >>
> > > > > >> 2. To remove tempest tests from the grenade job
> > > > > >>
> > > > > >> https://review.openstack.org/#/c/460810/
> > > > > >>
> > > > > >>
> > > > > >>
> > > > > >>> I proposed a patch to Tempest to start the deprecation counter
> > for
> > > > Heat
> > > > > >>> / orchestration related
> > > > > >>> configuration items in Tempest [4], and I would like to make sure
> > > > that
> > > > > >>> all tests and the service client
> > > > > >>> either find a new home outside of Tempest, or are removed, by
> > the end
> > > > > >>> the Pike cycle at the latest.
> > > > > >>>
> > > > > >>> Heat has in-tree integration tests and Gabbi based API tests,
> > but I
> > > > > >>> don't know if those provide
> > > > > >>> enough coverage to replace the tests on Tempest side.
> > > > > >>>
> > > > > >>>
> > > > > >> Yes, the heat gabbi api tests do not yet have the same coverage
> > as the
> > > > > >> tempest tree api tests (lacks tests using nova, neutron and swift
> > > > > >> resources),  but I think that should not stop us from *not*
> > running
> > > > the
> > > > > >> tempest tests in the grenade job.
> > > > > >>
> > > > > >> I also don't know if the tempest tree heat tests are used by any
> > other
> > > > > >> upstream/downstream jobs. We could surely add more tests to bridge
> > > > the gap.
> > > > > >>
> > > > > >> Also, It's possible to run the heat integration tests (we've
> > enough
> > > > > >> coverage there) with tempest plugin after doing some initial
> > setup,
> > > > as we
> > > > > >> do in all our dsvm gate jobs.
> > > > > >>
> > > > > >> It would propose to move tests and client to a Tempest plugin
> > owned /
> > > > > >>> maintained by
> > > > > >>> the Heat team, so that the Heat team can have full flexibility in
> > > > > >>> consolidating their integration
> > > > > >>> tests. For Murano and Daisycloud - and any other team that may
> > want
> > > > to
> > > > > >>> use the Heat service
> > > > > >>> client in their tests, even if the client is removed from
> > Tempest, it
> > > > > >>> would still be available via
> > > > > >>> the Heat Tempest plugin. As long as the plugin implements the
> > service
> > > > > >>> client interface,
> > > > > >>> the Heat service client will register automatically in the
> > service
> > > > > >>> client manager and be available
> > > > > >>> for use as today.
> > > > > >>>
> > > > > >>>
> > > > > >> if I understand correctly, you're proposing moving the existing
> > > > tempest
> > > > > >> tests and service clients to a separate repo managed by heat team.
> > > > Though
> > > > > >> that would be collective decision, I'm not sure that's something I
> > > > would
> > > > > >> like to do. To start with we may look at adding some of the
> > missing
> > > > pieces
> > > > > >> in heat tree itself.
> > > > > >>
> > > > > >
> > > > > > I'm proposing to move tests and the service client outside of
> > tempest
> > > > to a
> > > > > > new home.
> > > > > >
> > > > > > I also suggested that the new home could be a dedicate repo, since
> > that
> > > > > > would allow you to maintain the
> > > > > > current branchless nature of those tests. A more detailed
> > discussion
> > > > about
> > > > > > the topic can be found
> > > > > > in the corresponding proposed queens goal [5],
> > > > > >
> > > > > > Using a dedicated repo *is not* a precondition for moving tests and
> > > > > > service client out of Tempest.
> > > > > >
> > > > > >
> > > > > We probably are mixing two different things here.
> > > > >
> > > > > 1. Moving in-tree heat templest plugn and tests to a dedicated repo
> > > > >
> > > > > Though we don't have any plans for it now, we may have to do it
> > when/if
> > > > > it's accepted as a community goal.
> > > > >
> > > > > 2.  Moving tempest tree heat tests and heat service client to a new
> > home
> > > > > and owner.
> > > > >
> > > > > I don't think that's something heat team would like to do given that
> > we
> > > > > don't use these tests anywhere and would probably spend time
> > improving
> > > > the
> > > > > coverage of the gabbi api tests we already have.
> > > > >
> > > >
> > >
> > > The test coverage provided by those tests is not available anywhere else
> > at
> > > the moment, however the tests are not really used, so I'm confused about
> > > this.
> > >
> > > You could run the existing Tempest tests with very little extra effort
> > > until the
> > > corresponding coverage is available in Gabbi format.
> > >
> > >
> > > >
> > > > Ok, well if the heat team has no interest in maintaining these tests
> > there
> > > > is
> > > > no point in keeping them around anymore. I've pushed up:
> > > >
> > > > https://review.openstack.org/461841
> > >
> > >
> > > Thanks for this.
> > > We will need Heat PTL / QA Liaison +1 on the test removal before it goes
> > > through.
> > >
> > >
> > > >
> > > > to remove the tests. As for the clients we can just move those to
> > > > tempest.lib
> > > > to not break the plugins that are using them.
> > > >
> > >
> > > We could, but other projects maintain their own service clients, I'm not
> > > convinced we should
> > > make a special case for Heat.
> > >
> >
> > I'm not viewing it as a special case for heat, more treating it like any
> > other
> > tempest interface that has a number of external consumers already but
> > isn't in
> > lib yet. I don't think it's at all unreasonable to maintain the 321 lines
> > of
> > heat client code in tempest.lib so that the plugins identified on this this
> > thread (and any others out there) don't lose an interface they rely on. The
> > maintenance burden for the clients is not high at all, especially since the
> > Rest API is supposed to be stable interface nothing should really need to
> > change in the client code often. We won't be adding new features just
> > preserving the existing interfaces. We just will need to document that
> > they're
> > not self tested anymore because the tests no longer exist.
> >
> > Also, like it or not it this already is a special case. This is the first
> > and
> > only time a project team didn't take ownership of their project's tempest
> > tests
> > since we made the decision to migrate higher level services into plugins
> > back
> > at the Liberty summit. (it's also the last project left as part of that
> > migration)
> >
> 
> The heat tree has 215 tests exposed as a tempest plugin, including one
> which runs a gabbi suite. So I *really* struggle to respond to the
> suggestion that we "didn't take ownership" without feelings of
> defensiveness and anger.
> 
> What we didn't do is use the tempest client, so moving existing tests over
> isn't a simple copy. Also it took a while to get the gabbi stuff in place
> to be able to rewrite the api coverage that the heat tempest tests have.

I didn't mean for it to come across as offensive, but the facts here are
we're deleting these tests after waiting patiently ~2 years for heat team to
migrate the tests out of tempest and there still really isn't equivalent
coverage setup. And TBH I don't actually care, it's up to you what you end up
using for your testing. Whether it's gabbi, tempest clients, etc or even
whether you choose to expose any tests as a tempest plugin that's not what I
was talking about. What I meant by take ownership is literal, which has been
the thing we've been asking for the past ~2 years, and to just migrate the heat
tests from the tempest tree and expose them into a plugin owned by heat. (to
do whatever you wanted with, which could have been deprecate and delete) Every
other project team for a higher level service did this. Heat is the only project
that decided not to do this, which is fine. But, that's why this situation is
unique. It's not a judgement on the heat team (although I do wish you had said
something earlier about not wanting the tests so we could have cleaned things
up in tempest a long time ago) I was just expressing this situation is unique
because it has implications for other tempest users that are leveraging the
heat clients. This wasn't a concern for any other projects because we just
documented that the imports had to be updated after the migration.

-Matt Treinish
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: not available
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20170504/68a2ab45/attachment.sig>