Open Stack

> -----Original Message-----
> From: Graham Hayes [mailto:gr at ham.ie]
> Sent: Wednesday, November 29, 2017 10:20 AM
> To: openstack-dev at lists.openstack.org
> Subject: Re: [openstack-dev] [qa] [patrole] Question regarding Patrole API
> coverage
> 
> 
> 
> On 29/11/17 15:11, Andrea Frittoli wrote:
> >
> >
> > On Wed, Nov 29, 2017 at 2:28 PM Peng Liu <pliu at redhat.com
> > <mailto:pliu at redhat.com>> wrote:
> >
> >     Hi Team,
> >
> > Hi Peng
> >
> >
> >     I have a question regarding to the API coverage of Patrole.
> >     Currently, Patrole as a Tempest plugin heavily relys on the Tempest
> >     code. However, Tempest only contains the API tests for the most
> >     common APIs of the most common projects(Nova, Neutron, Cinder,
> >     Glance, Swift, Keystone).
> >
> >     So I want to know if it is possible to extend Patrole to:
> >     1) test the APIs of the common projects which was not yet covered in
> >     Tempest.
> >
> >     2) test projects which was not covered in Tempest?
> >
> >
> > You can use the Patrole framework to test services not covered by
> > Tempest by taking advantage of Tempest plugin mechanism.
> > Patrole itself is a Tempest plugin. If you install the plugin of a
> > service that includes a service client, you should be able to use it to
> > write Patrole tests for that service.
> > I believe this has not been done yet by any project though, so there may
> > be a few technical bits to be sorted out.
> 
> There has been a start with Designate + Neutron [1], it is just underway
> now though.
> 
> It could cause an issue as all of a sudden the internals of a project's
> plugin may need to be a stable interface, which may not be something
> they expect.

Patrole doesn't change its internals overnight, so to speak. We also use deprecation notes and typically wait one release cycle before removing the feature, communicated again with a release note. That being said, Patrole's framework does change more often than Tempest's and will continue to do so for some time yet. So any reliance on Patrole will require maintenance. This is why Patrole uses 0.x.0 release versioning right now. 

> 1 - https://review.openstack.org/#/c/520237/
> 
> > I don't think Patrole itself will have to be extended, however Patrole
> > does not yet include stable APIs.
> > If you're going to use Patrole APIs in your project you need to be aware
> > that there may be backward incompatible changes happening without a
> > deprecation period.

The other thing to keep in mind is to ensure that the clients that are being used are Tempest-based. I've seen a few Tempest plugins using python-<project-name>-client integrations which is problematic, because Patrole expects Tempest-based exceptions to be raised.

> >
> > There are several options on where to host such tests: in a dedicated
> > plugin, in the Tempest plugin for the service or in Patrole itself.
> > The latter would probably suffer from the same scalability issues that
> > lead us to create the plugin mechanism to begin with.

There was discussion long ago about a `patrole.lib` that could be its own package, but anything like that will not materialize until Patrole is stable, pending discussion of course.

> >
> > Andrea Frittoli (andreaf)
> >
> >
> >
> >     Thanks,
> >     --
> >     Peng Liu
> >

Thanks,

Felipe