Open Stack

Thanks Ghanshyam,

That was really helpful. If Tempest is versioned via tags, then I agree that Patrole should do the same thing. I didn't quite know what you meant by "feature flag" but after finding [0] it became apparent to me: Patrole has been doing this wherever Tempest does it. Granted, we probably need more participation in the project to catch minute details like that, as we currently have a large body of tests, but currently we have been using feature flags where appropriate.

Case 3 as you mentioned is something that we've already had to account for: When Nova and then Keystone moved their policies into code, we had to add logic to the framework to account for those changes. Case 3 in theory would not be a problem, if there was a way to verify whether a policy action was enforced, other than by checking logs. If there was an API in OpenStack that could be called to retrieve oslo_policy enforcements per request id, then Case 3 would just result in an error being thrown by Patrole for a given test, which we would then notice and fix.

[0] https://review.openstack.org/#/c/418928/ 

--Felipe

-----Original Message-----
From: Ghanshyam Mann [mailto:ghanshyammann at gmail.com] 
Sent: Wednesday, April 19, 2017 12:05 AM
To: OpenStack Development Mailing List (not for usage questions) <openstack-dev at lists.openstack.org>
Subject: Re: [openstack-dev] [qa] [patrole] Question regarding patrole release management

On Wed, Apr 19, 2017 at 2:29 AM, MONTEIRO, FELIPE C <fm577c at att.com> wrote:
> Hi all,
>
>
>
> I have a question regarding patrole and release management. Many projects
> like heat or murano have a tempest plugin within their repos, so by
> extension their tempest plugins have releases, as the projects change over
> time. However, since patrole is just a tempest plugin, yet heavily reliant
> on tempest, how should patrole do release management?

Hi Felipe,

Release management depends on whether Patrole is planning the
branchless model (i think it is) like Tempest or branch model., If
branchless, then it does not fall under release management. It can
adopt release model what Tempest has [1].

Branchless model gives many benefit like avoid backward incompatible
changes, avoid maintaining multiple Patrole repo across releases etc.

> Am I correct in
> thinking that it should, in the first place? With nova-network and other
> APIs slated for deprecation in Pike and beyond, Patrole will logically have
> to continuously be maintained to keep up, meaning that older tests, just
> like with Tempest, will have to be phased out. If Patrole, then, does not
> have releases, then older release-dependent tests and functionality will
> over time be lost.

We have 3 cases here:
1. Functionality is deprecated/removed in new release.
2. Functionality newly added in new release.
3. Policy enforcement change.

For case 1, Tempest keep testing deprecated functionality till it is
marked deprecated across all supported stable branches. Once all
stable branch has that functionality as deprecated marked, then we
discuss of removing its testing from Tempest.
With API Microversion model that is little bit different where
functionality might be deprecated after specific version or it is
deprecated from base version itself. For example, Nova proxy APIs
deprecated after 2.36, Certificate APIs might be gone from base
version (which is under discussion). This is more case by case and
based on all stack holder point of view, we will decide their testing
should be removed or stay till when.

For case 2, Tempest introduced testing of new functionality with
feature flag and those tests will be executed/skipped accordingly.

Case 3 is something important to consider for Patrole. Usually policy
changes will be done with backward compatible way where changes does
not break upgrade. Any change in policy enforcement will be done at
least with supporting old and new rules [2] or with old rules
deprecation of period of 1 release cycle at least. And branchless
model can detect any accidental changes which going to break upgrade.

IMO, branchless model is good value for Patrole in all 3 cases
consideration and feature flags to handle the new/old/policy-change
functionality. Similarly release model can be same as Tempest has.


>
>
>
> Thank You,
>
> Felipe Monteiro

..1 https://urldefense.proofpoint.com/v2/url?u=https-3A__wiki.openstack.org_wiki_QA_releases&d=DwIGaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=X4GwEru-SJ9DRnCxhze-aw&m=xgobDX0RcKxlQhqhOSmU6H11-k8u21HsinBnbYLB6mY&s=9eq8-o16PTMYfNYW0LjlVyRfN1W4wXWloI7jVij5W4g&e= 
..2 https://urldefense.proofpoint.com/v2/url?u=https-3A__review.openstack.org_-23_c_391113_13&d=DwIGaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=X4GwEru-SJ9DRnCxhze-aw&m=xgobDX0RcKxlQhqhOSmU6H11-k8u21HsinBnbYLB6mY&s=A5axQVIlZlVMGaW0vwQxpqwx5uMZvZ1ZNwWAXgrIoAc&e= 

-gmann


>
>
>
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.openstack.org_cgi-2Dbin_mailman_listinfo_openstack-2Ddev&d=DwIGaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=X4GwEru-SJ9DRnCxhze-aw&m=xgobDX0RcKxlQhqhOSmU6H11-k8u21HsinBnbYLB6mY&s=DKOquEt2EsEYictxTecUKp5fwVaWZ3kNoFdm_tYFSsc&e= 
>

__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.openstack.org_cgi-2Dbin_mailman_listinfo_openstack-2Ddev&d=DwIGaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=X4GwEru-SJ9DRnCxhze-aw&m=xgobDX0RcKxlQhqhOSmU6H11-k8u21HsinBnbYLB6mY&s=DKOquEt2EsEYictxTecUKp5fwVaWZ3kNoFdm_tYFSsc&e=