[openstack-dev] [qa] [keystone] Random Patrole failures related to Identity v3 Extensions API

Morgan Fainberg morgan.fainberg at gmail.com
Fri Aug 11 16:26:49 UTC 2017

On Fri, Aug 11, 2017 at 9:25 AM, Morgan Fainberg
<morgan.fainberg at gmail.com> wrote:
> On Fri, Aug 11, 2017 at 8:44 AM, Felipe Monteiro
> <felipe.carneiro.monteiro at gmail.com> wrote:
>> Patrole tests occasionally fail while executing tests that test the
>> Identity v3 Extensions API [0]. Previously, this was not the case when
>> we used Fernet tokens and used a time.sleep(1) to allow for
>> role-switching to work correctly. However, we recently changed over to
>> UUID tokens in the Patrole gates to avoid doing a time.sleep(1), as a
>> time efficiency change. Ordinarily -- for well over 500 or so tests --
>> this approach works successfully, with the exception of what appears
>> to be *random* v3 API extension tests [1][2] (random means different
>> tests pass or fail randomly across separate test runs).
>> While there are a few solutions that come to mind on how to solve this
>> Patrole-side (like re-introducing a time.sleep() for specific APIs or
>> even avoiding role-switching altogether which is not as
>> straightforward as it sounds), we would still not understand *why* the
>> issue is happening in the first place. Is it a data-race condition?
>> Something specific to the identity v3 extensions API? A potential bug
>> or intended behavior somewhere?
>> [0] https://developer.openstack.org/api-ref/identity/v3-ext/
>> [1] http://status.openstack.org/openstack-health/#/test/patrole_tempest_plugin.tests.api.identity.v3.test_ep_filter_groups_rbac.EndpointFilterGroupsV3RbacTest.test_create_endpoint_group
>> [2] http://logs.openstack.org/41/490641/3/gate/gate-tempest-dsvm-patrole-py35-member-ubuntu-xenial/be95da4/console.html#_2017-08-11_14_47_57_515906
>> __________________________________________________________________________
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> Are your tests causing token revocations? if so, there is a case where
> a revocation event is issued in the same second as a token (we've seen
> similar cases even in fernet) meaning the token is invalid when it is
> issued according to keystone. It's a long running bug.
> For the record, UUID tokens are deprecated and slated for removal in
> the R release. I recommend reverting to using Fernet tokens sooner
> rather than later.
> Last of all, the endpoint-filtering is generally not a great tool to
> use. I highly recommend not using it (or encouraging the use of it),
> it makes the catalog different depending on scope and provides zero
> added security benefit (anyone who knows the endpoint can use it
> anyway).

I am spinning up a change for Pike RC (right now) to address the
revocations occurring in the same second as the issuance of the token
(similar to a bug with password updates, which will also be fixed).

More information about the OpenStack-dev mailing list