[openstack-dev] [qa] [keystone] Random Patrole failures related to Identity v3 Extensions API

Morgan Fainberg morgan.fainberg at gmail.com
Fri Aug 11 16:25:43 UTC 2017

On Fri, Aug 11, 2017 at 8:44 AM, Felipe Monteiro
<felipe.carneiro.monteiro at gmail.com> wrote:
> Patrole tests occasionally fail while executing tests that test the
> Identity v3 Extensions API [0]. Previously, this was not the case when
> we used Fernet tokens and used a time.sleep(1) to allow for
> role-switching to work correctly. However, we recently changed over to
> UUID tokens in the Patrole gates to avoid doing a time.sleep(1), as a
> time efficiency change. Ordinarily -- for well over 500 or so tests --
> this approach works successfully, with the exception of what appears
> to be *random* v3 API extension tests [1][2] (random means different
> tests pass or fail randomly across separate test runs).
> While there are a few solutions that come to mind on how to solve this
> Patrole-side (like re-introducing a time.sleep() for specific APIs or
> even avoiding role-switching altogether which is not as
> straightforward as it sounds), we would still not understand *why* the
> issue is happening in the first place. Is it a data-race condition?
> Something specific to the identity v3 extensions API? A potential bug
> or intended behavior somewhere?
> [0] https://developer.openstack.org/api-ref/identity/v3-ext/
> [1] http://status.openstack.org/openstack-health/#/test/patrole_tempest_plugin.tests.api.identity.v3.test_ep_filter_groups_rbac.EndpointFilterGroupsV3RbacTest.test_create_endpoint_group
> [2] http://logs.openstack.org/41/490641/3/gate/gate-tempest-dsvm-patrole-py35-member-ubuntu-xenial/be95da4/console.html#_2017-08-11_14_47_57_515906
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Are your tests causing token revocations? if so, there is a case where
a revocation event is issued in the same second as a token (we've seen
similar cases even in fernet) meaning the token is invalid when it is
issued according to keystone. It's a long running bug.

For the record, UUID tokens are deprecated and slated for removal in
the R release. I recommend reverting to using Fernet tokens sooner
rather than later.

Last of all, the endpoint-filtering is generally not a great tool to
use. I highly recommend not using it (or encouraging the use of it),
it makes the catalog different depending on scope and provides zero
added security benefit (anyone who knows the endpoint can use it

More information about the OpenStack-dev mailing list