[openstack-dev] [requirements] race in keystone unit tests

Lance Bragstad lbragstad at gmail.com
Wed Aug 3 16:26:56 UTC 2016


Sending a follow-up because I think we ended up finding something relevant
to this discussion.

As keystone moves towards making fernet the default, one of our work items
was to mock the system clock in tests. This allows us to advance the clock
by one second where we need to avoid sub-second race conditions. To do this
we used freezegun [0]. We recently landed a bunch of fixes to do this.

It turns out that there is a possible race between when freezegun patches
it's modules and when the test runs. This turned up in a patch I was
working on locally and I noticed certain clock operations weren't using the
fake time object from freezegun. As a work-around, we can leverage the
set_time_override() method from oslo_utils.timeutils to make sure we are
using the fake time from within the frozen time context. In my testing
locally this worked.

If keystone requires a hybrid approach to patching
(oslo_utils.timeutils.set_time_override() + freezegun), we should build it
into a well documented hybrid context manager so that's its more apparent
why we need it.

Sean, I can start working on this to see if it starts mitigating the races
you're seeing.

[0] https://pypi.python.org/pypi/freezegun

On Tue, Aug 2, 2016 at 9:21 AM, Lance Bragstad <lbragstad at gmail.com> wrote:

> Hi Sean,
>
> Thanks for the information. This obviously looks Fernet-related and I
> would be happy to spend some cycles on it. We recently landed a bunch of
> refactors in keystone to improve Fernet test coverage. This could be
> related to those refactors. Just double checking - but you haven't opened a
> bug in launchpad for this yet have you?
>
> Thanks for the heads up!
>
> On Tue, Aug 2, 2016 at 5:32 AM, Sean Dague <sean at dague.net> wrote:
>
>> One of my concerns about stacking up project unit tests in the
>> requirements jobs, is the unit tests aren't as free of races as you
>> would imagine. Because they only previously impacted the one project
>> team, those teams are often just fast to recheck instead of get to the
>> bottom of it. Cross testing with them in a voting way changes their
>> impact.
>>
>> The keystone unit tests have a existing race condition in them, which
>> recently failed an unrelated requirements bump -
>>
>> http://logs.openstack.org/50/348250/6/check/gate-cross-keystone-python27-db-ubuntu-xenial/962327d/console.html#_2016-08-02_03_52_14_537923
>>
>> I'm not fully sure where to go from here. But wanted to make sure that
>> data is out there. Any keystone folks who can dive into and sort it out
>> would be highly appreciated.
>>
>>         -Sean
>>
>> --
>> Sean Dague
>> http://dague.net
>>
>> __________________________________________________________________________
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe:
>> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20160803/14e5ff8d/attachment.html>


More information about the OpenStack-dev mailing list