<div dir="ltr">Sending a follow-up because I think we ended up finding something relevant to this discussion.<div><br></div><div>As keystone moves towards making fernet the default, one of our work items was to mock the system clock in tests. This allows us to advance the clock by one second where we need to avoid sub-second race conditions. To do this we used freezegun [0]. We recently landed a bunch of fixes to do this.</div><div><br></div><div>It turns out that there is a possible race between when freezegun patches it's modules and when the test runs. This turned up in a patch I was working on locally and I noticed certain clock operations weren't using the fake time object from freezegun. As a work-around, we can leverage the set_time_override() method from oslo_utils.timeutils to make sure we are using the fake time from within the frozen time context. In my testing locally this worked.</div><div><br></div><div>If keystone requires a hybrid approach to patching (oslo_utils.timeutils.set_time_override() + freezegun), we should build it into a well documented hybrid context manager so that's its more apparent why we need it.</div><div><br></div><div>Sean, I can start working on this to see if it starts mitigating the races you're seeing.</div><div><br></div><div>[0] <a href="https://pypi.python.org/pypi/freezegun">https://pypi.python.org/pypi/freezegun</a></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Aug 2, 2016 at 9:21 AM, Lance Bragstad <span dir="ltr"><<a href="mailto:lbragstad@gmail.com" target="_blank">lbragstad@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Hi Sean,<div><br></div><div>Thanks for the information. This obviously looks Fernet-related and I would be happy to spend some cycles on it. We recently landed a bunch of refactors in keystone to improve Fernet test coverage. This could be related to those refactors. Just double checking - but you haven't opened a bug in launchpad for this yet have you?</div><div><br></div><div>Thanks for the heads up!</div></div><div class="HOEnZb"><div class="h5"><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Aug 2, 2016 at 5:32 AM, Sean Dague <span dir="ltr"><<a href="mailto:sean@dague.net" target="_blank">sean@dague.net</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">One of my concerns about stacking up project unit tests in the<br>
requirements jobs, is the unit tests aren't as free of races as you<br>
would imagine. Because they only previously impacted the one project<br>
team, those teams are often just fast to recheck instead of get to the<br>
bottom of it. Cross testing with them in a voting way changes their impact.<br>
<br>
The keystone unit tests have a existing race condition in them, which<br>
recently failed an unrelated requirements bump -<br>
<a href="http://logs.openstack.org/50/348250/6/check/gate-cross-keystone-python27-db-ubuntu-xenial/962327d/console.html#_2016-08-02_03_52_14_537923" rel="noreferrer" target="_blank">http://logs.openstack.org/50/348250/6/check/gate-cross-keystone-python27-db-ubuntu-xenial/962327d/console.html#_2016-08-02_03_52_14_537923</a><br>
<br>
I'm not fully sure where to go from here. But wanted to make sure that<br>
data is out there. Any keystone folks who can dive into and sort it out<br>
would be highly appreciated.<br>
<span><font color="#888888"><br>
-Sean<br>
<br>
--<br>
Sean Dague<br>
<a href="http://dague.net" rel="noreferrer" target="_blank">http://dague.net</a><br>
<br>
__________________________________________________________________________<br>
OpenStack Development Mailing List (not for usage questions)<br>
Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
</font></span></blockquote></div><br></div>
</div></div></blockquote></div><br></div>