<div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote">On Fri, Jul 21, 2017 at 9:39 PM, Monty Taylor <span dir="ltr"><<a href="mailto:mordred@inaugust.com" target="_blank">mordred@inaugust.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><span class="gmail-">On 07/22/2017 07:14 AM, Lance Bragstad wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
After a little head scratching and a Pantera playlist later, we ended up<br>
figuring out the main causes. The failures can be found in the gate [0].<br>
The two failures are detailed below:<br>
<br>
1.) Keystoneauth version 3.0.0 added a lot of functionality and might<br>
return a different url depending on discovery. Keystonemiddleware use to<br>
be able to mock urls to keystone in this case because keystoneauth<br>
didn't modify the url in between. Keystonemiddleware didn't know how to<br>
deal with the new url and the result was a Mock failure. This is<br>
something that we can fix in keystonemiddleware once we have a version<br>
of keystoneauth that covers all discovery cases and does the right<br>
thing. NOTE: If you're mocking requests to keystone and using<br>
keystoneauth somewhere in your project's tests, you'll have to deal with<br>
this. More on that below.<br>
</blockquote>
<br></span>
Upon further digging - this one is actually quite a bit easier. There are cases where keystoneauth finds an unversioned discovery endpoint from a versioned endpoint in the catalog. It's done for quite a while, so the behavior isn't new. HOWEVER - a bug snuck in that caused the url it infers to come back without a trailing '/'. So the requests_mock entry in keystonemiddleware was for <a href="http://keystone.url/admin/" rel="noreferrer" target="_blank">http://keystone.url/admin/</a> and keystoneauth was doing a get on <a href="http://keystone.url/admin" rel="noreferrer" target="_blank">http://keystone.url/admin</a>.<br>
<br>
It's a behavior change and a bug, so we're working up a fix for it. The short story is though that once we fix it it should not cause anyone to need to update requests_mock entries.</blockquote><div><br></div><div>Ah - thanks for keeping me honest here. Good to know both issues will be fixed with the same patch. For context, this was the thought process as we worked through things earlier [0].</div><div><br></div><div>I appreciate the follow-up!</div><div> </div><div><br></div><div>[0] <a href="http://eavesdrop.openstack.org/irclogs/%23openstack-keystone/%23openstack-keystone.2017-07-21.log.html#t2017-07-21T19:57:30">http://eavesdrop.openstack.org/irclogs/%23openstack-keystone/%23openstack-keystone.2017-07-21.log.html#t2017-07-21T19:57:30</a></div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><span class="gmail-"><br>
<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
2.) The other set of failures were because keystoneauth wasn't expecting<br>
a URL without a path [1], causing an index error. I tested the fix [2]<br>
against keystonemiddleware and it seems to take care of the issue. Eric<br>
is working on a fix. Once that patch is fully tested and vetted we'll<br>
roll another keystoneauth release (3.0.1) and use that to test<br>
keystonemiddleware to handle the mocking issues described in #1. From<br>
there we should be able to safely bump the minimum version to 3.0.1, and<br>
avoid 3.0.0 all together.<br>
</blockquote>
<br></span>
Patch is up for this one, and we've confirmed it fixes this issue.<br>
<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div><div class="gmail-h5">
Let me know if you see anything else suspicious with respect to<br>
keystoneauth. Thanks!<br>
<br>
<br>
[0]<br>
<a href="http://logs.openstack.org/84/486184/1/check/gate-keystonemiddleware-python27-ubuntu-xenial/7c079da/testr_results.html.gz" rel="noreferrer" target="_blank">http://logs.openstack.org/84/4<wbr>86184/1/check/gate-keystonemid<wbr>dleware-python27-ubuntu-<wbr>xenial/7c079da/testr_results.<wbr>html.gz</a><br>
[1]<br>
<a href="https://github.com/openstack/keystoneauth/blob/5715035f42780d8979d458e9f7e3c625962b2749/keystoneauth1/discover.py#L947" rel="noreferrer" target="_blank">https://github.com/openstack/k<wbr>eystoneauth/blob/5715035f42780<wbr>d8979d458e9f7e3c625962b2749/<wbr>keystoneauth1/discover.py#L947</a><br>
[2] <a href="https://review.openstack.org/#/c/486231/1" rel="noreferrer" target="_blank">https://review.openstack.org/#<wbr>/c/486231/1</a><br>
<br>
On 07/21/2017 04:43 PM, Lance Bragstad wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
The patch to blacklist version 3.0.0 is working through the moment [0].<br>
We also have a WIP patch proposed to handled the cases exposed by<br>
keystonemiddleware [1].<br>
<br>
<br>
[0] <a href="https://review.openstack.org/#/c/486223/" rel="noreferrer" target="_blank">https://review.openstack.org/#<wbr>/c/486223/</a><br>
[1] <a href="https://review.openstack.org/#/c/486231/" rel="noreferrer" target="_blank">https://review.openstack.org/#<wbr>/c/486231/</a><br>
<br>
<br>
On 07/21/2017 03:58 PM, Lance Bragstad wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
We have a patch up to blacklist version 3.0.0 from global-requirements<br>
[0]. We're also going to hold bumping the minimum version of<br>
keystoneauth until we have things back to normal [1].<br>
<br>
<br>
[0] <a href="https://review.openstack.org/#/c/486223/" rel="noreferrer" target="_blank">https://review.openstack.org/#<wbr>/c/486223/</a><br>
[1] <a href="https://review.openstack.org/#/c/486160/1" rel="noreferrer" target="_blank">https://review.openstack.org/#<wbr>/c/486160/1</a><br>
<br>
On 07/21/2017 03:00 PM, Lance Bragstad wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
I started noticing some trivial changes failing in the<br>
keystonemiddleware gate [0]. The failures are in tests that use the<br>
keystoneauth1 library (8 tests are failing by my count), which we<br>
released a new version of yesterday [1]. I've proposed a patch to<br>
blacklist keystoneauth1 3.0.0 from keystonemiddleware until we can<br>
figure out what happened [2]. Status is being tracked in a bug against<br>
keystonemiddleware [3], but might need to be broadened if these changes<br>
are affecting other projects.<br>
<br>
I'll be in -keystone working through some of the issues if you need me.<br>
<br>
Thanks,<br>
<br>
Lance<br>
<br>
[0] <a href="https://review.openstack.org/#/c/486184/" rel="noreferrer" target="_blank">https://review.openstack.org/#<wbr>/c/486184/</a><br>
[1] <a href="http://lists.openstack.org/pipermail/openstack-dev/2017-July/119969.html" rel="noreferrer" target="_blank">http://lists.openstack.org/pip<wbr>ermail/openstack-dev/2017-July<wbr>/119969.html</a><br>
[2] <a href="https://review.openstack.org/#/c/486213/" rel="noreferrer" target="_blank">https://review.openstack.org/#<wbr>/c/486213/</a><br>
[3] <a href="https://bugs.launchpad.net/keystonemiddleware/+bug/1705770" rel="noreferrer" target="_blank">https://bugs.launchpad.net/key<wbr>stonemiddleware/+bug/1705770</a><br>
<br>
<br>
</blockquote></blockquote>
<br>
</blockquote>
<br>
<br>
<br>
<br></div></div>
______________________________<wbr>______________________________<wbr>______________<br>
OpenStack Development Mailing List (not for usage questions)<br>
Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank">OpenStack-dev-request@lists.op<wbr>enstack.org?subject:unsubscrib<wbr>e</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" target="_blank">http://lists.openstack.org/cgi<wbr>-bin/mailman/listinfo/openstac<wbr>k-dev</a><br>
<br>
</blockquote>
<br>
<br>
______________________________<wbr>______________________________<wbr>______________<br>
OpenStack Development Mailing List (not for usage questions)<br>
Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank">OpenStack-dev-request@lists.op<wbr>enstack.org?subject:unsubscrib<wbr>e</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" target="_blank">http://lists.openstack.org/cgi<wbr>-bin/mailman/listinfo/openstac<wbr>k-dev</a><br>
</blockquote></div><br></div></div>