<html><body><p>So, digging into this a bit more, the failures that Barbican and Ironic saw were very different. The barbican team managed to fix their issues by replacing their `keystone endpoint-create` and such commands with the `openstack endpoint create` alternates.<br><br>Looking at the failure for why Ironic failed led me down a rabbit hole I wish I hadn't gone down. There are no plugins managed by the ironic, like there were in the barbican case, so there were easy commands to replace.Instead it was failing on a few swift related commands, as evidenced in the log that Lucas copied:<br><br><a href="http://logs.openstack.org/68/217068/14/check/gate-tempest-dsvm-ironic-agent_ssh/18d8590/logs/devstacklog.txt.gz#_2015-09-04_09_04_55_994">http://logs.openstack.org/68/217068/14/check/gate-tempest-dsvm-ironic-agent_ssh/18d8590/logs/devstacklog.txt.gz#_2015-09-04_09_04_55_994</a><br><a href="http://logs.openstack.org/68/217068/14/check/gate-tempest-dsvm-ironic-agent_ssh/18d8590/logs/devstacklog.txt.gz#_2015-09-04_09_04_55_527"><font face="Courier New">2015-09-04 09:04:55.527</font></a><font face="Courier New"> | + swift post -m 'Temp-URL-Key: secretkey'</font><br><a href="http://logs.openstack.org/68/217068/14/check/gate-tempest-dsvm-ironic-agent_ssh/18d8590/logs/devstacklog.txt.gz#_2015-09-04_09_04_55_994"><font face="Courier New">2015-09-04 09:04:55.994</font></a><font face="Courier New"> | Authorization Failure. Authorization Failed: The resource could not be found. (HTTP 404)</font><br><br>Jamie's patch sets everything to v3, so why is failing now? I tried this in my own environment to make sure:<br><font face="Courier New">steve@steve-vm:~/devstack$ export OS_IDENTITY_API_VERSION=3</font><br><font face="Courier New">steve@steve-vm:~/devstack$ export OS_AUTH_URL='http://10.0.2.15:5000/v3'</font><br><font face="Courier New">steve@steve-vm:~/devstack$ swift stat</font><br><font face="Courier New">Authorization Failure. Authorization Failed: The resource could not be found. (HTTP 404) (Request-ID: req-63bee2a6-ca9d-49f4-baad-b6a1eef916df)</font><br><font face="Courier New">steve@steve-vm:~/devstack$ swift --debug stat</font><br><font face="Courier New">DEBUG:keystoneclient.auth.identity.v2:Making authentication request to </font><a href="http://10.0.2.15:5000/v3/tokens"><font face="Courier New">http://10.0.2.15:5000/v3/tokens</font></a><br><br>And saw that swiftclient was creating a v2 client instance with a v3 endpoint, this is no beuno.<br><br>As I continued to dig into this, it seems like swiftclient doesn't honor the OS_IDENTITY_API_VERSION flag that was set, instead it relies on --auth-version or OS_AUTH_VERSION<br><font face="Courier New">steve@steve-vm:~/devstack$ export OS_AUTH_VERSION=3</font><br><font face="Courier New">steve@steve-vm:~/devstack$ swift --debug stat</font><br><font face="Courier New">DEBUG:keystoneclient.auth.identity.v3.base:Making authentication request to </font><a href="http://10.0.2.15:5000/v3/auth/tokens"><font face="Courier New">http://10.0.2.15:5000/v3/auth/tokens</font></a><br><font face="Courier New">INFO:requests.packages.urllib3.connectionpool:Starting new HTTP connection (1): 10.0.2.15</font><br>.... it continued and was happy<br><br>So, we could easily propose Jamies patch again, but this time also set OS_AUTH_VERSION, or we could fix swiftclient to honor the OS_IDENTITY_API_VERSION flag. I'd prefer doing the former first to get Jamie's patch back in, and the latter for the long term plan, but looking at the code, there doesn't seem to be a plan on deprecating OS_AUTH_VERSION.<br><br>Thanks,<br><br>Steve Martinelli<br>OpenStack Keystone Core<br><br><img width="16" height="16" src="cid:1__=8FBBF42BDFEBED138f9e8a93df938690918c8FB@" border="0" alt="Inactive hide details for Jamie Lennox ---2015/09/06 04:41:23 AM---Note that this fixing this does not mean ironic has to suppo"><font color="#424282">Jamie Lennox ---2015/09/06 04:41:23 AM---Note that this fixing this does not mean ironic has to support keystone v3 (but please fix that too)</font><br><br><font size="2" color="#5F5F5F">From: </font><font size="2">Jamie Lennox <jamielennox@redhat.com></font><br><font size="2" color="#5F5F5F">To: </font><font size="2">"OpenStack Development Mailing List (not for usage questions)" <openstack-dev@lists.openstack.org></font><br><font size="2" color="#5F5F5F">Date: </font><font size="2">2015/09/06 04:41 AM</font><br><font size="2" color="#5F5F5F">Subject: </font><font size="2">Re: [openstack-dev] [DevStack][Keystone][Ironic][Swit][Barbican] FYI: Defaulting to Keystone v3 API</font><br><hr width="100%" size="2" align="left" noshade style="color:#8091A5; "><br><br><br><tt>Note that this fixing this does not mean ironic has to support keystone v3 (but please fix that too). It just means that somewhere in ironic's gate it is doing like an "openstack user create" or a role assignment directly with the OSC tool assuming v2 rather than using the helpers that devstack provides like get_or_create_user. Keystone v2 still exists and is running we just changed the default API for devstack OSC commands.<br><br>I'm kind of annoyed we reverted this patch (though i was surprised to see it merge recently as it's been around for a while), as it was known to possibly break people which is why it was on the discussion for the qa meetings. However given that devstack has plugins and there is third party CI there is absolutely no way we can make sure that everyone has fixed this and we just need to make a breaking change. Granted coinciding with freeze is unfortunate. Luckily this doesn't affect most people because they use the devstack helper functions and for those that don't it's an almost trivial fix to start using them.<br><br>Jamie<br><br>----- Original Message -----<br>> From: "Steve Martinelli" <stevemar@ca.ibm.com><br>> To: "OpenStack Development Mailing List (not for usage questions)" <openstack-dev@lists.openstack.org><br>> Sent: Saturday, September 5, 2015 2:38:27 AM<br>> Subject: Re: [openstack-dev] [DevStack][Keystone][Ironic][Swit][Barbican] FYI: Defaulting to Keystone v3 API<br>> <br>> <br>> <br>> This change also affected Barbican too, but they quickly tossed up a patch to<br>> resolve the gate failures [1]. As much as I would like DevStack and<br>> OpenStackClient to default to Keystone's v3 API, we should - considering how<br>> close we are in the schedule, revert the initial patch (which I see sdague<br>> already did). We need to determine which projects are hosting their own<br>> devstack plugin scripts and update those first before bringing back the<br>> original patch.<br>> <br>> </tt><tt><a href="https://review.openstack.org/#/c/220396/">https://review.openstack.org/#/c/220396/</a></tt><tt><br>> <br>> Thanks,<br>> <br>> Steve Martinelli<br>> OpenStack Keystone Core<br>> <br>> Lucas Alvares Gomes ---2015/09/04 10:07:51 AM---Hi, This is email is just a<br>> FYI: Recently the patch [1] got merged in<br>> <br>> From: Lucas Alvares Gomes <lucasagomes@gmail.com><br>> To: OpenStack Development Mailing List <openstack-dev@lists.openstack.org><br>> Date: 2015/09/04 10:07 AM<br>> Subject: [openstack-dev] [DevStack][Keystone][Ironic][Swit] FYI: Defaulting<br>> to Keystone v3 API<br>> <br>> <br>> <br>> <br>> Hi,<br>> <br>> This is email is just a FYI: Recently the patch [1] got merged in<br>> DevStack and broke the Ironic gate [2], I haven't had time to dig into<br>> the problem yet so I reverted the patch [3] to unblock our gate.<br>> <br>> The work to convert to v3 seems to be close enough but not yet there<br>> so I just want to bring a broader attention to it with this email.<br>> <br>> Also, the Ironic job that is currently running in the DevStack gate is<br>> not testing Ironic with the Swift module, there's a patch [4] changing<br>> that so I hope we will be able to identify the problem before we break<br>> things next time .<br>> <br>> [1] </tt><tt><a href="https://review.openstack.org/#/c/186684/">https://review.openstack.org/#/c/186684/</a></tt><tt><br>> [2]<br>> </tt><tt><a href="http://logs.openstack.org/68/217068/14/check/gate-tempest-dsvm-ironic-agent_ssh/18d8590/logs/devstacklog.txt.gz#_2015-09-04_09_04_55_994">http://logs.openstack.org/68/217068/14/check/gate-tempest-dsvm-ironic-agent_ssh/18d8590/logs/devstacklog.txt.gz#_2015-09-04_09_04_55_994</a></tt><tt><br>> [3] </tt><tt><a href="https://review.openstack.org/220532">https://review.openstack.org/220532</a></tt><tt><br>> [4] </tt><tt><a href="https://review.openstack.org/#/c/220516/">https://review.openstack.org/#/c/220516/</a></tt><tt><br>> <br>> Cheers,<br>> Lucas<br>> <br>> __________________________________________________________________________<br>> OpenStack Development Mailing List (not for usage questions)<br>> Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe<br>> </tt><tt><a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a></tt><tt><br>> <br>> <br>> <br>> <br>> <br>> __________________________________________________________________________<br>> OpenStack Development Mailing List (not for usage questions)<br>> Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe<br>> </tt><tt><a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a></tt><tt><br>> <br><br>__________________________________________________________________________<br>OpenStack Development Mailing List (not for usage questions)<br>Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe<br></tt><tt><a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a></tt><tt><br><br></tt><br><br><BR>
</body></html>