Open Stack

So, digging into this a bit more, the failures that Barbican and Ironic saw
were very different. The barbican team managed to fix their issues by
replacing their `keystone endpoint-create` and such commands with the
`openstack endpoint create` alternates.

Looking at the failure for why Ironic failed led me down a rabbit hole I
wish I hadn't gone down. There are no plugins managed by the ironic, like
there were in the barbican case, so there were easy commands to
replace.Instead it was failing on a few swift related commands, as
evidenced in the log that Lucas copied:

http://logs.openstack.org/68/217068/14/check/gate-tempest-dsvm-ironic-agent_ssh/18d8590/logs/devstacklog.txt.gz#_2015-09-04_09_04_55_994
2015-09-04 09:04:55.527 | + swift post -m 'Temp-URL-Key: secretkey'
2015-09-04 09:04:55.994 | Authorization Failure. Authorization Failed: The
resource could not be found. (HTTP 404)

Jamie's patch sets everything to v3, so why is failing now? I tried this in
my own environment to make sure:
steve at steve-vm:~/devstack$ export OS_IDENTITY_API_VERSION=3
steve at steve-vm:~/devstack$ export OS_AUTH_URL='http://10.0.2.15:5000/v3'
steve at steve-vm:~/devstack$ swift stat
Authorization Failure. Authorization Failed: The resource could not be
found. (HTTP 404) (Request-ID: req-63bee2a6-ca9d-49f4-baad-b6a1eef916df)
steve at steve-vm:~/devstack$ swift --debug stat
DEBUG:keystoneclient.auth.identity.v2:Making authentication request to
http://10.0.2.15:5000/v3/tokens

And saw that swiftclient was creating a v2 client instance with a v3
endpoint, this is no beuno.

As I continued to dig into this, it seems like swiftclient doesn't honor
the OS_IDENTITY_API_VERSION flag that was set, instead it relies on
--auth-version or OS_AUTH_VERSION
steve at steve-vm:~/devstack$ export OS_AUTH_VERSION=3
steve at steve-vm:~/devstack$ swift --debug stat
DEBUG:keystoneclient.auth.identity.v3.base:Making authentication request to
http://10.0.2.15:5000/v3/auth/tokens
INFO:requests.packages.urllib3.connectionpool:Starting new HTTP connection
(1): 10.0.2.15
.... it continued and was happy

So, we could easily propose Jamies patch again, but this time also set
OS_AUTH_VERSION, or we could fix swiftclient to honor the
OS_IDENTITY_API_VERSION flag. I'd prefer doing the former first to get
Jamie's patch back in, and the latter for the long term plan, but looking
at the code, there doesn't seem to be a plan on deprecating
OS_AUTH_VERSION.

Thanks,

Steve Martinelli
OpenStack Keystone Core



From:	Jamie Lennox <jamielennox at redhat.com>
To:	"OpenStack Development Mailing List (not for usage questions)"
            <openstack-dev at lists.openstack.org>
Date:	2015/09/06 04:41 AM
Subject:	Re: [openstack-dev]
            [DevStack][Keystone][Ironic][Swit][Barbican] FYI: Defaulting to
            Keystone v3 API



Note that this fixing this does not mean ironic has to support keystone v3
(but please fix that too). It just means that somewhere in ironic's gate it
is doing like an "openstack user create" or a role assignment directly with
the OSC tool assuming v2 rather than using the helpers that devstack
provides like get_or_create_user. Keystone v2 still exists and is running
we just changed the default API for devstack OSC commands.

I'm kind of annoyed we reverted this patch (though i was surprised to see
it merge recently as it's been around for a while), as it was known to
possibly break people which is why it was on the discussion for the qa
meetings. However given that devstack has plugins and there is third party
CI there is absolutely no way we can make sure that everyone has fixed this
and we just need to make a breaking change. Granted coinciding with freeze
is unfortunate. Luckily this doesn't affect most people because they use
the devstack helper functions and for those that don't it's an almost
trivial fix to start using them.

Jamie

----- Original Message -----
> From: "Steve Martinelli" <stevemar at ca.ibm.com>
> To: "OpenStack Development Mailing List (not for usage questions)"
<openstack-dev at lists.openstack.org>
> Sent: Saturday, September 5, 2015 2:38:27 AM
> Subject: Re: [openstack-dev] [DevStack][Keystone][Ironic][Swit][Barbican]
FYI: Defaulting to Keystone v3 API
>
>
>
> This change also affected Barbican too, but they quickly tossed up a
patch to
> resolve the gate failures [1]. As much as I would like DevStack and
> OpenStackClient to default to Keystone's v3 API, we should - considering
how
> close we are in the schedule, revert the initial patch (which I see
sdague
> already did). We need to determine which projects are hosting their own
> devstack plugin scripts and update those first before bringing back the
> original patch.
>
> https://review.openstack.org/#/c/220396/
>
> Thanks,
>
> Steve Martinelli
> OpenStack Keystone Core
>
> Lucas Alvares Gomes ---2015/09/04 10:07:51 AM---Hi, This is email is just
a
> FYI: Recently the patch [1] got merged in
>
> From: Lucas Alvares Gomes <lucasagomes at gmail.com>
> To: OpenStack Development Mailing List
<openstack-dev at lists.openstack.org>
> Date: 2015/09/04 10:07 AM
> Subject: [openstack-dev] [DevStack][Keystone][Ironic][Swit] FYI:
Defaulting
> to Keystone v3 API
>
>
>
>
> Hi,
>
> This is email is just a FYI: Recently the patch [1] got merged in
> DevStack and broke the Ironic gate [2], I haven't had time to dig into
> the problem yet so I reverted the patch [3] to unblock our gate.
>
> The work to convert to v3 seems to be close enough but not yet there
> so I just want to bring a broader attention to it with this email.
>
> Also, the Ironic job that is currently running in the DevStack gate is
> not testing Ironic with the Swift module, there's a patch [4] changing
> that so I hope we will be able to identify the problem before we break
> things next time .
>
> [1] https://review.openstack.org/#/c/186684/
> [2]
>
http://logs.openstack.org/68/217068/14/check/gate-tempest-dsvm-ironic-agent_ssh/18d8590/logs/devstacklog.txt.gz#_2015-09-04_09_04_55_994

> [3] https://review.openstack.org/220532
> [4] https://review.openstack.org/#/c/220516/
>
> Cheers,
> Lucas
>
>
__________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe:
OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
>
>
>
>
__________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe:
OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>

__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20150906/c759f352/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: graycol.gif
Type: image/gif
Size: 105 bytes
Desc: not available
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20150906/c759f352/attachment.gif>