Strange behaviour of OSC in keystone MFA context
Hello, I'm experiencing the following strange behavior of openstack CLI with os-auth-methods option (most parameters are defined in clouds.yaml): $ openstack token issue --os-auth-type v3multifactor --os-auth-methods password,totp The plugin p could not be found Note that "p" is the first letter of "password". It looks like the option parser handled "password,totp" as a string instead of as a list of strings. Version of openstack CLI is 5.4.0. Any idea ? Thanks ! Jean-François
On Tue, 2021-01-26 at 17:46 +0000, Taltavull Jean-Francois wrote:
Hello,
I'm experiencing the following strange behavior of openstack CLI with os-auth-methods option (most parameters are defined in clouds.yaml):
$ openstack token issue --os-auth-type v3multifactor --os-auth-methods password,totp
--os-auth-methods does not appear to be a standard part of osc infact i cant find it in any openstack repo with i think this is the implemtaions https://opendev.org/openstack/keystoneauth/src/branch/master/keystoneauth1/l... this presumable is where it generates teh optins options.extend([ loading.Opt( 'auth_methods', required=True, help="Methods to authenticate with."), ]) if i do openstack help --os-auth-type v3multifactor it does show up with the following text --os-auth-methods <auth-auth-methods> With v3multifactor: Methods to authenticate with. (Env: OS_AUTH_METHODS) that does not say much but https://opendev.org/openstack/keystoneauth/src/branch/master/keystoneauth1/t... implies its a list with that said there are no test for multifactor as far as i can see like this one https://opendev.org/openstack/python-openstackclient/src/branch/master/opens... there also does not seam too be a release note declaring support. so while keystone auth support multi factor im not sure that osc actully does i specpec that the fild type is not correct and it is indeed been parsed as a string instead of a list of stirng field. it might be fixable via keystoneauth but it proably need osc support and testing.
The plugin p could not be found
Note that "p" is the first letter of "password". It looks like the option parser handled "password,totp" as a string instead of as a list of strings.
Version of openstack CLI is 5.4.0.
Any idea ?
Thanks !
Jean-François
-----Original Message----- From: Sean Mooney <smooney@redhat.com> Sent: mardi, 26 janvier 2021 20:01 To: openstack-discuss@lists.openstack.org Subject: Re: Strange behaviour of OSC in keystone MFA context
On Tue, 2021-01-26 at 17:46 +0000, Taltavull Jean-Francois wrote:
Hello,
I'm experiencing the following strange behavior of openstack CLI with os- auth-methods option (most parameters are defined in clouds.yaml):
$ openstack token issue --os-auth-type v3multifactor --os-auth-methods password,totp
--os-auth-methods does not appear to be a standard part of osc infact i cant find it in any openstack repo with
i think this is the implemtaions https://opendev.org/openstack/keystoneauth/src/branch/master/keystoneauth 1/loading/_plugins/identity/v3.py#L303-L340
this presumable is where it generates teh optins
options.extend([ loading.Opt( 'auth_methods', required=True, help="Methods to authenticate with."), ])
if i do openstack help --os-auth-type v3multifactor it does show up with the following text
--os-auth-methods <auth-auth-methods> With v3multifactor: Methods to authenticate with. (Env: OS_AUTH_METHODS)
that does not say much but
https://opendev.org/openstack/keystoneauth/src/branch/master/keystoneauth 1/tests/unit/identity/test_identity_v3.py#L762-L800 implies its a list
with that said there are no test for multifactor as far as i can see like this one https://opendev.org/openstack/python- openstackclient/src/branch/master/openstackclient/tests/functional/common/t est_args.py#L66-L79
there also does not seam too be a release note declaring support.
so while keystone auth support multi factor im not sure that osc actully does
i specpec that the fild type is not correct and it is indeed been parsed as a string instead of a list of stirng field. it might be fixable via keystoneauth but it proably need osc support and testing.
The plugin p could not be found
Note that "p" is the first letter of "password". It looks like the option parser handled "password,totp" as a string instead of as a list of strings.
Version of openstack CLI is 5.4.0.
Any idea ?
Thanks !
Jean-François
Thanks for your answer Sean. What can I do on my end to get things done ? Jean-François
On Thu, 2021-01-28 at 07:59 +0000, Taltavull Jean-Francois wrote:
-----Original Message----- From: Sean Mooney <smooney@redhat.com> Sent: mardi, 26 janvier 2021 20:01 To: openstack-discuss@lists.openstack.org Subject: Re: Strange behaviour of OSC in keystone MFA context
On Tue, 2021-01-26 at 17:46 +0000, Taltavull Jean-Francois wrote:
Hello,
I'm experiencing the following strange behavior of openstack CLI with os- auth-methods option (most parameters are defined in clouds.yaml):
$ openstack token issue --os-auth-type v3multifactor --os-auth-methods password,totp
--os-auth-methods does not appear to be a standard part of osc infact i cant find it in any openstack repo with
i think this is the implemtaions https://opendev.org/openstack/keystoneauth/src/branch/master/keystoneauth 1/loading/_plugins/identity/v3.py#L303-L340
this presumable is where it generates teh optins
options.extend([ loading.Opt( 'auth_methods', required=True, help="Methods to authenticate with."), ])
if i do openstack help --os-auth-type v3multifactor it does show up with the following text
--os-auth-methods <auth-auth-methods> With v3multifactor: Methods to authenticate with. (Env: OS_AUTH_METHODS)
that does not say much but
https://opendev.org/openstack/keystoneauth/src/branch/master/keystoneauth 1/tests/unit/identity/test_identity_v3.py#L762-L800 implies its a list
with that said there are no test for multifactor as far as i can see like this one https://opendev.org/openstack/python- openstackclient/src/branch/master/openstackclient/tests/functional/common/t est_args.py#L66-L79
there also does not seam too be a release note declaring support.
so while keystone auth support multi factor im not sure that osc actully does
i specpec that the fild type is not correct and it is indeed been parsed as a string instead of a list of stirng field. it might be fixable via keystoneauth but it proably need osc support and testing.
The plugin p could not be found
Note that "p" is the first letter of "password". It looks like the option parser handled "password,totp" as a string instead of as a list of strings.
Version of openstack CLI is 5.4.0.
Any idea ?
Thanks !
Jean-François
Thanks for your answer Sean.
What can I do on my end to get things done ? well unfortunetly i do not work on keystone or osc i just saw your mail while i was waiting for some tests to finish running.
with that said i have upstaed the subject to include both projects so hopefully that will get the attention of those that can help.
Jean-François
On Thu, 2021-01-28 at 12:38 +0000, Sean Mooney wrote:
On Thu, 2021-01-28 at 07:59 +0000, Taltavull Jean-Francois wrote:
-----Original Message----- From: Sean Mooney <smooney@redhat.com> Sent: mardi, 26 janvier 2021 20:01 To: openstack-discuss@lists.openstack.org Subject: Re: Strange behaviour of OSC in keystone MFA context
On Tue, 2021-01-26 at 17:46 +0000, Taltavull Jean-Francois wrote:
Hello,
I'm experiencing the following strange behavior of openstack CLI with os- auth-methods option (most parameters are defined in clouds.yaml):
$ openstack token issue --os-auth-type v3multifactor --os-auth-methods password,totp
--os-auth-methods does not appear to be a standard part of osc infact i cant find it in any openstack repo with
i think this is the implemtaions https://opendev.org/openstack/keystoneauth/src/branch/master/keystoneauth 1/loading/_plugins/identity/v3.py#L303-L340
this presumable is where it generates teh optins
options.extend([ loading.Opt( 'auth_methods', required=True, help="Methods to authenticate with."), ])
if i do openstack help --os-auth-type v3multifactor it does show up with the following text
--os-auth-methods <auth-auth-methods> With v3multifactor: Methods to authenticate with. (Env: OS_AUTH_METHODS)
that does not say much but
https://opendev.org/openstack/keystoneauth/src/branch/master/keystoneauth 1/tests/unit/identity/test_identity_v3.py#L762-L800 implies its a list
with that said there are no test for multifactor as far as i can see like this one https://opendev.org/openstack/python- openstackclient/src/branch/master/openstackclient/tests/functional/common/t est_args.py#L66-L79
there also does not seam too be a release note declaring support.
so while keystone auth support multi factor im not sure that osc actully does
i specpec that the fild type is not correct and it is indeed been parsed as a string instead of a list of stirng field. it might be fixable via keystoneauth but it proably need osc support and testing.
The plugin p could not be found
Note that "p" is the first letter of "password". It looks like the option parser handled "password,totp" as a string instead of as a list of strings.
Version of openstack CLI is 5.4.0.
Any idea ?
Thanks !
Jean-François
Thanks for your answer Sean.
What can I do on my end to get things done ? well unfortunetly i do not work on keystone or osc i just saw your mail while i was waiting for some tests to finish running.
with that said i have upstaed the subject to include both projects so hopefully that will get the attention of those that can help.
The definition for those opts can be found at [1]. As Sean thought it might be, that is using the default type defined in the parent 'Opt' class of 'str' [2]. We don't expose argparse's 'action' parameter that would allow us to use the 'append' action, so you'd have to fix this by parsing whatever the user provided after the fact. I suspect you could resolve the immediate issue by changing this line [3] from: self._methods = kwargs['auth_methods'] to: self._methods = kwargs['auth_methods'].split(',') However, I assume there's likely more to this issue. I don't have an environment to hand to validate this fix, unfortunately. If you do manage to test that change and it works, I'd be happy to help you in getting a patch proposed to 'keystoneauth'. Hope this helps, Stephen [1] https://github.com/openstack/keystoneauth/blob/4.3.0/keystoneauth1/loading/_... [2] https://github.com/openstack/keystoneauth/blob/4.3.0/keystoneauth1/loading/o... [3] https://github.com/openstack/keystoneauth/blob/4.3.0/keystoneauth1/loading/_...
Jean-François
*puts up hand* You can blame me for this. When I implemented this I didn't (and still don't) fully understand how the loading stuff works in Keystoneauth and how it works with other things like OSC. I was more focused on getting the direct auth/session side working because the loading stuff isn't that useful in OSC really (see below why). If anyone does know the loader side of keystoneauth better please chime in and save me from making too much of a fool of myself! I think adding `.split(',')` is probably enough, but you'd want to also update the help text clarify that the option is a 'comma separated list'. The biggest issue, and why this area never got much testing, is because it is effectively useless since you'd have to supply your MFA values EVERY command. Imagine how awful that would be for TOTP. The whole point of the MFA process in keystone with auth-receipt was a dynamic interactive login. Supplying the MFA upfront isn't that useful. What the OSC really needs is a `login` command, that goes through a login process using the auth-receipts workflow from keystone (asks for password/totp) and sets some sort of state file. We can't set the environment variables of the parent shell process, so we'd have to go with a state/session file. But to avoid it clashing with other state files and terminal sessions we'd need some way to tag them by the parent process ID so you can login to more than one cloud/project/user/etc across multiple terminals. In addition it would be really nice if the OSC had some way of reusing a scoped token/catalog rather than having to fetch it every time, but I feel that would have to include changes in Keystoneauth to supply it some cached data which tells it to not attempt to reauthenticate but rather trust the catalog/token supplied (does keystoneauth support that?). Because that reauth every command via Keystoneauth is actually what often takes longer than the actual API calls... We can also just throw catalog into that state/session file as json/yaml. On 29/01/21 7:03 am, Stephen Finucane wrote:
The definition for those opts can be found at [1]. As Sean thought it might be, that is using the default type defined in the parent 'Opt' class of 'str' [2]. We don't expose argparse's 'action' parameter that would allow us to use the 'append' action, so you'd have to fix this by parsing whatever the user provided after the fact. I suspect you could resolve the immediate issue by changing this line [3] from:
self._methods = kwargs['auth_methods']
to:
self._methods = kwargs['auth_methods'].split(',')
However, I assume there's likely more to this issue. I don't have an environment to hand to validate this fix, unfortunately.
If you do manage to test that change and it works, I'd be happy to help you in getting a patch proposed to 'keystoneauth'.
Hope this helps, Stephen
[1] https://github.com/openstack/keystoneauth/blob/4.3.0/keystoneauth1/loading/_... [2] https://github.com/openstack/keystoneauth/blob/4.3.0/keystoneauth1/loading/o... [3] https://github.com/openstack/keystoneauth/blob/4.3.0/keystoneauth1/loading/_...
Jean-François
Hi
On 2. Feb 2021, at 01:43, Adrian Turjak <adriant@catalystcloud.nz> wrote:
*puts up hand*
You can blame me for this. When I implemented this I didn't (and still don't) fully understand how the loading stuff works in Keystoneauth and how it works with other things like OSC. I was more focused on getting the direct auth/session side working because the loading stuff isn't that useful in OSC really (see below why).
If anyone does know the loader side of keystoneauth better please chime in and save me from making too much of a fool of myself!
I think adding `.split(',')` is probably enough, but you'd want to also update the help text clarify that the option is a 'comma separated list'.
The biggest issue, and why this area never got much testing, is because it is effectively useless since you'd have to supply your MFA values EVERY command. Imagine how awful that would be for TOTP. The whole point of the MFA process in keystone with auth-receipt was a dynamic interactive login. Supplying the MFA upfront isn't that useful.
What the OSC really needs is a `login` command, that goes through a login process using the auth-receipts workflow from keystone (asks for password/totp) and sets some sort of state file. We can't set the environment variables of the parent shell process, so we'd have to go with a state/session file. But to avoid it clashing with other state files and terminal sessions we'd need some way to tag them by the parent process ID so you can login to more than one cloud/project/user/etc across multiple terminals.
I guess we can do something about that. Recently Monty started and I took over the patch for adding token caching in the keyring[1]. As such it will not really help, but together with [2] and [3] we can use authorisation caching on the OSC side. I was never really giving priority to this, since in a regular use case it perhaps saves .5 - 1 second, what is not really noticeable (most time is wasted on initialization). However in this context it might become really handy. Feel free to trigger discussion if that looks important. And yes, I totally agree on the fact, that TOTP/MFA for scripting is a total disaster, therefore nobody really uses it.
In addition it would be really nice if the OSC had some way of reusing a scoped token/catalog rather than having to fetch it every time, but I feel that would have to include changes in Keystoneauth to supply it some cached data which tells it to not attempt to reauthenticate but rather trust the catalog/token supplied (does keystoneauth support that?). Because that reauth every command via Keystoneauth is actually what often takes longer than the actual API calls... We can also just throw catalog into that state/session file as json/yaml.
On 29/01/21 7:03 am, Stephen Finucane wrote:
The definition for those opts can be found at [1]. As Sean thought it might be, that is using the default type defined in the parent 'Opt' class of 'str' [2]. We don't expose argparse's 'action' parameter that would allow us to use the 'append' action, so you'd have to fix this by parsing whatever the user provided after the fact. I suspect you could resolve the immediate issue by changing this line [3] from:
self._methods = kwargs['auth_methods']
to:
self._methods = kwargs['auth_methods'].split(',')
However, I assume there's likely more to this issue. I don't have an environment to hand to validate this fix, unfortunately.
If you do manage to test that change and it works, I'd be happy to help you in getting a patch proposed to 'keystoneauth'.
Hope this helps, Stephen
[1] https://github.com/openstack/keystoneauth/blob/4.3.0/keystoneauth1/loading/_... [2] https://github.com/openstack/keystoneauth/blob/4.3.0/keystoneauth1/loading/o... [3] https://github.com/openstack/keystoneauth/blob/4.3.0/keystoneauth1/loading/_...
Jean-François
[1] https://review.opendev.org/c/openstack/openstacksdk/+/735352 <https://review.opendev.org/c/openstack/openstacksdk/+/735352> [2] https://review.opendev.org/c/openstack/python-openstackclient/+/765652 <https://review.opendev.org/c/openstack/python-openstackclient/+/765652> [3] https://review.opendev.org/c/openstack/osc-lib/+/765650 <https://review.opendev.org/c/openstack/osc-lib/+/765650>
On 3/02/21 7:22 am, Artem Goncharov wrote:
Hi
On 2. Feb 2021, at 01:43, Adrian Turjak <adriant@catalystcloud.nz <mailto:adriant@catalystcloud.nz>> wrote:
The biggest issue, and why this area never got much testing, is because it is effectively useless since you'd have to supply your MFA values EVERY command. Imagine how awful that would be for TOTP. The whole point of the MFA process in keystone with auth-receipt was a dynamic interactive login. Supplying the MFA upfront isn't that useful.
What the OSC really needs is a `login` command, that goes through a login process using the auth-receipts workflow from keystone (asks for password/totp) and sets some sort of state file. We can't set the environment variables of the parent shell process, so we'd have to go with a state/session file. But to avoid it clashing with other state files and terminal sessions we'd need some way to tag them by the parent process ID so you can login to more than one cloud/project/user/etc across multiple terminals.
I guess we can do something about that. Recently Monty started and I took over the patch for adding token caching in the keyring[1]. As such it will not really help, but together with [2] and [3] we can use authorisation caching on the OSC side. I was never really giving priority to this, since in a regular use case it perhaps saves .5 - 1 second, what is not really noticeable (most time is wasted on initialization). However in this context it might become really handy. Feel free to trigger discussion if that looks important.
And yes, I totally agree on the fact, that TOTP/MFA for scripting is a total disaster, therefore nobody really uses it. I definitely do think it is important, but then again so would getting MFA working in Horizon which I was planning to do, but circumstances beyond my control stopped me from doing that, and I didn't work on finding someone else to implement it.
If we did get it working on the CLI, then there might be more push to get it working on Horizon as well. How auth-receipts and MFA work is documented fairly well from memory, and we have a very clear error thrown that lets you build an interactive workflow for asking for the missing pieces of auth: https://docs.openstack.org/keystoneauth/latest/authentication-plugins.html#m... I can't find the time to implement anything here right now because of so much internal work, but anything related to MFA feel free to ping me or just outright add me as reviewer!
participants (5)
-
Adrian Turjak
-
Artem Goncharov
-
Sean Mooney
-
Stephen Finucane
-
Taltavull Jean-Francois