<div dir="ltr"><div class="gmail_extra"><div><div class="gmail-m_-2688336320334480940gmail_signature"><div dir="ltr"><div>Hi all,</div><div><br></div><div>thanks Monty for the feedback. I've started this set of patches in ironic [0] (very WiP currently), and would really like some eyes on them (not right now, as I plan to rewrite those basing on this conversation :) but soon I hope). Also I have some questions/comments inline:</div></div></div></div>
<br><div class="gmail_quote">On Thu, May 25, 2017 at 12:36 AM, Monty Taylor <span dir="ltr"><<a href="mailto:mordred@inaugust.com" target="_blank">mordred@inaugust.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><span class="gmail-m_-2688336320334480940gmail-">On 05/24/2017 12:51 PM, Eric Fried wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
Pavlo-<br>
<br>
There's a blueprint [1] whereby we're trying to address a bunch of<br>
these same concerns in nova. You can see the first part in action here<br>
[2]. However, it has become clear that nova is just one of the many<br>
services that would benefit from get_service_url(). With the full<br>
support of mordred (let's call it The Full Monty), we've got our sights<br>
on moving that method into ksa itself for that purpose.<br>
</blockquote>
<br></span>
Yes - this has started with documenting how to consume Keystone Catalog and discovery properly.<br>
<br>
<a href="https://review.openstack.org/#/q/topic:version-discovery" rel="noreferrer" target="_blank">https://review.openstack.org/#<wbr>/q/topic:version-discovery</a><br>
<br>
(it's a big stack)<br>
<br>
Once we're good with that - the next step is getting ksa updated to be able to handle the end-to-end. It does most of it today, but there are enough edgecases it doesn't that you wind up having to do something else, like efried just did in nova. The goal is to make that not necessary - and so that it's both possible and EASY for everyone to CORRECTLY consume catalog and version discovery.<br>
<br>
(more comments inline below)<div><div class="gmail-m_-2688336320334480940gmail-h5"><br>
<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
Please have a look at this blueprint and change set. Let us know if<br>
your concerns would be addressed if this were available to you from ksa.<br>
<br>
[1]<br>
<a href="https://specs.openstack.org/openstack/nova-specs/specs/pike/approved/use-service-catalog-for-endpoints.html" rel="noreferrer" target="_blank">https://specs.openstack.org/op<wbr>enstack/nova-specs/specs/pike/<wbr>approved/use-service-catalog-f<wbr>or-endpoints.html</a><br>
[2] <a href="https://review.openstack.org/#/c/458257/" rel="noreferrer" target="_blank">https://review.openstack.org/#<wbr>/c/458257/</a><br>
<br>
Thanks,<br>
efried<br>
<br>
On 05/24/2017 04:46 AM, Pavlo Shchelokovskyy wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
Hi all,<br>
<br>
There are several problems or inefficiencies in how we are dealing with<br>
auth to other services. Although it became much better in Newton, some<br>
things are still to be improved and I like to discuss how to tackle<br>
those and my ideas for that.<br>
<br>
Keystone endpoints<br>
===============<br>
<br>
Apparently since February-ish DevStack no longer sets up 'internal'<br>
endpoints for most of the services in core devstack [0].<br>
Luckily we were not broken by that right away - although when<br>
discovering a service endpoint from keystone catalog we default to<br>
'internal' endpoint [1], for most services our devstack plugin still<br>
configures explicit service URL in the corresponding config section, and<br>
thus the service discovery from keystone never takes place (or that code<br>
path is not tested by functional/integration testing).<br>
<br>
AFAIK different endpoint types (internal vs public) are still quite used<br>
by deployments (and IMO rightfully so), so we have to continue<br>
supporting that. I propose to take the following actions:<br>
</blockquote></blockquote>
<br></div></div>
I agree you should continue supporting it.<br>
<br>
I'm not sure it's important for you to change your defaults ... as long at it's possible to consistently set "interface=public" or "interface=internal" and have the results be correct, I think that's the big win.<span class="gmail-m_-2688336320334480940gmail-"><br>
<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
- in our devstack plugin, stop setting up the direct service URLs in<br>
config, always use keystone catalog for discovery<br>
</blockquote></blockquote>
<br></span>
YES<span class="gmail-m_-2688336320334480940gmail-"><br>
<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
- in every conf section related to external service add<br>
'endpoint_type=[internal|publi<wbr>c]' option, defaulting to 'internal', with<br>
a warning in option description (and validated on conductor start) that<br>
it will be changed to 'public' in the next release<br>
</blockquote></blockquote>
<br></span>
efried just added a call to keystoneauth which will register all of the appropriate CONF options that are needed to request a service endpoint from the catalog - register_adapter_conf_options:<br>
<br>
<a href="http://git.openstack.org/cgit/openstack/keystoneauth/tree/keystoneauth1/loading/__init__.py#n39" rel="noreferrer" target="_blank">http://git.openstack.org/cgit/<wbr>openstack/keystoneauth/tree/ke<wbr>ystoneauth1/loading/__init__.p<wbr>y#n39</a><br>
<br>
The word "adapter" in this case isn't directly important - but there are three general concepts in keystoneauth that relate to how you connect:<br>
<br>
auth<br>
- how you authenticate - auth_type, username, password, etc.<br>
session<br>
- how the transport layer connects - certs, timeouts, etc.<br>
adapter<br>
- what base endpoint to mount from the catalog - service_type, interface, endpoint_override, api_version</blockquote><div><br></div><div>Currently I'm trying to understand how to use adapters for creating clients. It seems that not all of clients of interest for me support this, as some ignore 'interface', 'endpoint_override' etc of the adapter instance if I'm passing it instead of a session, and always rely on the same/similar options passed to client. Are there any examples of how to use them? Also, some clients (e.g. a neutron [1]) already base their SessionClient on keystoneauth1.adapter.Adapter, so how could one create such client instance with adapter options loaded from config?</div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><span class="gmail-m_-2688336320334480940gmail-"><br>
<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
- use those values from CONF wherever we ask for service URL from<br>
catalog or instantiate client with session.<br>
</blockquote></blockquote>
<br></span>
YES<span class="gmail-m_-2688336320334480940gmail-"><br>
<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
- populate these options in our devstack plugin to be 'public'<br>
- in Queens, switch the default to 'public' and use defaults in devstack<br>
plugin, remove warnings.<br>
<br>
Unify clients creation<br>
================<br>
<br>
again, in those config sections related to service clients, we have many<br>
options to instantiate clients from (especially glance section, see my<br>
other recent ML about our image service code). Many of those seem to be<br>
from the time when keystone catalog was missing some functionality or<br>
not existing at all, and keystoneauth lib abstracting identity and<br>
client sessions was not there either.<br>
<br>
To simplify setup and unify as much code as possible I'd like to propose<br>
the following:<br>
<br>
- in each config section for service client add (if missing) a<br>
'<service>_url' option that should point to the API of given service and<br>
will be used *only in noauth mode* when there's no Keystone catalog to<br>
discover the service endpoint from<br>
</blockquote></blockquote>
<br></span>
I disagre with this one.<br>
<br>
The option exists and is called "endpoint_override" and it skips the catalog completely. It will get registered if you use register_adapter_conf_options. It's also useful even in auth scenarios sometimes.<br></blockquote><div><br></div><div>OK, will use endpoint_override instead, would indeed be more unified experience.</div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<br>
AND/OR ... (we could do both)<br>
<br>
I do this in os-client-config:<br>
<br>
<a href="http://git.openstack.org/cgit/openstack/os-client-config/tree/os_client_config/config.py#n857" rel="noreferrer" target="_blank">http://git.openstack.org/cgit/<wbr>openstack/os-client-config/tre<wbr>e/os_client_config/config.py#n<wbr>857</a><br>
<br>
to support ironic clients without keystone (yay bifrost!)<br>
<br>
However, that's a hack. Perhaps what we should do is add an auth plugin to keystoneauth called "none" or "noauth". It's the auth plugins that actually implement the auth_url support - so a noauth plugin that takes no arguments and does nothing - combined with the already existing endpoint_override argument could be a path forward that fits into the existing structure consistently?</blockquote><div><br></div><div>Yes please :) as this is exactly what I was looking for - the NoAuth auth_plugin. This would really simplify the handling of auth/noauth case in a single code base. But again, for now it seems I would need to resort to instantiating both session and adapter from config, and passing session and explicit adapter attributes to clients directly.</div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><span class="gmail-m_-2688336320334480940gmail-"><br>
<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
- in the code creating service clients, always create a keystoneauth<br>
session from config sections, using appropriate keystoneauth identity<br>
plugin - 'token_endpoint' with fake token <service>_url for noauth mode,<br>
'password' for service user client, 'token' when using a token from<br>
incoming request. The latter will have a benefit to make it possible for<br>
the session to reauth itself when user token is about to expire, but<br>
might require changes in some public methods to pass in the full<br>
task.context instead of just token<br>
</blockquote></blockquote>
<br></span>
Yes. There is actually a thing in keystoneauth called ServiceTokenAuthWrapper<br>
<br>
<a href="http://git.openstack.org/cgit/openstack/keystoneauth/tree/keystoneauth1/service_token.py" rel="noreferrer" target="_blank">http://git.openstack.org/cgit/<wbr>openstack/keystoneauth/tree/ke<wbr>ystoneauth1/service_token.py</a><br>
<br>
which is used in Nova for cases where that is important.</blockquote><div><br></div><div>This is another point I'd have to wrap my head around. Where is the user_auth coming from in ServiceToken? Can we somehow instantiate a user_auth from oslo_context.RequestContext or its subclasses? Preferably by some base method of RequestContext or from keystoneauth loading. I see an example in Nova, but am still puzzled about user_auth.</div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><span class="gmail-m_-2688336320334480940gmail-"><br>
<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
- always create clients from sessions. Although AFAIK all clients ironic<br>
uses already support this, some in ironic code (e.g. glance) still<br>
always create a client from token and endpoint directly.<br>
</blockquote></blockquote>
<br></span>
YES DEAR GOD YES this is very important.<span class="gmail-m_-2688336320334480940gmail-"><br>
<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
- deprecate some options explicitly registered by ironic in those<br>
sections that are becoming redundant - including those that relate to<br>
HTTP session settings (like timeout, retries, SSL certs and settings) as<br>
those will be used from options registered by keystoneauth Session, and<br>
those multiple options that piece together a single service URL.<br>
</blockquote></blockquote>
<br></span>
Yes. this is very great. As I mentioned earlier, "session" options from ksa cover this- so register_session_conf_options will take care of these.<span class="gmail-m_-2688336320334480940gmail-"><br>
<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
This will decrease the complexity of service client-related code and<br>
will make configuring those cleaner.<br>
<br>
Of course all of this has to be done minding proper deprecation process,<br>
although that might complicate things (as usual :/).<br>
</blockquote></blockquote>
<br></span>
Nice things are never easy. :)<span class="gmail-m_-2688336320334480940gmail-"><br>
<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
Legacy auth<br>
=========<br>
<br>
Probably not worth specific mention, but we implemented a proper<br>
keystoneauth-based loading of client auth options back in Newton almost<br>
a year ago, so the code attempting to load auth for clients in a<br>
deprecated way from "[keystone_authtoken]" section can be safely removed<br>
already.<br>
<br>
As always, I'm eager to hear your comments.<br>
</blockquote></blockquote>
<br></span>
Thank you for writing this up - it's very well timed with the other work going on related to catalog and version discovery. Please let me know if I can help in any way or if anything is unclear.<div class="gmail-m_-2688336320334480940gmail-HOEnZb"><div class="gmail-m_-2688336320334480940gmail-h5"><br>
<br>
______________________________<wbr>______________________________<wbr>______________<br>
OpenStack Development Mailing List (not for usage questions)<br>
Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank">OpenStack-dev-request@lists.op<wbr>enstack.org?subject:unsubscrib<wbr>e</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" target="_blank">http://lists.openstack.org/cgi<wbr>-bin/mailman/listinfo/openstac<wbr>k-dev</a></div></div></blockquote><div><br></div><div><br></div><div>[0] <a href="https://review.openstack.org/#/q/status:open+project:openstack/ironic+branch:master+topic:rework-auth">https://review.openstack.org/#/q/status:open+project:openstack/ironic+branch:master+topic:rework-auth</a></div><div>[1] <a href="http://git.openstack.org/cgit/openstack/python-neutronclient/tree/neutronclient/client.py#n306">http://git.openstack.org/cgit/openstack/python-neutronclient/tree/neutronclient/client.py#n306</a></div><div><br></div><div>Cheers, </div></div><div class="gmail_extra"><br></div>Dr. Pavlo Shchelokovskyy<div>Senior Software Engineer</div><div>Mirantis Inc</div><div><a href="http://www.mirantis.com/" target="_blank">www.mirantis.com</a></div></div></div>