Open Stack

On 10/19/2016 2:27 PM, Brian Curtin wrote:
> On Wed, Oct 19, 2016 at 2:03 PM, Sean Dague <sean at dague.net> wrote:
>> On 10/19/2016 01:40 PM, Brian Curtin wrote:
>>> On Wed, Oct 19, 2016 at 12:59 PM, Jay Pipes <jaypipes at gmail.com> wrote:
>>>> On 10/19/2016 05:32 PM, Brian Curtin wrote:
>>>>>
>>>>> I'm currently facing what looks more and more like an impossible
>>>>> problem in determining the root of each service on a given cloud. It
>>>>> is apparently a free-for-all in how endpoints can be structured, and I
>>>>> think we're out of ways to approach it that catch all of the ways that
>>>>> all people can think of.
>>>>>
>>>>> In openstacksdk, we can no longer use the service catalog for
>>>>> determining each service's endpoints. Among other things, this is due
>>>>> to a combination of some versions of some services not actually being
>>>>> listed, and with things heading the direction of version-less services
>>>>> anyway. Recently we changed to using the service catalog as a pointer
>>>>> to where services live and then try to find the root of that service
>>>>> by stripping the path down and making some extra requests on startup
>>>>> to find what's offered. Despite a few initial snags, this now works
>>>>> reasonably well in a majority of cases.
>>>>>
>>>>> We have seen endpoints structured in the following ways:
>>>>>  A. subdomains, e.g., https://service.cloud.com/v2
>>>>>  B. paths, e.g., https://cloud.com/service/v2 (sometimes there are
>>>>> more paths in between the root and /service/)
>>>>>  C. service-specific ports, e.g., https://cloud.com:1234/v2
>>>>>  D. both A and B plus ports
>>>>>
>>>>> Within all of these, we can find the root of the given service just
>>>>> fine. We split the path and build successively longer paths starting
>>>>> from the root. In the above examples, we need to hit the path just
>>>>> short of the /v2, so in B it actually takes two requests as we'd make
>>>>> one to cloud.com which fails, but then a second one to
>>>>> cloud.com/service gives us what we need.
>>>>>
>>>>> However, another case came up: the root of all endpoints is itself
>>>>> another service. That makes it look like this:
>>>>>
>>>>>  E. https://cloud.com:9999/service/v2
>>>>>  F. https://cloud.com:9999/otherservice
>>>>>
>>>>> In this case, https://cloud.com:9999 is keystone, so trying to get E's
>>>>> base by going from the root and outward will give me a versions
>>>>> response I can parse properly, but it points to keystone. We then end
>>>>> up building requests for 'service' that go to keystone endpoints and
>>>>> end up failing. We're doing this using itertools.accumulate on the
>>>>> path fragments, so you might think 'just throw it through
>>>>> `reversed()`' and go the other way. If we do that, we'll also get a
>>>>> versions response that we can parse, but it's the v2 specific info,
>>>>> not all available versions.
>>>>>
>>>>> So now that we can't reliably go from the left, and we definitely
>>>>> can't go from the right, how about the middle?
>>>>>
>>>>> This sounds ridiculous, and if it sounds familiar it's because they
>>>>> devise a "middle out" algorithm on the show Silicon Valley, but in
>>>>> most cases it'd actually work. In E above, it'd be fine. However,
>>>>> depending on the number of path fragments and which direction we chose
>>>>> to move first, we'd sometimes hit either a version-specific response
>>>>> or another service's response, so it's not reliable.
>>>>>
>>>>> Ultimately, I would like to know how something like this can be solved.
>>>>>
>>>>> 1. Is there any reliable, functional, and accurate programmatic way to
>>>>> get the versions and endpoints that all services on a cloud offer?
>>>>
>>>>
>>>> The Keystone service catalog should be the thing that provides the endpoints
>>>> for all services in the cloud. Within each service, determining the
>>>> (micro)version of the API is unfortunately going to be a per-service
>>>> endeavour. For some APIs, a microversion header is returned, others don't
>>>> have microversions. The microversion header is unfortunately not
>>>> standardized for all APIs that use microversions, though a number of us
>>>> would like to see a single:
>>>>
>>>> OpenStack-API-Version: <service-type> <microversion>, ...
>>>>
>>>> header supported. This is the header supported in the new placement REST
>>>> API, for what it's worth.
>>>
>>> I get the microversion part, and we support that (for some degree of
>>> support), but this is about the higher level major versions. The
>>> example that started this was Keystone only listing a v2 endpoint in
>>> the service catalog, at least on devstack. I need to be able to hit v3
>>> APIs when a user wants to do v3 things, regardless of which version
>>> they auth to, so the way to get it was to get the root and go from
>>> there. That both versions weren't listed was initially confusing to
>>> me, but that's where the suggestion of "go to the root and get
>>> everything" started out.
>>>
>>> The service catalog holding providing all of the available endpoints
>>> made sense to me from what I understood in the past, but two things
>>> are for sure about this: it doesn't work that way, and I've been told
>>> several times that it's not going to work that way even in cases where
>>> it is apparently working. I don't have sources to cite, but it's come
>>> up a few times that the goal is one entry per service and you talk to
>>> the service to find out all of its details - major versions, micro
>>> versions, etc.
>>>
>>>>> 2. Are there any guidelines, rules, expectations, or other
>>>>> documentation on how services can be installed and their endpoints
>>>>> structured that are helpful to people build apps that use them, not in
>>>>> those trying to install and operate them? I've looked around a few
>>>>> times and found nothing useful. A lot of what I've found has
>>>>> referenced suggestions for operators setting them up behind various
>>>>> load balancing tools.
>>>>
>>>>
>>>> I presume you are referring to the "internal" vs "public" endpoint stuff? If
>>>> so, my preference has been that such "internal vs. external" routing should
>>>> be handled via the Keystone service catalog returning a set of endpoints
>>>> depending on the source (or X-forwarded-for) IP. So, requests from
>>>> "internal" networks (for whatever definition of "internal" you want) return
>>>> a set of endpoint URLs reflecting the "internal" endpoints.
>>>
>>> This is more about structuring of the URLs in terms of forming a root,
>>> port, paths, etc, like the examples A-F. Are those all of the forms
>>> that could be expected? Are there others? Are any of those not
>>> actually supported? So far it's been a lot of guesswork that started
>>> with being built to work with devstack (so service-per-port), but then
>>> people have come up and said the assumption we made doesn't work with
>>> their cloud, so we've broadened support to catch the various cases.
>>> That's still ongoing as cases are coming up.
>>>
>>> I'm going to guess that operators are allowed to structure endpoints
>>> in whatever way suits their needs and that there's no "your endpoints
>>> should be formed like this..." document. If that's how it is, that's
>>> fine, but that'll mean something—service catalog, an API, a new
>>> project, whatever—is going to have to give me the roots of each
>>> service.
>>
>> We should really unwind where you got that information from. The point
>> of the service catalog is to be exactly what you want it to be. If you,
>> as a writer of consuming software, have found problems with it's schema,
>> or the way it's commonly implemented, that make it fail for that
>> purpose, please be specific about those.
>>
>> For instance, there was a long standing feeling that major versions
>> should not be listed in the catalog. However, that was mostly from a
>> purity perspective than a real world one. And at the end of the day we
>> *definitely don't* want to make every SDK build a heuristic for guessing
>> where endpoints are. If listing major versions is key, lets take that as
>> feedback and make the schema changes to do that.
>>
>> We had a bit of an effort to work on this 2 cycles ago, but it got back
>> burnered in getting the api-ref work sorted (which is sort of a prereq),
>> but as that is finalizing this is something we can and should jump back
>> into.
>
> This started back in August when it came up that we didn't know where
> that Keystone v3 endpoint was. After talking with a few people, Steve
> Martinelli mentioned that at least as of then, hitting the unversioned
> endpoint was the way to solve that. It being unlisted anywhere was
> something for me to figure out (via that path manipulation from the
> given v2), but it was later mentioned that ideally they would like to
> have unversioned endpoints in the catalog anyway. I'm talking to Steve
> now and perhaps I took that too far in extrapolating which direction
> things were going in reality, but it was a solution that had to be
> undertaken nonetheless and was seen as the best way forward at the
> time. It's also the only one that mostly works at the moment.
>
> In the end, I'll take listing major versions as long as it's accurate
> and complete, but I'll also take listing the service root even if it
> means an extra request for me to determine those versions.
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>

I personally thought long-term we wanted unversioned endpoints in the 
service catalog, and if you want to do version discovery, you do a GET 
on a particular service's endpoint URL in the service catalog, and that 
returns the list of available API versions for that service along with a 
status value (CURRENT, SUPPORTED, DEPRECATED) and any microversion 
ranges within those.

I know we have 3 versions of keystone in the service catalog, which I 
find pretty nasty personally, plus the fact that a lot of the client 
code we have has had to burn 'volumev2' in as a default endpoint_type to 
use cinder. IMO the service catalog should just have a 'volume' endpoint 
and if I want to know the supported versions (v1, v2 and/or v3), I do a 
GET on the publicURL for the volume endpoint from the service catalog.

-- 

Thanks,

Matt Riedemann