[openstack-dev] Endpoint structure: a free-for-all

Brian Curtin brian at python.org
Wed Oct 19 19:27:57 UTC 2016


On Wed, Oct 19, 2016 at 2:03 PM, Sean Dague <sean at dague.net> wrote:
> On 10/19/2016 01:40 PM, Brian Curtin wrote:
>> On Wed, Oct 19, 2016 at 12:59 PM, Jay Pipes <jaypipes at gmail.com> wrote:
>>> On 10/19/2016 05:32 PM, Brian Curtin wrote:
>>>>
>>>> I'm currently facing what looks more and more like an impossible
>>>> problem in determining the root of each service on a given cloud. It
>>>> is apparently a free-for-all in how endpoints can be structured, and I
>>>> think we're out of ways to approach it that catch all of the ways that
>>>> all people can think of.
>>>>
>>>> In openstacksdk, we can no longer use the service catalog for
>>>> determining each service's endpoints. Among other things, this is due
>>>> to a combination of some versions of some services not actually being
>>>> listed, and with things heading the direction of version-less services
>>>> anyway. Recently we changed to using the service catalog as a pointer
>>>> to where services live and then try to find the root of that service
>>>> by stripping the path down and making some extra requests on startup
>>>> to find what's offered. Despite a few initial snags, this now works
>>>> reasonably well in a majority of cases.
>>>>
>>>> We have seen endpoints structured in the following ways:
>>>>  A. subdomains, e.g., https://service.cloud.com/v2
>>>>  B. paths, e.g., https://cloud.com/service/v2 (sometimes there are
>>>> more paths in between the root and /service/)
>>>>  C. service-specific ports, e.g., https://cloud.com:1234/v2
>>>>  D. both A and B plus ports
>>>>
>>>> Within all of these, we can find the root of the given service just
>>>> fine. We split the path and build successively longer paths starting
>>>> from the root. In the above examples, we need to hit the path just
>>>> short of the /v2, so in B it actually takes two requests as we'd make
>>>> one to cloud.com which fails, but then a second one to
>>>> cloud.com/service gives us what we need.
>>>>
>>>> However, another case came up: the root of all endpoints is itself
>>>> another service. That makes it look like this:
>>>>
>>>>  E. https://cloud.com:9999/service/v2
>>>>  F. https://cloud.com:9999/otherservice
>>>>
>>>> In this case, https://cloud.com:9999 is keystone, so trying to get E's
>>>> base by going from the root and outward will give me a versions
>>>> response I can parse properly, but it points to keystone. We then end
>>>> up building requests for 'service' that go to keystone endpoints and
>>>> end up failing. We're doing this using itertools.accumulate on the
>>>> path fragments, so you might think 'just throw it through
>>>> `reversed()`' and go the other way. If we do that, we'll also get a
>>>> versions response that we can parse, but it's the v2 specific info,
>>>> not all available versions.
>>>>
>>>> So now that we can't reliably go from the left, and we definitely
>>>> can't go from the right, how about the middle?
>>>>
>>>> This sounds ridiculous, and if it sounds familiar it's because they
>>>> devise a "middle out" algorithm on the show Silicon Valley, but in
>>>> most cases it'd actually work. In E above, it'd be fine. However,
>>>> depending on the number of path fragments and which direction we chose
>>>> to move first, we'd sometimes hit either a version-specific response
>>>> or another service's response, so it's not reliable.
>>>>
>>>> Ultimately, I would like to know how something like this can be solved.
>>>>
>>>> 1. Is there any reliable, functional, and accurate programmatic way to
>>>> get the versions and endpoints that all services on a cloud offer?
>>>
>>>
>>> The Keystone service catalog should be the thing that provides the endpoints
>>> for all services in the cloud. Within each service, determining the
>>> (micro)version of the API is unfortunately going to be a per-service
>>> endeavour. For some APIs, a microversion header is returned, others don't
>>> have microversions. The microversion header is unfortunately not
>>> standardized for all APIs that use microversions, though a number of us
>>> would like to see a single:
>>>
>>> OpenStack-API-Version: <service-type> <microversion>, ...
>>>
>>> header supported. This is the header supported in the new placement REST
>>> API, for what it's worth.
>>
>> I get the microversion part, and we support that (for some degree of
>> support), but this is about the higher level major versions. The
>> example that started this was Keystone only listing a v2 endpoint in
>> the service catalog, at least on devstack. I need to be able to hit v3
>> APIs when a user wants to do v3 things, regardless of which version
>> they auth to, so the way to get it was to get the root and go from
>> there. That both versions weren't listed was initially confusing to
>> me, but that's where the suggestion of "go to the root and get
>> everything" started out.
>>
>> The service catalog holding providing all of the available endpoints
>> made sense to me from what I understood in the past, but two things
>> are for sure about this: it doesn't work that way, and I've been told
>> several times that it's not going to work that way even in cases where
>> it is apparently working. I don't have sources to cite, but it's come
>> up a few times that the goal is one entry per service and you talk to
>> the service to find out all of its details - major versions, micro
>> versions, etc.
>>
>>>> 2. Are there any guidelines, rules, expectations, or other
>>>> documentation on how services can be installed and their endpoints
>>>> structured that are helpful to people build apps that use them, not in
>>>> those trying to install and operate them? I've looked around a few
>>>> times and found nothing useful. A lot of what I've found has
>>>> referenced suggestions for operators setting them up behind various
>>>> load balancing tools.
>>>
>>>
>>> I presume you are referring to the "internal" vs "public" endpoint stuff? If
>>> so, my preference has been that such "internal vs. external" routing should
>>> be handled via the Keystone service catalog returning a set of endpoints
>>> depending on the source (or X-forwarded-for) IP. So, requests from
>>> "internal" networks (for whatever definition of "internal" you want) return
>>> a set of endpoint URLs reflecting the "internal" endpoints.
>>
>> This is more about structuring of the URLs in terms of forming a root,
>> port, paths, etc, like the examples A-F. Are those all of the forms
>> that could be expected? Are there others? Are any of those not
>> actually supported? So far it's been a lot of guesswork that started
>> with being built to work with devstack (so service-per-port), but then
>> people have come up and said the assumption we made doesn't work with
>> their cloud, so we've broadened support to catch the various cases.
>> That's still ongoing as cases are coming up.
>>
>> I'm going to guess that operators are allowed to structure endpoints
>> in whatever way suits their needs and that there's no "your endpoints
>> should be formed like this..." document. If that's how it is, that's
>> fine, but that'll mean something—service catalog, an API, a new
>> project, whatever—is going to have to give me the roots of each
>> service.
>
> We should really unwind where you got that information from. The point
> of the service catalog is to be exactly what you want it to be. If you,
> as a writer of consuming software, have found problems with it's schema,
> or the way it's commonly implemented, that make it fail for that
> purpose, please be specific about those.
>
> For instance, there was a long standing feeling that major versions
> should not be listed in the catalog. However, that was mostly from a
> purity perspective than a real world one. And at the end of the day we
> *definitely don't* want to make every SDK build a heuristic for guessing
> where endpoints are. If listing major versions is key, lets take that as
> feedback and make the schema changes to do that.
>
> We had a bit of an effort to work on this 2 cycles ago, but it got back
> burnered in getting the api-ref work sorted (which is sort of a prereq),
> but as that is finalizing this is something we can and should jump back
> into.

This started back in August when it came up that we didn't know where
that Keystone v3 endpoint was. After talking with a few people, Steve
Martinelli mentioned that at least as of then, hitting the unversioned
endpoint was the way to solve that. It being unlisted anywhere was
something for me to figure out (via that path manipulation from the
given v2), but it was later mentioned that ideally they would like to
have unversioned endpoints in the catalog anyway. I'm talking to Steve
now and perhaps I took that too far in extrapolating which direction
things were going in reality, but it was a solution that had to be
undertaken nonetheless and was seen as the best way forward at the
time. It's also the only one that mostly works at the moment.

In the end, I'll take listing major versions as long as it's accurate
and complete, but I'll also take listing the service root even if it
means an extra request for me to determine those versions.



More information about the OpenStack-dev mailing list