Open Stack

On 10/19/2016 3:22 PM, Matt Riedemann wrote:
> On 10/19/2016 2:27 PM, Brian Curtin wrote:
>> On Wed, Oct 19, 2016 at 2:03 PM, Sean Dague <sean at dague.net> wrote:
>>> On 10/19/2016 01:40 PM, Brian Curtin wrote:
>>>> On Wed, Oct 19, 2016 at 12:59 PM, Jay Pipes <jaypipes at gmail.com> wrote:
>>>>> On 10/19/2016 05:32 PM, Brian Curtin wrote:
>>>>>>
>>>>>> I'm currently facing what looks more and more like an impossible
>>>>>> problem in determining the root of each service on a given cloud. It
>>>>>> is apparently a free-for-all in how endpoints can be structured,
>>>>>> and I
>>>>>> think we're out of ways to approach it that catch all of the ways
>>>>>> that
>>>>>> all people can think of.
>>>>>>
>>>>>> In openstacksdk, we can no longer use the service catalog for
>>>>>> determining each service's endpoints. Among other things, this is due
>>>>>> to a combination of some versions of some services not actually being
>>>>>> listed, and with things heading the direction of version-less
>>>>>> services
>>>>>> anyway. Recently we changed to using the service catalog as a pointer
>>>>>> to where services live and then try to find the root of that service
>>>>>> by stripping the path down and making some extra requests on startup
>>>>>> to find what's offered. Despite a few initial snags, this now works
>>>>>> reasonably well in a majority of cases.
>>>>>>
>>>>>> We have seen endpoints structured in the following ways:
>>>>>>  A. subdomains, e.g., https://service.cloud.com/v2
>>>>>>  B. paths, e.g., https://cloud.com/service/v2 (sometimes there are
>>>>>> more paths in between the root and /service/)
>>>>>>  C. service-specific ports, e.g., https://cloud.com:1234/v2
>>>>>>  D. both A and B plus ports
>>>>>>
>>>>>> Within all of these, we can find the root of the given service just
>>>>>> fine. We split the path and build successively longer paths starting
>>>>>> from the root. In the above examples, we need to hit the path just
>>>>>> short of the /v2, so in B it actually takes two requests as we'd make
>>>>>> one to cloud.com which fails, but then a second one to
>>>>>> cloud.com/service gives us what we need.
>>>>>>
>>>>>> However, another case came up: the root of all endpoints is itself
>>>>>> another service. That makes it look like this:
>>>>>>
>>>>>>  E. https://cloud.com:9999/service/v2
>>>>>>  F. https://cloud.com:9999/otherservice
>>>>>>
>>>>>> In this case, https://cloud.com:9999 is keystone, so trying to get
>>>>>> E's
>>>>>> base by going from the root and outward will give me a versions
>>>>>> response I can parse properly, but it points to keystone. We then end
>>>>>> up building requests for 'service' that go to keystone endpoints and
>>>>>> end up failing. We're doing this using itertools.accumulate on the
>>>>>> path fragments, so you might think 'just throw it through
>>>>>> `reversed()`' and go the other way. If we do that, we'll also get a
>>>>>> versions response that we can parse, but it's the v2 specific info,
>>>>>> not all available versions.
>>>>>>
>>>>>> So now that we can't reliably go from the left, and we definitely
>>>>>> can't go from the right, how about the middle?
>>>>>>
>>>>>> This sounds ridiculous, and if it sounds familiar it's because they
>>>>>> devise a "middle out" algorithm on the show Silicon Valley, but in
>>>>>> most cases it'd actually work. In E above, it'd be fine. However,
>>>>>> depending on the number of path fragments and which direction we
>>>>>> chose
>>>>>> to move first, we'd sometimes hit either a version-specific response
>>>>>> or another service's response, so it's not reliable.
>>>>>>
>>>>>> Ultimately, I would like to know how something like this can be
>>>>>> solved.
>>>>>>
>>>>>> 1. Is there any reliable, functional, and accurate programmatic
>>>>>> way to
>>>>>> get the versions and endpoints that all services on a cloud offer?
>>>>>
>>>>>
>>>>> The Keystone service catalog should be the thing that provides the
>>>>> endpoints
>>>>> for all services in the cloud. Within each service, determining the
>>>>> (micro)version of the API is unfortunately going to be a per-service
>>>>> endeavour. For some APIs, a microversion header is returned, others
>>>>> don't
>>>>> have microversions. The microversion header is unfortunately not
>>>>> standardized for all APIs that use microversions, though a number
>>>>> of us
>>>>> would like to see a single:
>>>>>
>>>>> OpenStack-API-Version: <service-type> <microversion>, ...
>>>>>
>>>>> header supported. This is the header supported in the new placement
>>>>> REST
>>>>> API, for what it's worth.
>>>>
>>>> I get the microversion part, and we support that (for some degree of
>>>> support), but this is about the higher level major versions. The
>>>> example that started this was Keystone only listing a v2 endpoint in
>>>> the service catalog, at least on devstack. I need to be able to hit v3
>>>> APIs when a user wants to do v3 things, regardless of which version
>>>> they auth to, so the way to get it was to get the root and go from
>>>> there. That both versions weren't listed was initially confusing to
>>>> me, but that's where the suggestion of "go to the root and get
>>>> everything" started out.
>>>>
>>>> The service catalog holding providing all of the available endpoints
>>>> made sense to me from what I understood in the past, but two things
>>>> are for sure about this: it doesn't work that way, and I've been told
>>>> several times that it's not going to work that way even in cases where
>>>> it is apparently working. I don't have sources to cite, but it's come
>>>> up a few times that the goal is one entry per service and you talk to
>>>> the service to find out all of its details - major versions, micro
>>>> versions, etc.
>>>>
>>>>>> 2. Are there any guidelines, rules, expectations, or other
>>>>>> documentation on how services can be installed and their endpoints
>>>>>> structured that are helpful to people build apps that use them,
>>>>>> not in
>>>>>> those trying to install and operate them? I've looked around a few
>>>>>> times and found nothing useful. A lot of what I've found has
>>>>>> referenced suggestions for operators setting them up behind various
>>>>>> load balancing tools.
>>>>>
>>>>>
>>>>> I presume you are referring to the "internal" vs "public" endpoint
>>>>> stuff? If
>>>>> so, my preference has been that such "internal vs. external"
>>>>> routing should
>>>>> be handled via the Keystone service catalog returning a set of
>>>>> endpoints
>>>>> depending on the source (or X-forwarded-for) IP. So, requests from
>>>>> "internal" networks (for whatever definition of "internal" you
>>>>> want) return
>>>>> a set of endpoint URLs reflecting the "internal" endpoints.
>>>>
>>>> This is more about structuring of the URLs in terms of forming a root,
>>>> port, paths, etc, like the examples A-F. Are those all of the forms
>>>> that could be expected? Are there others? Are any of those not
>>>> actually supported? So far it's been a lot of guesswork that started
>>>> with being built to work with devstack (so service-per-port), but then
>>>> people have come up and said the assumption we made doesn't work with
>>>> their cloud, so we've broadened support to catch the various cases.
>>>> That's still ongoing as cases are coming up.
>>>>
>>>> I'm going to guess that operators are allowed to structure endpoints
>>>> in whatever way suits their needs and that there's no "your endpoints
>>>> should be formed like this..." document. If that's how it is, that's
>>>> fine, but that'll mean something—service catalog, an API, a new
>>>> project, whatever—is going to have to give me the roots of each
>>>> service.
>>>
>>> We should really unwind where you got that information from. The point
>>> of the service catalog is to be exactly what you want it to be. If you,
>>> as a writer of consuming software, have found problems with it's schema,
>>> or the way it's commonly implemented, that make it fail for that
>>> purpose, please be specific about those.
>>>
>>> For instance, there was a long standing feeling that major versions
>>> should not be listed in the catalog. However, that was mostly from a
>>> purity perspective than a real world one. And at the end of the day we
>>> *definitely don't* want to make every SDK build a heuristic for guessing
>>> where endpoints are. If listing major versions is key, lets take that as
>>> feedback and make the schema changes to do that.
>>>
>>> We had a bit of an effort to work on this 2 cycles ago, but it got back
>>> burnered in getting the api-ref work sorted (which is sort of a prereq),
>>> but as that is finalizing this is something we can and should jump back
>>> into.
>>
>> This started back in August when it came up that we didn't know where
>> that Keystone v3 endpoint was. After talking with a few people, Steve
>> Martinelli mentioned that at least as of then, hitting the unversioned
>> endpoint was the way to solve that. It being unlisted anywhere was
>> something for me to figure out (via that path manipulation from the
>> given v2), but it was later mentioned that ideally they would like to
>> have unversioned endpoints in the catalog anyway. I'm talking to Steve
>> now and perhaps I took that too far in extrapolating which direction
>> things were going in reality, but it was a solution that had to be
>> undertaken nonetheless and was seen as the best way forward at the
>> time. It's also the only one that mostly works at the moment.
>>
>> In the end, I'll take listing major versions as long as it's accurate
>> and complete, but I'll also take listing the service root even if it
>> means an extra request for me to determine those versions.
>>
>> __________________________________________________________________________
>>
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe:
>> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>
> I personally thought long-term we wanted unversioned endpoints in the
> service catalog, and if you want to do version discovery, you do a GET
> on a particular service's endpoint URL in the service catalog, and that
> returns the list of available API versions for that service along with a
> status value (CURRENT, SUPPORTED, DEPRECATED) and any microversion
> ranges within those.
>
> I know we have 3 versions of keystone in the service catalog, which I
> find pretty nasty personally, plus the fact that a lot of the client
> code we have has had to burn 'volumev2' in as a default endpoint_type to
> use cinder. IMO the service catalog should just have a 'volume' endpoint
> and if I want to know the supported versions (v1, v2 and/or v3), I do a
> GET on the publicURL for the volume endpoint from the service catalog.
>

Sorry, correction, I know we have 3 versions of *cinder* in the service 
catalog...

-- 

Thanks,

Matt Riedemann