Open Stack

On 08/04/2017 04:03 PM, Dean Troyer wrote:
> On Fri, Aug 4, 2017 at 11:52 AM, Monty Taylor <mordred at inaugust.com> wrote:
> [...]
>> * Both shade and python-openstackclient have investigated using openstacksdk
>> as their REST layer but were unable to because it puts some abstractions in
>> layers that make it impossible to do some of the advanced things we need.
> 
> OSC _does_ use the SDK for all Network API commands.  That was a
> mistake in the sense that we did it before SDK 1.0 was released and
> still do not have 1.0.

Yah - although it got you out of the competing cliff issue, so that was 
good.

>> * openstacksdk has internal implementations of things that exist at
>> different points in the stack. We just added full support for version
>> service and version discovery to keystoneauth, but openstacksdk has its own
>> layer for that so it both can't use the ksa implementation and is not
>> compliant with the API-WG consume guidelines.
> 
> This was intended to make it free from dependencies, when first
> concieved, none of these other libs existed.

Oh - totally. It made total sense at the time ... it's just the 
surrounding context has changed.

> I am coming around to believe that we need to slice a couple more
> things up so they can be composed as needed rather then bite off the
> entire thing in once piece.
> 
> [...]
> 
>> I'd propose we have the shade team adopt the python-openstacksdk codebase.
> 
> ++
> 
>> This is obviously an aggressive suggestion and essentially represents a
>> takeover of a project. We don't have the luxury of humans to work on things
>> that we once had, so I think as a community we should be realistic about the
>> benefits of consolidation and the downside to continuing to have 2 different
>> python SDKs.
> 
> ++
> 
> I thought it would be natural for OSC to adopt the SDK someday if
> Brian did not get around to making it official, but the current
> circumstances make it clear that we (OSC) do not have the resources to
> do this.  This proposal is much better and leads to a natural
> coalescence of the parallel goals and projects.
> 
>> Doing that implies the following:
>>
>> * Rework the underlying guts of openstacksdk to make it possible to replace
>> shade's REST layer with openstacksdk. openstacksdk still doesn't have a 1.0
>> release, so we can break the few things we'll need to break.
> 
> Sigh.  OSC has been using the Network components of the SDK for a long
> time in spite of SDK not being at 1.0.  In retrospect that was a
> mistake on my part but I believed at the time that 1.0 was close and
> we had already ignored Network for far too long.  We already have one
> compatibility layer in the SDK due to the proxy refactor work that was
> supposed to be the last thing before 1.0.

I don't think we need to break the things you're using, fwiw. In fact, 
we can probably take it as a first-order requirement to not break OSC - 
unless we find something SUPER intractable - and even then we should 
talk about it first.

> [...]
> 
>> * Merge the two repos and retire one of them. Specifics on the mechanics of
>> this below, but this will either result in moving the resource and service
>> layer in openstacksdk into shade and adding appropriate attributes to the
>> shade.OpenStackCloud object, or moving the shade.OpenStackCloud into
>> something like openstack.cloud and making a shade backwards-compate shim. I
>> lean towards the first, as we've been telling devs "use shade to talk to
>> OpenStack" at hackathons and bootcamps and I'd rather avoid the messaging
>> shift. However, pointing to an SDK called "The Python OpenStack SDK" and
>> telling people to use it certainly has its benefits from a messaging
>> perspective.
> 
> I don't have a big concern about which repo is maintained.  For OSC I
> want what amounts to a low-level REST API, one example of which can be
> found in openstackclient.api.*.  Currently Shade is not quite the
> right thing to back a CLI but now has the layer I want, and SDK does
> not have that layer at all (it was proposed very early on and not
> merged).

FWIW - now that we landed version discovery and microversion support in 
keystoneauth - I'd say keystoneauth Adapter is actually now the 
low-level REST API:

   client = keystoneauth1.adapter.Adapter(
     session=session,
     service_type='compute',
     region_name='DFW',
     min_version='2',
     max_version='2.latest')
   endpoint_data = client.get_endpoint_data()
   print(
     "Microversion range is: ",
     endpoint_data.min_microversion,
     endpoint_data.max_microversion)
   # want 2.31 of servers
   server_response = client.get('/servers', microversion='2.31')
   server = server_response.json()['servers'][0]
   # want 2.14 of keypairs
   keypair_response = client.get('/keypairs', microversion='2.14')
   # Don't care on volume attachments - don't specify one.
   volume_attachments_response = client.get(
       '/servers/{id}/os-volume_attachments'.format(id=server['id']))

> Is it better to have a single monolithic thing that has three
> conceptual layers internally that can be individually consumed or to
> have three things that get layered as needed?

It's a good question. I'm leaning to single repo for shade and sdk 
mainly because since both do direct REST and dont' have other 
dependencies, there's not really much difference from a composability 
standpoint. If you just want the low-level SDK functions, use them. If 
you want the hide-the-difference functions, use them - doesn't gain much 
to have them split.

That said - if merging them is too much work, it's not super IMPORTANT 
to merge them either.

>> * drop keystoneauth.Session subclass. It's over-riding things at the wrong
>> layer. keystoneauth Adapter is the thing it wants to be.
> 
> FWIW, OSC has this problem too...

Yah - I'm totally sorry I forgot to try to get your timing wrapper in to 
ksa Session this pass so that you could drop that wrapper...

>> * suitability for python-openstackclient. Dean and Steve have been laying in
>> the groundwork for doing direct-REST in python-openstackclient because
>> python-*client are a mess from an end-user perspective and openstacksdk
>> isn't suitable. If we can sync on requirements hopefully we can produce
>> something that python-openstackclient can honestly use for that layer
>> instead of needing local code.
> 
> As I mentioned, we already use the Networking portions of SDK, even
> without a 1.0, and it has bit us already a couple of times.  It has
> long been my plan to convert to using the SDK, but that was when I
> believed there would also be a lower-level API exposed that did not
> require all of the application-level goodness and abstractions.
> 
> I personally feel like splitting out the low-level REST API layers
> into a stand-alone piece that shade, SDK and OSC can all benefit from
> would be our best course, but then I have been wrong about this
> layering thing in the past, so I throw it out there to have something
> that can be used to push against to get what everyone else seems to
> want.

Nod. I think that's ksa. We're doing almost nothing at the REST layer in 
shade at this point - we're just using ksa Adapter. We have a wrapper 
around it for two reasons:

- TaskManager support for nodepool
- translating ksa REST exceptions to shade exceptions for backwards compat

Then we do data normalization - but that's a 'hide cloud differences' 
thing, not a thing that should be at that lower level.

rods actually just finished removing one additional thing we were doing, 
which was stripping the top-level key from the results if there was one. 
Turns out doing that really screws with pagination. whoops.

In any case, I agree with the need for a low-level REST interface. 
Mostly since last we talked about it the missing pieces have hit ksa.

Looking at openstackclient.api.* though - we may be meaning different 
things when we say "low level REST interface".

 From the shade/sdk perspective, the extra metadata that's tracked for 
each Resource object will, I _believe_ make it easier to add basic new 
CRUD for each resource type. I could certainly be VERY wrong about that 
though.