Open Stack

On 08/04/2017 03:24 AM, Thierry Carrez wrote:
> Michael Johnson wrote:
>> I was wondering what is the current status of the python-openstacksdk
>> project.  The Octavia team has posted some patches implementing our new
>> Octavia v2 API [1] in the SDK, but we have not had any reviews.  I have also
>> asked some questions in #openstack-sdks with no responses.
>> I see that there are some maintenance patches getting merged and a pypi
>> release was made 6/14/17 (though not through releases project).  I'm not
>> seeing any mailing list traffic and the IRC meetings seem to have ended in
>> 2016.
>>
>> With all the recent contributor changes, I want to make sure the project
>> isn't adrift in the sea of OpenStack before we continue to spend development
>> time implementing the SDK for Octavia. We were also planning to use it as
>> the backing for our dashboard project.
>>
>> Since it's not in the governance projects list I couldn't determine who the
>> PTL to ping would be, so I decided to ping the dev mailing list.
>>
>> My questions:
>> 1. Is this project abandoned?
>> 2. Is there a plan to make it an official project?
>> 3. Should we continue to develop for it?
> 
> Thanks for raising this.
> 
> Beyond its limited activity, another issue is that it's not an official
> project while its name make it a "default choice" for a lot of users
> (hard to blame them for thinking that
> http://git.openstack.org/cgit/openstack/python-openstacksdk is not the
> official Python SDK for OpenStack, but I digress). So I agree that the
> situation should be clarified.
> 
> I know that Monty has pretty strong feelings about this too, so I'll
> wait for him to comment.

Oh boy. I'd kind of hoped we'd make it to the PTG before starting this 
conversation ... guess not. :)

Concerns
--------

I share the same concerns Thierry listed above. Specifically:

* It is not an official project, but its name leads people to believe 
it's the "right" thing to use if they want to talk to OpenStack clouds 
using Python.

* The core team is small to begin with, but recently got hit in a major 
way by shifts in company priorities.

I think we can all agree that those are concerns.

Three additional points:

* The OpenStack AppDev group and the various appdev hackathons use 
shade, not openstacksdk. This is what we have people out in the world 
recommending people use when they write code that consumes OpenStack 
APIs. The Interop challenges at the Summits so far have all used 
Ansible's OpenStack modules, which are based on shade, because they were 
the thing that works.

* Both shade and python-openstackclient have investigated using 
openstacksdk as their REST layer but were unable to because it puts some 
abstractions in layers that make it impossible to do some of the 
advanced things we need.

* openstacksdk has internal implementations of things that exist at 
different points in the stack. We just added full support for version 
service and version discovery to keystoneauth, but openstacksdk has its 
own layer for that so it both can't use the ksa implementation and is 
not compliant with the API-WG consume guidelines.

It's not all bad! There is some **great** work in openstacksdk and it's 
a shame there are some things that make it hard to consume. Brian, 
Qiming and Terry have done a bunch of excellent work - and I'd like to 
not lose it to the dustbin of corporate shifting interest.

**warning** - there is a very large text wall that follows. If you don't 
care a ton on this topic, please stop reading now, otherwise you  might 
rage-quit computers altogether.

Proposal
--------

I'd propose we have the shade team adopt the python-openstacksdk codebase.

This is obviously an aggressive suggestion and essentially represents a 
takeover of a project. We don't have the luxury of humans to work on 
things that we once had, so I think as a community we should be 
realistic about the benefits of consolidation and the downside to 
continuing to have 2 different python SDKs.

Doing that implies the following:

* Rework the underlying guts of openstacksdk to make it possible to 
replace shade's REST layer with openstacksdk. openstacksdk still doesn't 
have a 1.0 release, so we can break the few things we'll need to break.

* Update the shade mission to indicate its purpose in life isn't just 
hiding deployer differences but rather is to provide a holistic 
cloud-centric (rather than service-centric) end-user API library.

* Merge the two repos and retire one of them. Specifics on the mechanics 
of this below, but this will either result in moving the resource and 
service layer in openstacksdk into shade and adding appropriate 
attributes to the shade.OpenStackCloud object, or moving the 
shade.OpenStackCloud into something like openstack.cloud and making a 
shade backwards-compate shim. I lean towards the first, as we've been 
telling devs "use shade to talk to OpenStack" at hackathons and 
bootcamps and I'd rather avoid the messaging shift. However, pointing to 
an SDK called "The Python OpenStack SDK" and telling people to use it 
certainly has its benefits from a messaging perspective.

* Collapse the core teams - members of the python-openstacksdk-core team 
who desire to stick around (I see Qiming doing reviews still, and Brian 
has been doing occasional ones even after his day-job shift) are welcome 
to be added to the shade-core team, but should not feel compelled to or 
like they'd be letting anyone down if they didn't. Day job priorities 
shift, it turns out.

Reworking the Guts
------------------

I did a scan through openstacksdk the other day to catalog what would 
need to be reworked. The following are the big-ticket items:

* drop stevedore/plugin support. An OpenStack REST client has no need 
for plugins. All services are welcome. *note below*

* drop keystoneauth.Session subclass. It's over-riding things at the 
wrong layer. keystoneauth Adapter is the thing it wants to be.

* stop using endpoint_filter in each Session call. Instead create an 
Adapter with the discovery parameters needed.

* add support for per-request microversions. Based on the structure 
currently in openstacksdk once the wrapped Session has been replaced 
with keystoneauth Adapter this should work really nicely.

* drop incomplete version discovery support in favor of the support in 
keystoneauth

* drop Profile object completely and replace its use internally with the 
os-client-config CloudConfig object.

That's not a ton of work, TBH- I could probably do all of it in a single 
long plane flight. It will break advanced users who might have been 
using Profile (should be transparent to normal users) but as there is no 
1.0 I think we should live with that. We might be able to make a shim 
layer for the Profile interface to avoid breaking people using the 
interface.

*note on plugins*

shade has a philosophy of not using plugins for service support that I'd 
like to apply here. All OpenStack services are welcome to add code 
directly. openstacksdk ALREADY contains code for tons of services. The 
only thing plug-ability adds in this context is the ability to use 
openstacksdk to support non-OpenStack services... and at this point I do 
not think that is valuable. The only place this is currently used is in 
the Profile anyway which allows defining an entrypoint to use to 
override a service - and since I'm proposing we kill the Profile object 
this all falls out as a matter of consequence.

Integrating with shade
----------------------

The primary end-user concept in shade is an OpenStackCloud object, on 
which one performs actions. The service that provides the action is 
abstracted (this is done because actions such as 'list_images' may need 
to be done on the image service or the compute service, depending on the 
cloud). So a user does:

   cloud = shade.openstack_cloud(cloud='example')
   images = cloud.list_images()

The primary end-user concept in openstacksdk is the Connection, which 
has an object for each service. For example:

   conn = openstack.connection.from_config(cloud_name='example')
   images = conn.image.images()

If we merge the two with the shade library being the primary interface, 
we could add the current sdk service proxy objects as attributes to the 
OpenStackCloud object, so that the following would work:

   cloud = shade.openstack_cloud(cloud='example')
   images = cloud.list_images()
   images = cloud.image.images()

If we did the merge the other way, we could either keep the Connection 
concept and stitch the shade helpers on to it:

   conn = openstack.connection.from_config(cloud_name='example')
   images = conn.list_images()
   images = conn.image.images()

Or do a hybrid:

   cloud = openstack.cloud(name='example')
   images = cloud.list_images()
   images = cloud.image.images()

If we go either of the routes of merging shade into openstacksdk then 
the shade library itself could just be a simple sugar layer for 
backwards compat that has things like:

   def openstack_cloud(cloud=None, *args, **kwargs):
     return openstack.cloud(name=cloud, *args, **kwargs)

and

   class OpenStackCloud(openstack.cloud):
     def __init__(self, cloud=None, *args, **kwargs):
       super(OpenStackCloud, self).__init__(name=cloud, *args, **kwargs)

I kind of like the 'Connection' term, as it communicates that this is a 
thing that has and shares a discrete remote connection (which is a 
shared keystoneauth Session) OpenStackCloud in shade **ACTUALLY** 
describes a cloud-region (as regions in OpenStack are essentially 
independent clouds from an API consumption perspective. So I may be 
leaning more towards merging in that direction.

* data model - shade has a data model contract for the resources its 
knows about. This actually fits nicely with the Resource construct in 
openstacksdk, although there are some differences. We should likely push 
the data model and normalization contract into the openstacksdk resource 
layer so that people get matching resources regardless of whether they 
use the shade interop layer or the low-level per-service layer.

* Implement some constructor smarts for easy pass-through of sdk service 
proxy methods into shade wrapper methods. For MANY of the remote calls, 
the list_ get_ search_ create_ update_ and delete_ methods are (or can 
be) mechanical passthrough from the SDK objects. We should be able to 
write some smart constructor logic that makes all the passthrough 
methods for us and just have explicit methods defined for the places 
where the shade layer legitimately needs to do a bunch of logic (like 
image upload and auto-ip support)

* Make openstack.resource2.Resource a subclass of munch.Munch. This will 
be fun. If we want to be able to have model/normalization happen at the 
lower-level, we'd ultimately want the shade methods to be able to just 
return the object the sdk layer produces. Shade's interface defines that 
we return Munch objects (basically things that behave like dicts and 
objects) That's VERY similar to what the Resource already does - so if 
we subclass Munch the shade behavior will hold and the sdk coding style 
should also be able to hold as it is today. Otherwise we'd have to have 
every return in shade wrap the object in a 
munch.Munch(resource.to_dict()) which would lose information for when 
that object wants to be passed back in to the API later. (there are 
smarts in Resource for doing update things)

* Migrate openstacksdk unit tests to use requests_mock. We're pretty 
much finished doing this in shade and it's been SUPER helpful. 
openstacksdk has a mock of the Session object in test_proxy_base2.py, so 
there's some good places to lay this in ... and as we merge the two 
obviously shade's requests_mock unittests will continue to apply - but 
given the sdk's test organization I think we can get some really solid 
results by moving the mocking to the actual rest payload layer instead 
of mocking out the Session itself. It also is a great way to verify that 
things work as expected with varying payloads - so as a user finds an 
edge-case from a specific cloud, grabbing an http trace from them and 
constructing a specific test is a great way to deal with regressions.

We'll also need to make a specific rollout plan. shade has a strict 
backwards-compat policy, so if we merge sdk into shade, it goes from 
being 0.9 to being fully supported very quickly and we need to make sure 
we don't have anything exposed in a public interface we don't want to 
support for ages (resource -> resource2 should likely get finished and 
then resource2 should likely get renamed back to resource before such a 
release, for instance) If we merge the other direction and make the 
current shade a backwards-compat shim lib, we'll also need to cut a 1.0 
of sdk pretty quickly, as whatever passthrough object we are exposing 
via the shade layer from sdk just adopted the shade backwards compat 
contract. I don't have a detailed plan for this yet, but if we decide to 
go down this path I'll make one.

Other TODO list items
---------------------

It's not just all grunt work nobody can see. There's fun things to do too!

* Add openstack.OpenStack or shade.AllClouds class - It's been a 
todo-list item for me for a while to make a wrapper class in shade that 
allows easy operations on a set of Clouds. Again, I like the semantics 
of openstack.OpenStack better - so that's another argument in favor of 
that direction of merge ... it would look something like this:

   # make Connection objects for every cloud-region in clouds.yaml
   clouds = openstack.OpenStack()
   # thread/asyncio parallel fetch of images across all clouds
   images = clouds.list_images()
   # all objects in shade have a "location" field which contains
   # cloud, region, domain and project info for the resource
   print([image for image in images if image.location.cloud='vexxhost'])
   # Location is a required parameter for creation
   vexxhost = clouds.get_location(name='vexxhost')
   clouds.create_image(
     location=vexxhost, name='my-fedora-26',
     filename='fedora26.qcow2')

Most of this work can actually likely be done with one smart metaclass 
... finding the methods on OpenStackCloud and either doing a set of 
parallel gets on a list of objects or adding a location argument to 
write operations doesn't vary depending on the type of object. Since all 
the methods are named consistently, finding 'list_*' and making a set of 
corresponding list_ methods on the OpenStack object is essentially just 
one chunk of smart constructor.

* Finish per-resource caching / batched client-side rate-limiting work. 
shade has a crazy cool ability to do batched and rate-limited operations 
... this is how nodepool works at the scale it does. But it's currently 
only really plumbed through for server, floating-ip and port. (guess 
what nodepool has to deal with) This should be generalized to all of the 
resources, configurable on a per-resource name in clouds.yaml, and 
should work whether high or low level interfaces are used. This is super 
hard to get RIGHT, so it's one of those "spend 4 weeks writing 10 lines 
of code" kind of things, but it's also super important.

* Implement a flag for toggling list/client-side-filter vs. remote-get 
operations. Most single-resource operations in shade are actually done 
as a list followed by a client-side filter. Again this model is there to 
support nodepool-scale (amusingly enough it makes it less load on the 
clouds at scale) but at small scale it is more costly and some users 
find it annoying. We've discussed having the ability to toggle this at 
constructor time - and then having things like the ansible modules 
default the flag to use remote-get instead of list/filter - since those 
are doing lots of independent processes so the optimization of 
list/filter isn't ever realized.

* Implement smarter and more comprehensive "pushdown" filtering. I think 
we can piggyback a bunch of this off of the SDK layer - but there are 
attributes that can be used for server-side filtering, there are 
attributes that can't, and there are attributes that are client-side 
created via normalization that either can be translated into a 
server-side filter or must be client-side filtered. Resource has the 
structure for dealing with this sanely I believe, but it needs to be 
tracked through.

* Add python3-style type annotations. We just started doing this in zuul 
and it's pretty cool - and is possible to do in a python2 compatible way.

Longer Goals
------------

That gets us a home for openstacksdk, a path towards consolidation of 
effort and a clear story for our end users. There are a few longer-term 
things we should be keeping in mind as we work on this:

* suitability for python-openstackclient. Dean and Steve have been 
laying in the groundwork for doing direct-REST in python-openstackclient 
because python-*client are a mess from an end-user perspective and 
openstacksdk isn't suitable. If we can sync on requirements hopefully we 
can produce something that python-openstackclient can honestly use for 
that layer instead of needing local code.

* suitability for heat/horizon - both heat and horizon make calls to 
other OpenStack services as a primary operation (plenty of services make 
service-to-service calls, but for heat and horizon is a BIG part of 
their life) The results of this work should allow heat and horizon to 
remove local work they have using python-*client, doing local version 
discovery or any of the rest - and should expose to them rich primitives 
they can use easily.

Conclusion
----------

As I mentioned at the top, I'd been thinking some of this already and 
had planned on chatting with folks in person at the PTG, but it seems 
we're at a place where that's potentially counter productive.

Depending on what people think I can follow this up with some governance 
resolutions and more detailed specs.

Thanks!
Monty