Open Stack

On Jan 15, 2014, at 1:37 PM, Doug Hellmann <doug.hellmann at dreamhost.com> wrote:

> Several people have mentioned to me that they are interested in, or actively working on, code related to a "common" client library -- something meant to be reused directly as a basis for creating a common library for all of the openstack clients to use. There's a blueprint [1] in oslo, and I believe the keystone devs and unified CLI teams are probably interested in ensuring that the resulting API ends up meeting all of our various requirements.
> 
> If you're interested in this effort, please subscribe to the blueprint and use that to coordinate efforts so we don't produce more than one common library. ;-)
> 
> Thanks,
> Doug
> 
> 
> [1] https://blueprints.launchpad.net/oslo/+spec/common-client-library-2

*raises hand* 

Me me! 

I’ve been talking to many contributors about the Developer Experience stuff I emailed out prior to the holidays and I was starting blueprint work, but this is a great pointer. I’m going to have to sync up with Alexei.

I think solving this for openstack developers and maintainers as the blueprint says is a big win in terms of code reuse / maintenance and consistent but more so for *end-user developers* consuming openstack clouds.

Some background - there’s some terminology mismatch but the rough idea is the same:

* A centralized “SDK” (Software Development Kit) would be built condensing the common code and logic and operations into a single namespace.

* This SDK would be able to be used by “downstream” CLIs - essentially the CLIs become a specialized front end - and in some cases, only an argparse or cliff front-end to the SDK methods located in the (for example) openstack.client.api.compute 

* The SDK would handle Auth, re-auth (expired tokens, etc) for long-lived clients - all of the openstack.client.api.** classes would accept an Auth object to delegate management / mocking of the Auth / service catalog stuff to. This means developers building applications (say for example, horizon) don’t need to worry about token/expired authentication/etc. 

* Simplify the dependency graph & code for the existing tools to enable single binary installs (py2exe, py2app, etc) for end users of the command line tools.

Short version: if a developer wants to consume an openstack cloud; the would have a single SDK with minimal dependencies and import from a single namespace. An example application might look like:

from openstack.api import AuthV2
from openstack.api import ComputeV2

myauth = AuthV2(…., connect=True)
compute = ComputeV2(myauth)

compute.list_flavors()

This greatly improves the developer experience both internal to openstack and externally. Currently OpenStack has 22+ (counting stackforge) potential libraries a developer may need to install to use a full deployment of OpenStack:

  * python-keystoneclient (identity)
  * python-glanceclient (image)
  * python-novaclient (compute)
  * python-troveclient (database)
  * python-neutronclient (network)
  * python-ironicclient (bare metal)
  * python-heatclient (orchestration)
  * python-cinderclient (block storage)
  * python-ceilometerclient (telemetry, metrics & billing)
  * python-swiftclient (object storage)
  * python-savannaclient (big data)
  * python-openstackclient (meta client package)
  * python-marconiclient (queueing)
  * python-tuskarclient (tripleo / management)
  * python-melangeclient (dead)
  * python-barbicanclient (secrets)
  * python-solumclient (ALM)
  * python-muranoclient (application catalog)
  * python-manilaclient (shared filesystems)
  * python-libraclient (load balancers)
  * python-climateclient (reservations)
  * python-designateclient (Moniker/DNS)

If you exclude the above and look on PyPI:

On PyPi (client libraries/SDKs only, excluding the above - not maintained by openstack):

 * hpcloud-auth-openstack 1.0
 * python-openstackclient 0.2.2
 * rackspace-auth-openstack 1.1
 * posthaste 0.2.2
 * pyrax 1.6.2
 * serverherald 0.0.1
 * warm 0.3.1
 * vaporize 0.3.2
 * swiftsc (https://github.com/mkouhei/swiftsc)
 * bookofnova 0.007
 * nova-adminclient 0.1.8
 * python-quantumclient 2.2.4.3
 * python-stackhelper 0.0.7.1.gcab1eb0
 * swift-bench 1.0
 * swiftly 1.12
 * txAWS 0.2.3 
 * cfupload 0.5.1
 * python-reddwarfclient 0.1.2
 * python-automationclient 1.2.1
 * rackspace-glanceclient 0.9
 * rackspace-novaclient 1.4

If you ignore PyPI and just want to install the base say - 7 services, each one of the python-** clients has its own dependency tree and may or may not build from one of the others. If a vendor wants to extend any of them, it’s basically a fork instead of a clean plugin system.

On the CLI front - this would *greatly* simplify the work openstackclient needs to do - it would be able to import from the main SDK and simply provide the noun-verb command line and any other end-user sugar it wanted to. Even if each service wanted to keep its own python-X client instead of relying on openstackclient it would be minimal to depend on the core SDK and then plugin/extend to build a specialized CLI for the project - if you really wanted, you could also extend openstackclient directly.

Roughly this is the punch list I was looking at:

1: a blueprint that explains the rationale behind unifying the Client code from the openstack clients; using a single REST interface, common object hierarchy, etc. 
2: A path for implementation of the common SDK including operational code

4: dealing with a single binary cross platform for the CLI that derives from the common SDK (hard requirement) 
5: Standardization of names (e.g Compute != Nova, use the real names, not project names)
6: Allow vendors to alias names for services to match their offerings

I’ll begin working on the blueprint you pointed to - given this is more akin to a horizon-like UX project than a sub project of Oslo itself; does it really belong there? I do see the work within the individual clients:

https://review.openstack.org/#/q/topic:bp/common-client-library-2,n,z

Jamie’s comments in:

https://review.openstack.org/#/c/63164/

Do concern me as I’d like to not do this as a lowest common denominator; in this case the client code for keystoneclient might be in openstack.api.auth but it would be able to be as advanced as it would like from an api standpoint - and whatever subset of functionality could be exposed in higher level abstractions (such as a CLI). Bonus is that horizon could potentially use this work.

thoughts welcome, I think the blueprint is a good start - I shot it out to all of us discussing this to take a look

jesse