Open Stack

Hi Monty,

Thanks for sending.

For reference, this was the link you posted last week:

  http://wiki.openstack.org/Governance/Proposed/LibraryProjectDefinition

One question I had on that is re:

  the ability to release a client library outside of the core project 
  release cycle (requests have been coming in to our release manager for
  this)

Who were these request from and why? That would help understand what
we're trying to do here.

In any case, my tl;dr suggestion is:

  1) The client libs get released along with OpenStack releases and 
     versioned the same.

     So, we release glanceclient-2012.2 with the Folsom release.

     This makes it really easy for users to understand. If you want a 
     client lib for the REST API versions first released in Folsom, use 
     the Folsom (or later) lib.

  2) We also include client libs in milestone releases, for people 
     wanting to use the libs against development releases of OpenStack. 

     One issue is how to version these in pypi? Don't know enough about 
     pypi's setup to comment, but e.g. do we want the latest milestone
     release to be what people get when they do 'pip install'?

     Another issue is if folks want a release of a client lib between 
     milestones. We should be able to do that, so long as it doesn't 
     mean prematurely locking us into API compat before work on a new 
     API is complete.

  3) For folks who want a safe source of bugfixes, we can maintain a
     stable branch of the libs and do releases of those.

On Mon, 2012-06-18 at 14:11 -0700, Monty Taylor wrote:
> We're trying to figure out how we release client libraries. We're really
> close - but there are some sticking points.
> 
> First of all, things that don't really have dissent (with reasoning)
> 
> - We should release client libs to PyPI
> 
> Client libs are for use in other python things, so they should be able
> to be listed as dependencies. Additionally, proper releases to PyPI will
> make our cross project depends work more sensibly
> 
> - They should not necessarily be tied to server releases
> 
> There could be a whole version of the server which sees no needed
> changes in the client. Alternately, there could be new upcoming server
> features which need to go into a released version of the library even
> before the server is released.

At what point does the server commit to maintaining compatibility with
its new API?

e.g. if we add a new API and, a week later an official release is done
of the client lib to support the new API ... have we then prevented
ourselves from fixing issues with the API?

> - They should not be versioned with the server
> 
> See above.
> 
> - Releases of client libs should support all published versions of
> server APIs
> 
> An end user wants to talk to his openstack cloud - not necessarily to
> his Essex cloud or his Folsom cloud. That user may also have accounts on
> multiple providers, and would like to be able to write one program to
> interact with all of them - if the user needed the folsom version of the
> client lib to talk to the folsom cloud and the essex version to talk to
> the essex cloud, his life is very hard. However, if he can grab the
> latest client lib and it will talk to both folsom and essex, then he
> will be happy.
> 
> There are three major points where there is a lack of clear agreement.
> Here they are, along with suggestions for what we do about them.
> 
> - need for "official" stable branches
> 
> I would like to defer on this until such a time as we actually need it,
> rather than doing the engineering for in case we need it. But first, I'd
> like to define we, and that is that "we" are OpenStack as an upstream.
> As a project, we are at the moment probably the single friendliest
> project for the distros in the history of software. But that's not
> really our job.

That's quite a bunch of claims ... I'll ignore them for now :)

> Most people out there writing libraries do not have multiple parallel
> releases of those libraries - they have the stable library, and then
> they release a new one, and people either upgrade their apps to use
> the new lib or they don't.

Don't agree, here's an example - libvirt 0.9.11 announced:

  https://www.redhat.com/archives/libvir-list/2012-April/msg00085.html

and then 4 "maintenance releases" of 0.9.11.x:

  https://www.redhat.com/archives/libvir-list/2012-April/msg01410.html
  https://www.redhat.com/archives/libvir-list/2012-April/msg01429.html
  https://www.redhat.com/archives/libvir-list/2012-May/msg00370.html
  https://www.redhat.com/archives/libvir-list/2012-June/msg00686.html

and here's a release of 0.9.12:

  https://www.redhat.com/archives/libvir-list/2012-May/msg00707.html

an, lo, here's where I first made the case of "bugfix only" releases of
libvirt:

  http://www.redhat.com/archives/libvir-list/2009-March/msg00485.html

> One of the reasons this has been brought up as a need is to allow for
> drastic re-writes of a library. I'll talk about that in a second, but I
> think that is a thing that needs to have allowances for happening.
> 
> So the model that keystone-lite used - create an experimental branch for
> the new work, eventually propose that it becomes the new master - seems
> like a better fit for the "drastic rewrite" scenario than copying the
> stable/* model from the server projects, because I think the most common
> thing will be that library changes are evolutionary, and having two
> mildly different branches that both represent something that's actually
> pretty much stable will just be more confusing than helpful.
> 
> That being said - at such a time that there is actually a pain-point or
> a specific need for a stable branch, creating branches is fairly easy
> ... but I think once we have an actual burning need for such a thing, it
> will make it easier for us to look at models of how we'll use it.

I think you've done a tonne of hand-waving there and not really
addressed the very simple reason that a stable branch is useful:

  Some people just want a safe source of bug fixes for the features 
  they are already using, without the additional risk of an update that 
  includes new features

For example, when someone creates a distribution of Essex they will
include the client libraries which work with the Essex APIs. They may
also want to provide a steady stream of bugfix updates to their Essex
users. They will not include a version of the client library that
supports Folsom APIs in that update stream, purely because of the risk
of regression.

>  - API or major-rewrite-driven versioning scheme
> 
> I was wondering why bcwaldon and I were missing each other so strongly
> in the channel the other day when we were discussing this, and then I
> realized that it's because we have one word "API" that's getting
> overloaded for a couple of different meanings - and also that I was
> being vague in my usage of the word. So to clarify, a client library has:
> 
>  * programming level code APIs
>  * supported server REST APIs
> 
> So I back off everything I said about tying client libs version to
> server REST API support. Brian was right, I was wrong. The thing that's
> more important here is that the version should indicate programmer
> contract, and if it that is changed in a breaking manner, the major
> number should bump.
> 
> If we combine that with the point from above that our libraries should
> always support the existing server REST APIs, then I think we can just
> purely have statements like "support for compute v3 can be found in
> 2.7.8 and later" and people will likely be fine, because it will map
> easily to the idea "just grab the latest lib and you should be able to
> talk to the latest server" Yea?
> 
> So in that case, the client libs versions are purely whatever they are
> right now, and we'll increase them moving forward using normal library
> version thoughts.

Right, the choice of version number is arbitrary (it does not say
anything about the REST APIs it supports), but the major/minor/micro
numbers roughly mean what they mean every where else (in terms of python
API contract).

You know what? A version like 2012.1.1 sounds like a good version number
to me :)

>  - room for deprecating old APIs
> 
> The above then leads us to wanting to know what we do about supported
> server REST APIs over time, especially since I keep making sweeping
> statements about "should support all available server versions" ... How
> about this as a straw man: Since we're planning on beginning to run
> tests of the client libs against previous versions (so we'll test trunk
> novaclient against essex nova in addition to trunk nova) ... we need
> support for past versions of servers as long as our automation can
> sensibly spin up a past version. (Since the support for that API version
> shouldn't need huge amounts of work moving forward) But there will reach
> a point where old server versions require stuff that's older than we
> feel like supporting, and at that point we drop it. (or, more to the
> point, that we reserve the right in the future to declare that we're
> going to drop old server API versions - but the general policy is that
> we'll keep old support until it becomes a pain in the ass)

Hmm, the "as long as our automation ..." thing sounds bizarre to me - we
remove support for Diablo when our CI can't spin up Diablo for testing
any more? This should be about our users, not about us.

Where the client libs are part of an OpenStack distribution and intended
for use against deployments of that distribution, we could probably just
support the previous version of OpenStack - e.g. drop Diablo support in
Folsom.

But where the client libs are intended for use against any public
OpenStack cloud out there, we should only drop support for API versions
which no longer exist in the cloud.

Cheers,
Mark.