Open Stack

I'm going to top-post, because there is a whole other thing which is not
a response to points below. Basically, this is yet-another-instance of
two competing and partially contradictory sets of use cases and usage
patterns that we're trying to find the right compromise for.

Tying client libs to server releases is really handy for the distros. It
is terrible for the public cloud implementations who are doing rolling
releases - not in as much as it effects the public cloud's ability to
use the client libs - they can obviously pull trunk client lib from git
and use that for intra-server communication just fine as part of their
deployment. Rather, it is a terrible experience for the end-users of
OpenStack public clouds.

I'm going to use myself as an example, because it's specific.

A few months ago, HP Cloud added keystone-based auth to their
diablo-based public cloud. This was before essex released. This meant
that to use HP Cloud's public service, there were two choices:

- trunk python-novaclient
- crazy library released directly from HP

The fact that there was a stable diablo python-novaclient in existence
was useless - it did not work with this cloud that I had an account to
that was branded OpenStack (and mind you, not branded OpenStack Diablo
or OpenStack Essex - just branded OpenStack, because that's how it works)

So as an end user, if we had the 6-month client lib release model, I'm
hosed, and all of a sudden I'm installing libraries from git, or I'm
starting to use a non-standard "hpcloud" library which is then tying my
use of APIs to a non-standard thing, which is bad for all of us.

What version of a library ubuntu or fedora shipped could not be less
interesting to me - using my cloud accounts is what is important.

Thing is - trunk python-novaclient at that point in time worked just
fine with vanilla diablo clouds (as well as anything did) So there was
no reason to not do an actual pypi release of python-novaclient.

This is the model and the use case that I'm talking about here. The best
part is - there is absolutely nothing in the more-frequent client
release model which subverts the abilities of the distros to do the
thing they do - which is release software on schedules and then maintain
a version. If you're interested in Ubuntu Enterprise Cloud, and you
install UEC Essex on your local machines, then you will also install the
version of the libs that your distro released at the same time, and the
fact that there are newer libs on pypi is irrelevant to you. The distro
release model FOR CLIENT LIBS on the other hand, makes it hard for end
users to consume public cloud resources - which is why I'm suggesting
aligning this release model more closely with the needs of public
rolling releases. It seems like the right side of the line on which to
put the compromise this time. Some of the rolling-release installations
would likely prefer if we did more frequent releases for the server
projects as well... but that _does_ impact the ability of the distros to
do their thing, so we're in more alignment with the distros on that one.

So while I hear all of your points (and I promise, Thierry makes them
too) - we have many more end users and use cases that the distros or the
distro model, and I think that's most evident in the case of the client
libraries... especially in so much as I imagine the predominant channel
for the client libs with be pypi and not apt, yum or zypper.

I hope this helps understand where this approach is coming from.


As for CI - we do not strictly need to release to PyPI for CI purposes -
github zipball support allows us to meet our needs from a
pip-requires/test-requires perspective. It will be easier/make more
sense/more cacheable if it's on pypi because then our pypi mirror can
handle things, which means we aren't exposed to github breakages, but
it's certainly not needed for it. Red herring.

Monty

PS. Doing separate channels in pypi is a complete non-starter. It's a
terrible piece of software which is horribly designed, but it's the
standard mechanism for distributing python libraries, so we have to live
with it.

On 06/19/2012 08:06 AM, Mark McLoughlin wrote:
> On Tue, 2012-06-19 at 16:48 +0200, Thierry Carrez wrote:
>> Mark McLoughlin wrote:
> [..]
>>>> However they also inherited the release scheme of the server project
>>>> (new version every 6 months), which was (or was not) synced to PyPI
>>>> depending of who was owning the PyPI project. More confusion as PyPI
>>>> rarely contained the just-released version, since releasing to PyPI was
>>>> not handled by OpenStack release management.
>>>
>>> I would have thought that only OpenStack release management should be
>>> publishing those libs to pypi now.
>>
>> PyPI doesn't support complex versioning, nor does it support "channels"
>> (think separate repositories for the same library from which you can get
>> only daily/milestones/stable-releases).
> 
> Nice way of putting it - "channels". I think that helps clarify the
> discussion hugely :)
> 
> (I guess you could have python-glanceclient-dev as a separate pypi
> package to give a similar effect, but that would probably be evil)
> 
> [..]
> 
>> PyPI only supports one channel. Which one should it be ?
> 
> IMHO, PyPI should be a channel where you get:
> 
>   - bug fixes
> 
>   - support for features which exist in the currently released version
>     of OpenStack
> 
>   - every six months, support for new features in the just-released 
>     version of OpenStack

> i.e. we push 2012.1 to pypi when Essex is released and we do 2012.1.x
> releases as needed to pypi from our "stable" branch.
> 
> Where the definition of "stable" is slightly different - it includes new
> features which expose existing REST API features of the associated
> release.
> 
> i.e. a channel containing (1) and (2) in the split I made.
> 
>>> If the issue is integration *within* OpenStack - e.g. Nova needing
>>> cinderclient - then I don't think that should dictate the release
>>> schedule for these libs. We can surely figure out how to do that without
>>> doing more regular official releases of the libs? But first, is that
>>> what we're talking about?
>>
>> I think so. My understanding is that it's a lot more elegant to rely on
>> PyPI for our own libraries, rather than special-casing them for our
>> developers and our CI.
> 
> For integration in the project, we need the "development channel" and I
> don't think we want to pollute pypi with that.
> 
> (Again the point about the REST API potentially changing during
> development)
> 
> So, I don't think I buy this bit and think it's worth digging into the
> details.
> 
> [..]
>>> [...]
>>> The way I'm seeing it now is there are three development streams:
>>>
>>>   1) bugfixes (i.e. suitable for a stable branch)
>>>
>>>   2) new client lib features built on top of existing REST API features 
>>>      (e.g. feature additions to the client libs which were released as 
>>>      part of Essex which do not require REST API features from Folsom)
>>>
>>>   3) new client lib features built on top of new REST API features 
>>>      (e.g. the Folsom development stream)
>>>
>>> IMHO, if you're J. Random Developer doing 'pip install glanceclient',
>>> you probably want (2).
>>>
>>> Particularly risk-averse distros may want (1), since (2) is more risky
>>> to pull updates from.
>>>
>>> And for integration within the project, or folks testing the latest
>>> development stream, you want (3).
>>>
>>> The project probably can't hope to support all 3 development streams,
>>> but can we do (1) and (2) in a "stable" branch and (3) using our usual
>>> development release cycle?
>>>
>>> Would that satisfy everyone but the "particularly risk-averse distros"?
>>
>> That forces us to handle our client library releases outside of PyPI
>> though.
> 
> It forces us to handle our "development stream" outside of pypi, yes.
> 
> Cheers,
> Mark.
> 
> 
> _______________________________________________
> Mailing list: https://launchpad.net/~openstack
> Post to     : openstack at lists.launchpad.net
> Unsubscribe : https://launchpad.net/~openstack
> More help   : https://help.launchpad.net/ListHelp
>