Open Stack

Mark McLoughlin wrote:
>> Originally the client library projects were independent and were
>> released by their author to PyPI. People built their tooling and
>> automation
> 
> Interested in specific examples ...

Most developers apparently pull their libraries from PyPI. It's a bit
strange for people working on distro development like you and me, but
that's what other people do. Our continuous integration also uses PyPI
to pull Python libraries.

>> However they also inherited the release scheme of the server project
>> (new version every 6 months), which was (or was not) synced to PyPI
>> depending of who was owning the PyPI project. More confusion as PyPI
>> rarely contained the just-released version, since releasing to PyPI was
>> not handled by OpenStack release management.
> 
> I would have thought that only OpenStack release management should be
> publishing those libs to pypi now.

PyPI doesn't support complex versioning, nor does it support "channels"
(think separate repositories for the same library from which you can get
only daily/milestones/stable-releases). I didn't want to let PyPI
limitations affect the versioning that makes the most sense for the
(server-side) core projects... so I ignored it. Turns out you can't
ignore PyPI, which is why Monty came up with this solution to our
problem: use a separate release scheme for client libraries, since they
*are* separate projects anyway.

>> People kept on asking for new PyPI releases to access recent features.
>> Sometimes it's just that they need an early version of the client
>> somewhere to start integrating it before the milestone hits
>> (cinderclient, swiftclient). A release every 6 months is not enough.
> 
> Well, we do releases more often that that:
> 
>  - milestone releases provide early access to features currently under 
>    development
> 
>  - stable releases provide a safe source of bugfixes to previous 
>    releases

But those are not "released" in the same channels. We have the
per-commit channel, released several times a day. We have the milestone
channel, every ~2 months... and the release channel, every 6 months.
PyPI only supports one channel. Which one should it be ?

> If the issue is integration *within* OpenStack - e.g. Nova needing
> cinderclient - then I don't think that should dictate the release
> schedule for these libs. We can surely figure out how to do that without
> doing more regular official releases of the libs? But first, is that
> what we're talking about?

I think so. My understanding is that it's a lot more elegant to rely on
PyPI for our own libraries, rather than special-casing them for our
developers and our CI.

> I think we'd like to be able to publish milestone releases of the client
> libs to pypi, but not have "pip install" install milestones unless they
> are explicitly requested? Right?
> 
> pypi probably doesn't support that, but do we want to allow pypi's
> limitations dictate our schedule either?

Yes, PyPI doesn't support that. Don't get me wrong, I'd love that PyPI
suddenly becomes smart (or that all developers used Linux distros with
sane packaging), but as long as it's the de-facto standard for Python
developers to get Python libraries we have to try our best to use it. In
the precise case of client libraries, accepting PyPI limitations sounds
like a small price to pay to simplify and enable generic access to our
libraries using a well-established system.

> [...]
> Ok, so this *is* all about pypi? I'm starting to understand now :)

It's been my nightmare for more than a year.

> [...]
> The way I'm seeing it now is there are three development streams:
> 
>   1) bugfixes (i.e. suitable for a stable branch)
> 
>   2) new client lib features built on top of existing REST API features 
>      (e.g. feature additions to the client libs which were released as 
>      part of Essex which do not require REST API features from Folsom)
> 
>   3) new client lib features built on top of new REST API features 
>      (e.g. the Folsom development stream)
> 
> IMHO, if you're J. Random Developer doing 'pip install glanceclient',
> you probably want (2).
> 
> Particularly risk-averse distros may want (1), since (2) is more risky
> to pull updates from.
> 
> And for integration within the project, or folks testing the latest
> development stream, you want (3).
> 
> The project probably can't hope to support all 3 development streams,
> but can we do (1) and (2) in a "stable" branch and (3) using our usual
> development release cycle?
> 
> Would that satisfy everyone but the "particularly risk-averse distros"?

That forces us to handle our client library releases outside of PyPI
though. My understanding is that not using PyPI for our client libraries
is not an option. We tried that before, and people kept on publishing
random unofficial versions there. So we tried that, and it failed. PyPI
is there to stay and can't be ignored. So we have to accept that it
supports only basic versioning and that it is single-channel. Monty's
proposal, I think, makes the best of our situation.

-- 
Thierry Carrez (ttx)
Release Manager, OpenStack