Open Stack

I think its a nice ideal to not depend on < 1.0 libraries but I don't
think it's possible currently,

When I did the design session I gathered some stats about this:

Global-requirements.txt installed in a venv (u can try this at home).
 - 206 requirements installed
 - 102 were >= 1.0
 - 104 were < 1.0

I think we'd be shooting ourselves in the foot if we tried this at the
current stage and I'm not sure if we would have as many contributors
involved in openstack if we also do this (sometimes it's actually
beneficial to adopt things even if they are < 1.0, that¹s how software
matures to be 1.0, by users adopting it, and if we aren't helping mature
the openstack community and related communities then I think we failed our
jobs).

Maybe it can be a long-term goal?

-Josh

-----Original Message-----
From: Mark McLoughlin <markmc at redhat.com>
Reply-To: "OpenStack Development Mailing List (not for usage questions)"
<openstack-dev at lists.openstack.org>
Date: Tuesday, June 10, 2014 at 2:19 PM
To: "OpenStack Development Mailing List (not for usage questions)"
<openstack-dev at lists.openstack.org>
Subject: Re: [openstack-dev] [oslo] versioning and releases

>On Tue, 2014-06-10 at 12:24 -0400, Doug Hellmann wrote:
>> As part of the push to release code from the oslo incubator in
>> stand-alone libraries, we have had several different discussions about
>> versioning and release schedules. This is an attempt to collect all of
>> the decisions we have made in those discussions and to lay out the
>> rationale for the approach we are taking. I don't expect any of this
>> to be a surprise, since we've talked about it, but I haven't actually
>> written it all down in one place before so some of you might not have
>> seen all of the points. Please let me know if you see any issues with
>> the proposal or have questions. If everyone agrees that this makes
>> sense, I'll put it in the wiki.
>
>Nice work Doug, this is looking great.
>
>> Background:
>> 
>> We have two types of oslo libraries. Libraries like oslo.config and
>> oslo.messaging were created by extracting incubated code, updating the
>> public API, and packaging it. Libraries like cliff and taskflow were
>> created as standalone packages from the beginning, and later adopted
>> by the oslo team to manage their development and maintenance.
>> 
>> Incubated libraries have been released at the end of a release cycle,
>> as with the rest of the integrated packages. Adopted libraries have
>> historically been released "as needed" during their development. We
>> would like to synchronize these so that all oslo libraries are
>> officially released with the rest of the software created by OpenStack
>> developers.
>> 
>> The first release of oslo.config was 1.1.0, as part of the grizzly
>> release. The first release of oslo.messaging was 1.2.0, as part of the
>> havana release. oslo.config was also updated to 1.2.0 during havana.
>> All current adopted libraries have release numbers less than 1.0.0.
>> 
>> In the past, alpha releases of oslo libraries have been distributed as
>> tarballs on an openstack server, with official releases going to PyPI.
>> Applications that required the alpha release specified the tarball in
>> their requirements list, followed by a version specifier. This allowed
>> us to prepare alpha releases, without worrying that their release
>> would break continuous-deployment systems by making new library
>> releases available to pip. This approach still made it difficult for
>> an application developer to rely on new features of an oslo library,
>> until an alpha version was produced.
>> 
>> When the PyPI mirror was introduced in our CI system, relying on
>> tarballs not available on the mirror conflicted with our desire to
>> have the gate system install *only* from the package mirror. As we are
>> now installing only from the mirror, we need to publish our alpha
>> releases in a format that will work with the mirror.
>> 
>> We already gate OpenStack applications and oslo libraries using
>> integration tests using the normal devstack-gate jobs. During Icehouse
>> we had a couple of oslo library releases that broke unit tests of
>> applications after the library was released. We plan to address that
>> with separate gating jobs during juno. In addition to that gating, we
>> need to support developers who want to use new features of oslo
>> libraries before official releases are available.
>> 
>> A Version Numbering Scheme:
>> 
>> At the Juno summit, Josh proposed that we use semantic versioning
>> (SemVer) for oslo libraries [1]. Part of that proposal also included
>> ideas for allowing breaking backwards compatibility at some release
>> boundaries, and I am explicitly *not* addressing
>> backwards-incompatible changes beyond saying that we do not expect to
>> have any during Juno. We do need to solve the problem of breaking API
>> compatibility, but I want to take one step at a time. The first step
>> is choosing a rational release versioning scheme.
>> 
>> SemVer is widely used and gives us relatively clear guidelines about
>> choosing new version numbers. It supports alpha releases, which are
>> going to be key to meeting some of our other requirements. I propose
>> that we adopt pbr's modified SemVer [2] for new releases, beginning
>> with Juno.
>> 
>> The versions for existing libraries oslo.config and oslo.messaging
>> will be incremented from their Icehouse versions but updating the
>> minor number (1.x.0) at the end of the Juno cycle.
>> 
>> All adopted libraries using numbers less than 1.0 will be released as
>> 1.0.0 at the end of the Juno cycle, based on the fact that we expect
>> deployers to use them in production.
>
>Perhaps not directly related to this proposal, but IMHO OpenStack should
>not depend on any semver-versioned libraries which are still on 0.x.
>
>i.e. we should not depend on API unstable projects (because of the pain
>it imposes on everyone), particularly ones that are likely to feel
>empowered to break their API at will because of their version number :)
>
>> Releases during Juno should *all* be marked as alphas of the
>> anticipated upcoming SemVer-based release number (1.0.0.0aN or
>> 1.2.0.0aN or whatever).
>
>Why not simply 1.2.0aN btw?
>
>>  The new CI system can create packages as
>> Python wheels and publish them to the appropriate servers, which means
>> projects will no longer need to refer explicitly to pre-release
>> tarballs.
>
>The details are a bit more nuanced here - pip won't install alpha
>libraries unless you explicitly request them with a command line flag to
>install any alphas available or you explicitly require the alpha
>version.
>
>The wheels aspect is that pip <= 1.3 didn't support this, but also
>didn't support wheels ... so we publish alphas only as wheels to ensure
>that older pips don't see them.
>
>> Releases after Juno will follow a similar pattern, incrementing the
>> minor number and using alpha releases within the development cycle.
>> 
>> [1] https://etherpad.openstack.org/p/juno-oslo-semantic-versioning
>> [2] http://docs.openstack.org/developer/pbr/semver.html
>> 
>> Frequent Alpha Releases:
>> 
>> While we can run gate jobs using the master branch of oslo libraries,
>> developers will have to take extra steps to run unit tests this way
>> locally. To reduce this process overhead, while still ensuring that
>> developers use current versions of the code, we should produce alpha
>> releases of libraries during the release cycle fairly frequently. I
>> suggest that we start with a weekly check-up during the oslo team
>> meetings, and that we tag releases early on Mondays when deemed
>> necessary.
>> 
>> Patch Releases:
>> 
>> Updates to existing library releases can be made from stable branches.
>> Checking out stable/icehouse of oslo.config for example would allow a
>> release 1.3.1. We don't have a formal policy about whether we will
>> create patch releases, or whether applications are better off using
>> the latest release of the library. Do we need one?
>
>I'm not sure we need one, but if we did I'd expect them to be aligned
>with stable releases.
>
>Right now, I think they'd just be as-needed - if there's enough
>backported to the stable branch to warrant a release, we just cut one.
>
>> We do not typically use upper bounds on the requirements
>> specifications for oslo libraries, so new releases may automatically
>> be adopted by continuous-deployment systems building packages for
>> stable branches of OpenStack applications. Although we have been
>> careful about API compatibility in the past, there is a chance that a
>> new release could break an older application. Applications could add
>> an upper bound using SemVer numbering if they choose, although that
>> may prevent them from seeing bug fixes.
>
>I think applications should only add an upper bound on the major number
>so they don't see API breakage e.g.
>
>  oslo.config>=1.2.0,<2
>
>If applications have an upper bound on the minor number, that indicates
>a lack of confidence in our ability to add new features without breaking
>the API.
>
>> Unit test gate jobs:
>> 
>> We have a blueprint for Juno to add cross-project unit test gating for
>> applications and oslo libraries [3]. This will allow us to verify that
>> tests for applications do not break when oslo libraries change, but
>> also that those tests do not make assumptions about oslo library
>> implementation details.
>> 
>> Depending on how the job is implemented, we may be able to reuse the
>> tools to let developers run the same tests locally. More thought is
>> needed here; stay tuned.
>> 
>> [3] 
>>https://blueprints.launchpad.net/oslo/+spec/enhance-cross-test-gate-job
>
>Very cool stuff!
>
>Mark.
>
>
>_______________________________________________
>OpenStack-dev mailing list
>OpenStack-dev at lists.openstack.org
>http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev