[openstack-dev] [oslo] versioning and releases

Doug Hellmann doug.hellmann at dreamhost.com
Wed Jun 11 14:32:06 UTC 2014

On Tue, Jun 10, 2014 at 5:19 PM, Mark McLoughlin <markmc at redhat.com> wrote:
> On Tue, 2014-06-10 at 12:24 -0400, Doug Hellmann wrote:
>> As part of the push to release code from the oslo incubator in
>> stand-alone libraries, we have had several different discussions about
>> versioning and release schedules. This is an attempt to collect all of
>> the decisions we have made in those discussions and to lay out the
>> rationale for the approach we are taking. I don't expect any of this
>> to be a surprise, since we've talked about it, but I haven't actually
>> written it all down in one place before so some of you might not have
>> seen all of the points. Please let me know if you see any issues with
>> the proposal or have questions. If everyone agrees that this makes
>> sense, I'll put it in the wiki.
> Nice work Doug, this is looking great.
>> Background:
>> We have two types of oslo libraries. Libraries like oslo.config and
>> oslo.messaging were created by extracting incubated code, updating the
>> public API, and packaging it. Libraries like cliff and taskflow were
>> created as standalone packages from the beginning, and later adopted
>> by the oslo team to manage their development and maintenance.
>> Incubated libraries have been released at the end of a release cycle,
>> as with the rest of the integrated packages. Adopted libraries have
>> historically been released "as needed" during their development. We
>> would like to synchronize these so that all oslo libraries are
>> officially released with the rest of the software created by OpenStack
>> developers.
>> The first release of oslo.config was 1.1.0, as part of the grizzly
>> release. The first release of oslo.messaging was 1.2.0, as part of the
>> havana release. oslo.config was also updated to 1.2.0 during havana.
>> All current adopted libraries have release numbers less than 1.0.0.
>> In the past, alpha releases of oslo libraries have been distributed as
>> tarballs on an openstack server, with official releases going to PyPI.
>> Applications that required the alpha release specified the tarball in
>> their requirements list, followed by a version specifier. This allowed
>> us to prepare alpha releases, without worrying that their release
>> would break continuous-deployment systems by making new library
>> releases available to pip. This approach still made it difficult for
>> an application developer to rely on new features of an oslo library,
>> until an alpha version was produced.
>> When the PyPI mirror was introduced in our CI system, relying on
>> tarballs not available on the mirror conflicted with our desire to
>> have the gate system install *only* from the package mirror. As we are
>> now installing only from the mirror, we need to publish our alpha
>> releases in a format that will work with the mirror.
>> We already gate OpenStack applications and oslo libraries using
>> integration tests using the normal devstack-gate jobs. During Icehouse
>> we had a couple of oslo library releases that broke unit tests of
>> applications after the library was released. We plan to address that
>> with separate gating jobs during juno. In addition to that gating, we
>> need to support developers who want to use new features of oslo
>> libraries before official releases are available.
>> A Version Numbering Scheme:
>> At the Juno summit, Josh proposed that we use semantic versioning
>> (SemVer) for oslo libraries [1]. Part of that proposal also included
>> ideas for allowing breaking backwards compatibility at some release
>> boundaries, and I am explicitly *not* addressing
>> backwards-incompatible changes beyond saying that we do not expect to
>> have any during Juno. We do need to solve the problem of breaking API
>> compatibility, but I want to take one step at a time. The first step
>> is choosing a rational release versioning scheme.
>> SemVer is widely used and gives us relatively clear guidelines about
>> choosing new version numbers. It supports alpha releases, which are
>> going to be key to meeting some of our other requirements. I propose
>> that we adopt pbr's modified SemVer [2] for new releases, beginning
>> with Juno.
>> The versions for existing libraries oslo.config and oslo.messaging
>> will be incremented from their Icehouse versions but updating the
>> minor number (1.x.0) at the end of the Juno cycle.
>> All adopted libraries using numbers less than 1.0 will be released as
>> 1.0.0 at the end of the Juno cycle, based on the fact that we expect
>> deployers to use them in production.
> Perhaps not directly related to this proposal, but IMHO OpenStack should
> not depend on any semver-versioned libraries which are still on 0.x.
> i.e. we should not depend on API unstable projects (because of the pain
> it imposes on everyone), particularly ones that are likely to feel
> empowered to break their API at will because of their version number :)
>> Releases during Juno should *all* be marked as alphas of the
>> anticipated upcoming SemVer-based release number ( or
>> or whatever).
> Why not simply 1.2.0aN btw?

I read Robert's pbr-semver spec
to mean that we needed the extra digit to comply with PEP-440, but
after looking at the PEP (http://legacy.python.org/dev/peps/pep-0440/)
I don't think we actually do.

>>  The new CI system can create packages as
>> Python wheels and publish them to the appropriate servers, which means
>> projects will no longer need to refer explicitly to pre-release
>> tarballs.
> The details are a bit more nuanced here - pip won't install alpha
> libraries unless you explicitly request them with a command line flag to
> install any alphas available or you explicitly require the alpha
> version.
> The wheels aspect is that pip <= 1.3 didn't support this, but also
> didn't support wheels ... so we publish alphas only as wheels to ensure
> that older pips don't see them.

Thanks, I'll include those details in the final write-up.

>> Releases after Juno will follow a similar pattern, incrementing the
>> minor number and using alpha releases within the development cycle.
>> [1] https://etherpad.openstack.org/p/juno-oslo-semantic-versioning
>> [2] http://docs.openstack.org/developer/pbr/semver.html
>> Frequent Alpha Releases:
>> While we can run gate jobs using the master branch of oslo libraries,
>> developers will have to take extra steps to run unit tests this way
>> locally. To reduce this process overhead, while still ensuring that
>> developers use current versions of the code, we should produce alpha
>> releases of libraries during the release cycle fairly frequently. I
>> suggest that we start with a weekly check-up during the oslo team
>> meetings, and that we tag releases early on Mondays when deemed
>> necessary.
>> Patch Releases:
>> Updates to existing library releases can be made from stable branches.
>> Checking out stable/icehouse of oslo.config for example would allow a
>> release 1.3.1. We don't have a formal policy about whether we will
>> create patch releases, or whether applications are better off using
>> the latest release of the library. Do we need one?
> I'm not sure we need one, but if we did I'd expect them to be aligned
> with stable releases.
> Right now, I think they'd just be as-needed - if there's enough
> backported to the stable branch to warrant a release, we just cut one.

That's pretty much what I thought, too. We shouldn't need to worry
about alphas for patch releases, since we won't add features.

>> We do not typically use upper bounds on the requirements
>> specifications for oslo libraries, so new releases may automatically
>> be adopted by continuous-deployment systems building packages for
>> stable branches of OpenStack applications. Although we have been
>> careful about API compatibility in the past, there is a chance that a
>> new release could break an older application. Applications could add
>> an upper bound using SemVer numbering if they choose, although that
>> may prevent them from seeing bug fixes.
> I think applications should only add an upper bound on the major number
> so they don't see API breakage e.g.
>   oslo.config>=1.2.0,<2
> If applications have an upper bound on the minor number, that indicates
> a lack of confidence in our ability to add new features without breaking
> the API.
>> Unit test gate jobs:
>> We have a blueprint for Juno to add cross-project unit test gating for
>> applications and oslo libraries [3]. This will allow us to verify that
>> tests for applications do not break when oslo libraries change, but
>> also that those tests do not make assumptions about oslo library
>> implementation details.
>> Depending on how the job is implemented, we may be able to reuse the
>> tools to let developers run the same tests locally. More thought is
>> needed here; stay tuned.
>> [3] https://blueprints.launchpad.net/oslo/+spec/enhance-cross-test-gate-job
> Very cool stuff!
> Mark.
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

More information about the OpenStack-dev mailing list