Open Stack

(Apologies, this started on the TC list, and really should have started
on -dev, correctly posting here now for open discussion)

There were a few chats at summit about this, mostly on the infra /
devstack / qa side of the house. Consider the following a straw man to
explain the current state of the world, and what I'd like to see change here
I call out projects by name here, not to
make fun of them, but that I think concrete examples bring the point
home much more than abstract arguments (at least for me).

This is looking at raising the bar quite a bit along the way. However,
as someone that spends a lot of time trying to keep the whole ball of
wax holding together, and is spending a lot of time retroactively trying
to get projects into our integrated gate (and huge pain to everyone, as
their gate commits get bounced by racey projects), I think we really
need to up this bar if we want a 20 integrated project version of
OpenStack to hold together.


=====================================================
 Proposed new Incubation and Graduation Requirements
=====================================================


The Current State of the World
==============================
The integrated gate in OpenStack is a set of devstack / tempest tests
which run symmetrically between projects. I.e. we run all the tempest
tests (nova, keystone, swift, etc.), on a change to cinder. That means
that nova can ensure that a cinder change that would break them will
be prevented from landing. An example being the
gate-tempest-devstack-vm-full job.

OpenStack is currently 9 integrated projects in Havana, 10 once we get
to Icehouse (+Trove).

Heat added tests to our integrated gate during the Havana cycle,
though the deeper guest testing is not part of our gate. Ceilometer
has been integrated for 2 releases, and doesn't have integrated gate
testing. Trove is just now exploring how they'd get into the gate for
Icehouse release.

Upgrade testing situation is worse. Only 5 projects currently are part
of upgrade testing: Nova, Cinder, Swift, Glance, Keystone. The rest
are not set up in that model at all, so we have no infrastructure to
ensure the other projects don't break upgrades of themselves.

Heat, Trove, and Savana represent an interesting challenge from
current gate models in that real validation requires something beyond
a trivial guest, as they care about the contents inside compute
resources. This desire is likely to grow with other "Layer 4" projects
coming into the OpenStack ecosystem [1].

Ironic as an incubated project provides an even different set of
challenges, as not only do we not have a gating approach, but we also
really should have an approach to test an upgrade from nova-baremetal
=> ironic to ensure there is a migration path for people in the
future.

Ceilometer once relied on a version of MongoDB that worked in the
gate, but they were never gating, so now they rely on a version of
MongoDB that doesn't work in the gate. They made a technical decision
to upgrade requirements with no gate feedback because they weren't
actually integration testing on a regular basis.



Proposed Incubation requirements
================================
Once something becomes an integrated project, it's important that they
are able to run in the gate.

Both devstack and devstack-gate now support hooks, so with a couple of
days of work any project in stackforge could build a gate job which
sets up a devstack of their configuration, including their code,
running some project specific test they feel is appropriate to ensure
they could run in the gate environment.

This would ensure an incubated project works with OpenStack global
requirements, or if it requires something new, that's known very
clearly before incubation.


Proposed Graduation requirements
================================
All integrated projects should be in the integrated gate, as this is
the only way we provably know that they can all work together, at the
same level of requirements, in a consistent way.

During incubation landing appropriate tests in Tempest is fair
game. So the expectation would be that once a project is incubated
they would be able to land tests in tempest. Before integrated we'd
need to ensure the project had tests which could take part in the
integrated gate, so as soon as a project is voted integrated, it has
some working integrated gate tests. (Note: there is actually a
symmetric complexity here, to be worked out later).


Proposed Stable Release requirements
====================================
We have this automatic transition that happens when a project that's
integrated for a release, actually releases as part of
that. I.e. Trove and Icehouse. There is no additional TC decision
about whether or not Trove is part of the stable release, once
integrated, it just is. Nothing that it does over that cycle will kick
it out of the stable release. This is one of the reasons it needs to
be in the integrated gate **before** graduation.

Additionally, upgrade path is critically important to our users, and
the number one piece of feedback we received from the User Survey. It
was also important enough to our developers that it was scattered all
over the Icehouse Design Summit. All integrated projects should be
included in upgrade testing the moment they are in a stable
release. (ex: when Icehouse is released, Trove should be in master
grenade, and upgrade testing from Icehouse -> master for the J cycle
from day one).



TL;DR
=====
- To get to incubated you must have a devstack-gate job
- To get to integrated you must have useful tests in the integrated
  gate
- To get into the stable/release you must have upgrade testing enabled

Raised Questions
================
- what about existing incubated projects, what would be their time
  frame to get with this new program
- what about existing integrated projects that currently don't exist
  with either an upgrade or gate story?
- what about an upgrade deprecation path (i.e. nova-network =>
  neutron, nova-baremetal => ironic)



1.
http://hackstack.org/x/blog/2013/09/05/openstack-seven-layer-dip-as-a-service/
   (the layers pattern is useful from a testing requirements model)


-- 
Sean Dague
http://dague.net

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 482 bytes
Desc: OpenPGP digital signature
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20131113/4fbbb71e/attachment.pgp>