Open Stack

[Note - I understand there are ongoing discussion that may lead to a 
proposal for an out-of-tree incubation process for new Neutron features. 
This is a complementary proposal that describes how our existing 
development process can be used to stabilize new features in-tree over 
the time frame of a release cycle or two. We should fully consider both 
proposals, and where each might apply. I hope something like the 
approach I propose here will allow the implementations of Neutron BPs 
with non-trivial APIs that have been targeted for the Juno release to be 
included in that release, used by early adopters, and stabilized as 
quickly as possible for general consumption.]

According to our existing development process, once a blueprint and 
associated specification for a new Neutron feature have been reviewed, 
approved, and targeted to a release, development proceeds, resulting in 
a series of patches to be reviewed and merged to the Neutron source 
tree. This source tree is then the basis for milestone releases and the 
final release for the cycle.

Ideally, this development process would conclude successfully, well in 
advance of the cycle's final release, and the resulting feature and its 
API would be considered fully "stable" in that release. Stable features 
are ready for widespread general deployment. Going forward, any further 
modifications to a stable API must be backwards-compatible with 
previously released versions. Upgrades must not lose any persistent 
state associated with stable features. Upgrade processes and their 
impact on a deployments (downtime, etc.) should be consistent for all 
stable features.

In reality, we developers are not perfect, and minor (or more 
significant) changes may be identified as necessary or highly desirable 
once early adopters of the new feature have had a chance to use it. 
These changes may be difficult or impossible to do in a way that honors 
the guarantees associated with stable features.

For new features that effect the "core" Neutron API and therefore impact 
all Neutron deployments, the stability requirement is strict. But for 
features that do not effect the core API, such as services whose 
deployment is optional, the stability requirement can be relaxed 
initially, allowing time for feedback from early adopters to be 
incorporated before declaring these APIs stable. The key in doing this 
is to manage the expectations of developers, packagers, operators, and 
end users regarding these new optional features while they stabilize.

I therefore propose that we manage these expectations, while new 
optional features in the source tree stabilize, by clearly labeling 
these features with the term "preview" until they are declared stable, 
and sufficiently isolating them so that they are not confused with 
stable features. The proposed guidelines would apply during development 
as the patches implementing the feature are first merged, in the initial 
release containing the feature, and in any subsequent releases that are 
necessary to fully stabilize the feature.

Here are my initial not-fully-baked ideas for how our current process 
can be adapted with a "preview feature" concept supporting in-tree 
stabilization of optional features:

* Preview features are implementations of blueprints that have been 
reviewed, approved, and targeted for a Neutron release. The process is 
intended for features for which there is a commitment to add the feature 
to Neutron, not for experimentation where "failing fast" is an 
acceptable outcome.

* Preview features must be optional to deploy, such as by configuring a 
service plugin or some set of drivers. Blueprint implementations whose 
deployment is not optional are not eligible to be treated as preview 
features.

* Patches implementing a preview feature are merged to the the master 
branch of the Neutron source tree. This makes them immediately available 
to all direct consumers of the source tree, such as developers, 
trunk-chasing operators, packagers, and evaluators or end-users that use 
DevStack, maximizing the opportunity to get the feedback that is 
essential to quickly stabilize the feature.

* The process for reviewing, approving and merging patches implementing 
preview features is exactly the same as for all other Neutron patches. 
The patches must meet HACKING standards, be production-quality code, 
have adequate test coverage, have DB migration scripts, etc., and 
require two +2s and a +A from Neutron core developers to merge.

* DB migrations for preview features are treated similarly to other DB 
migrations, forming a single ordered list that results in the current 
overall DB schema, including the schema for the preview feature. But DB 
migrations for a preview feature are not yet required to preserve 
existing persistent state in a deployment, as would be required for a 
stable feature.

* All code that is part of a preview feature is located under 
neutron/preview/<feature>/. Associated unit tests are located under 
neutron/tests/unit/preview/<feature>/, and similarly for other test 
categories. This makes the feature's status clear to developers and 
other direct consumers of the source tree, and also allows packagers to 
easily partition all preview features or individual preview features 
into separate optionally installable packages.

* The tree structures underneath these locations should make it 
straightforward to move the preview feature code to its proper tree 
location once it is considered stable.

* Tempest API and scenario tests for preview features are highly 
desirable. We need to agree on how to accomplish this without preventing 
necessary API changes. Posting WIP patches to the Tempest project may be 
sufficient initially. Putting Tempest-like tests in the Neutron tree 
until preview features stabilize, then moving them to Tempest when 
stabilization is complete, might be a better long term solution.

* No non-preview Neutron code should import code from anywhere under the 
neutron.preview module, unless necessary for special cases like DB 
migrations.

* URIs for the resources provided by preview features should contain the 
string "preview".

* Configuration file content related to preview features should be 
clearly labeled as "preview".

* Preview features should be documented similarly to any stable Neutron 
feature, but documents or sections of documents related to preview 
features should have an easily recognizable label that clearly 
identifies the feature as a "preview".

* Support for preview features in client libraries, and in other 
projects such as Horizon, Heat, and DevStack, are essential to get the 
feedback needed from early adopters during feature stabilization. They 
are implemented normally, but should be labeled "preview" appropriately, 
such as in GUIs, in CLI help strings and in documentation so that end 
user expectations regarding stability are managed.

* A process is needed to prevent long-term stagnation of features in the 
preview sub-tree. It is reasonable to expect a new feature to remain for 
one or two cycles, possibly with little change (other than bug fixes), 
before stabilizing. A suggested rule is that a new approved BP is 
required after two cycles, or the feature gets removed from the Neutron 
source tree (maybe moved (back) to an incubation repository).


I would appreciate feedback via this email thread on whether this 
"preview feature" concept is worth further consideration, refinement and 
potential usage for approved feature blueprints, especially during the 
Juno cycle. I've also posted the proposal text at 
https://etherpad.openstack.org/p/neutron-preview-features for those 
interested in helping refine the proposal.

Thanks,

-Bob