Open Stack

Hi everyone,

The starting point for this post is a specific Neutron sub-project 
(networking-bgpvpn) but I believe the issues raised are shared with 
other neutron stadium project and possibly relevant beyond Neutron, to 
projects tagged release-independent in general.

In the context of the networking-bgpvpn project, we have a few unsolved 
issues related to branches, more specifically about which branches we 
can create to host active development to make our subprojet (a Neutron 
service plugin) work with the last Openstack release and to backport it 
with to earlier releases.

Here are precisely the assumptions that we've made, largely based on the 
fact that the project is tagged 'release-independent' (meaning 
"completely bypass the 6-month cycle and release independently" [1]):
a) that we can make a release targeting Liberty much later than the 
release date of Liberty
b) that we could make releases more frequent than the 6-month cycle ; 
not only bugfix releases but also feature releases
c) that the idea of a release-independent project being backported to 
work with past Openstack releases is acceptable (of course, without 
requiring any change in release-managed projects, something largely 
possible in many cases for projects such as a Neutron service plugin or 
an ML2 driver)

Note that we aren't the only big tent project having this kind of 
expectations (e.g. [3]).

The rest of this email follows from these assumptions.

To do (a) and (c) we need separate branches in which to do active work.
We have understood clearly that given the contract around 'stable/*' 
branches, the branches in which to do this active work cannot be named 
'stable/*'.

Where we are today:
- we do active development, planning a liberty release soon, on our 
'master' branch
- we have a 'backport/kilo' and a 'backport/juno' branches in which we 
plan to make the necessary changes to adapt with Neutron kilo and juno; 
these branches were created by Neutron core devs for us [5], based on 
the common understanding that choosing 'stable/*' branch names for 
branches with active development would be a huge no no

Let me take a few example of the issues we hit...

I apologize in advance for imprecisions that may result from my limited 
experience with the Openstack project at large, and CI subtleties in 
particular.

### Continuous Integration issue

Our master branch is written to work with stable/liberty of other 
projects we depend on (neutron, python-neutronclient, devstack for the 
CI devstack VMs).  The issue we hit, at least this is my understanding, 
is that the CI system (at least the devstack VM parts) has a built-in 
implicit assumption that branch X of project foo is meant to work with 
branch X of project bar, but that this assumption is not true in our case.

The solution we could consider is to tweak CI gate jobs to add jobs that 
force a specific branch to be used for Openstack key project our job 
depend on (e.g. devstack, neutron, in our case), and filter which of 
these jobs would be triggered for each specific branch of our project.

For instance:
- define a gate-install-dsvm-networking-bgpvpn-kilo with branch-override 
stable/kilo  (will result in creating a devstack stable/kilo with 
Neutron stable/kilo inside)
- configure zuul to apply this job *only if* the gerrit change 
triggering the test is for our backport/kilo branch
(do that again for our juno branch, and again to map our master to 
Openstack stable/liberty)

It seems like it would work, although possible too verbose.

With the infra team support we already have something close to this 
configuration in place for our master (two jobs, one to test with 
master, one to force testing against stable/liberty). [4]

### Synchronizing requirements issue

Trying to be good pupils we had activated the check-requirements jobs in 
our zuul config, and the requirements proposal bot as well.
Just like the CI system, these requirements tools have a built-in 
implicit assumption that branch X of project foo is meant to work with 
branch X of project bar, which breaks in our case.

The issues we have are the following:
1- the botis proposing on our 'master' branch, updates to 
requirements.txt to make us in sync with the requirements 'master' 
branch, which is irrelevant since our master in facts targets Liberty  
-- this is easily "solved": we just ignore those proposed changes
2- the bot won't be able to propose anything relevant on our 
backport/kilo and backport/juno branches, since he does not know they 
target stable/kilo and stable/juno  -- we can live with that and update 
requirements ourselves
3- we can't update our requirements.txt on our master branch: we want to 
add requirements conforming with Liberty, but the check-requirements 
jobs thinks this is wrong since it checks our master branch against its 
master branch (that now has mitaka requirements)
4- same as 3 for our backport/kilo and backport/juno branches (the job 
does not know these branch should be checked against the requirements in 
stable/kilo, resp. stable/juno)

Note that 3 and 4 are **blocker issues**: they prevent us form merging 
any requirements file update to prepare a first Liberty-compatible 
release or for backports.

The temporary solution we've investigated, for lack of a better one, is 
to make the check-requirements job non-voting.
That would make our project further from being a good pupil than we 
would like to, but on the other hand a large majority of Openstack 
projects do not enable these automated checks either.
We have received negative reviews on this [2], from both Infra 
contributors and Armando as Neutron PTL ; my understanding being that 
this opposition is founded on the idea that the way we plan to do 
development (cf assumptions at the start of this email) is not inline 
with widespread practices (read [2] if you want to interpret by yourself).

### Doing multiple releases inside one 6-month Openstack cycle issue

Our initial plan was to fork a 'stable/liberty' branch from our master 
at the same time as our first release.
But after this we don't know how to work on new features for a release 
still targeting Liberty:
- adding features on stable/liberty is  a no no
- there is no established practice, as far as we know, to name such 
branches, nor to track that they aim to work with Liberty

We haven't a solution for that right now, and we definitely want guidance.

A related question is when/how to start working on code targeting mitaka .
There are different things we want to take into account:
- there are many things we can do without requiring anything new from 
Neutron
- in an ideal world, the interfaces between Neutron and service plugins 
would be stable enough
- in practice we expect that we'll have to follow some changes in 
Neutron internals that will arrive in Mitaka release

Based on the above, one possibility would maybe be to have 
release/[major-version] branches forked from master from each of our 
Liberty-targeted release,
the last of these branches could become our stable/liberty release and 
at the same time our trunk would start targeting Mitaka-compatibility.

### Other issues ...

(there may even be other issues we haven't seen yet related to the same 
root question on how to branch and preserve clear

### Very Humble Suggestions

Based on the above, I have a few suggestions... (to take as humble 
suggestions from someone relatively new to the Openstack community 
practices...).

It seems that part of the issue is that the only way CI tools have to 
know what Openstack branch a said branch of a project is targeted is the 
implicit assumption that branches have the same name.  Something that 
may help would be to let a subproject indicate in a file in its repo 
what Openstack branch it is targeting (e.g. the 
'openstack-target-branch.txt' file in the 'master' branch of 
networking-bgpbpn could say 'stable/liberty' and the CI tools would know 
have the information they need; this file would be optional with the 
current implicit behavior being applied by default in the case where it 
does not exist). This kind of information could be used by tools to 
consolidate where the different projects stand with reference to what 
Openstack release they target.

Another thing that might be considered is to accept active development 
in a 'stable/*' branch **until** a first release is tagged on this 
branch, and then apply stable branches standard of accepting bugfixes 
only.   This would however not give an answer on how to do multiple 
feature release targeting a same Openstack major release.

### ...

If our assumptions are legitimate, then we need guidance and possibly 
changes in tools or practices to proceed.
(if our assumptions are not legitimate, then discussing why will be a 
very useful discussion for us as well!)

Thanks for reading this far...
Hoping this email will be help...

-Thomas


[1] http://governance.openstack.org/reference/tags/release_independent.html
[2] https://review.openstack.org/#/c/243610/
[3] 
http://lists.openstack.org/pipermail/openstack-dev/2015-November/078676.html
[4] https://review.openstack.org/#/c/233703/
[5] https://bugs.launchpad.net/bgpvpn/+bug/1513870