Open Stack

Hi everyone,

with recent disruption in our CI process I'd like to discuss again the
issues in our merge workflow.

See the summary at the end.


As a starting point, here is the list of patches which were merged
into fuel-library repository without "Verified +1" label from Fuel CI:

https://review.openstack.org/#/q/project:stackforge/fuel-library+AND+status:merged+AND+NOT+label:Verified%252B1%252Cuser%253Dfuel-ci,n,z

And the list of merged patches with explicit "Verified -1" label:

https://review.openstack.org/#/q/project:stackforge/fuel-library+AND+status:merged+AND+label:Verified-1%252Cuser%253Dfuel-ci,n,z

There are two common reasons I know why these patchsets exist:

Case 1: "Who cares about CI anyway".

Case 2: These patches can not pass CI because of some real reason,
which makes Fuel CI result irrelevant.

I am not sure, if i need to comment on the first one, but please just
remember: CI is not a devops playground made to disrupt your otherwise
clean and smooth development process. It is an extremely critical
service, providing the clear reference point for all the work we do.
And we all know how important the reference point is [1].

So let's move on to the Case 2 and talk about our CI limitations and
what could possibly make the test result irrelevant.

1) Dependencies.

Let's say you have a chain of dependent patchsets and none of them
could pass the CI on its own. How do you merge it?

Here is the trick: the "leaf", i.e. last, topmost patch in the chain
should pass the CI.

The test we run for this patchset automatically pulls all dependencies
involved. Which makes Fuel CI result for this patchset perfectly
relevant for the whole chain.

2) Environment.

Fuel CI test environment usually uses slightly outdated version of
Fuel iso image and fuel-main code. Therefore it happens that you write
and test your patch against latest code via custom iso builds and it
works, but it can not pass CI. Does it make test results irrelevant?
No. It makes them even more valuable.

CI environment can be broken and can be outdated. This is the part of
the process. To deal with these situations we first need to fix the
environment, then run tests, and then merge the code.

And it helps if you contact devops team in advance  and inform us that
you soon will need the ISO with this particular features.

3) ?

Please add your examples and let's deal with them one by one.


Summary:

I'd like to propose the following merge policy:

1. any individual patchset MUST have +1 from Fuel CI;

2. any chain of dependent patchsets MUST have +1 from Fuel CI for the
topmost patch;

3. for all exceptional cases the person who does the merge MUST
explicitly contact devops team, and make sure that there will be
devops engineer available who will run additional checks before or
right after the merge. The very same person who does the merge also
MUST be available for some time after the merge to help the devops
engineer to deal with the test failures if they appear.



[1] http://www.youtube.com/watch?feature=player_embedded&v=QkCQ_-Id8zI#t=211


-- 
Aleksandra Fedorova
Fuel Devops Engineer
bookwar