Open Stack

Now that the branchless tempest idea is gaining steam, we have to 
address this issue along with how we handle releases that are not longer 
supported upstream. The current state of running master tempest against 
stable havana can be seen in the bottom two jenkins entries at 
https://review.openstack.org/#/c/86416/. The number of failures is 
manageable and they are due to a variety of issues including:

1. Changes to returned status failure codes
2. dictionary changes that are being checked by schemas now but were not 
in havana
3 havana failures that still need to bee investigated

I would really like to see master running against havana in the gate 
which would provide value to the refstack effort as well as almost any 
one who is not doing CD.
I proposed a "grandfathering" of havana by putting a skip decorator for 
that in master so we could get this in the gate and then figure out how 
much effort to put in getting more tests to run. This would also be a 
mechanism to allow upcoming work to not get tripped up on havana issues 
if we decide that is not worth it. Sean and Matt have objected to this 
idea. I really don't care that much exactly how we allow master to run 
against havana as long as the mechanism is in-tree. How do other folks 
feel about this, and are there other implementation suggestions?

Another related issue is what happens when a stable release becomes 
"unsupported" given that many OpenStack users will continue to run 
"unsupported" releases. For example, there may be a desire to remove xml 
support in nova. When that happens, any one who still cares about 
running tempest against an older release will have to fork tempest. That 
is ok but there could be sharing opportunities. IIRC, this was the 
original motivation behind stable branches in the first place which was 
a little controversial at the time.

  -David