Open Stack

On 16/06/2016 00:30, Matthew Treinish wrote:
> On Wed, Jun 15, 2016 at 09:10:30AM -0400, Doug Hellmann wrote:
>> Excerpts from Chris Hoge's message of 2016-06-14 16:37:06 -0700:
>>> Top posting one note and direct comments inline, I’m proposing
>>> this as a member of the DefCore working group, but this
>>> proposal itself has not been accepted as the forward course of
>>> action by the working group. These are my own views as the
>>> administrator of the program and not that of the working group
>>> itself, which may independently reject the idea outside of the
>>> response from the upstream devs.
>>>
>>> I posted a link to this thread to the DefCore mailing list to make
>>> that working group aware of the outstanding issues.
>>>
>>>> On Jun 14, 2016, at 3:50 PM, Matthew Treinish <mtreinish at kortar.org> wrote:
>>>>
>>>> On Tue, Jun 14, 2016 at 05:42:16PM -0400, Doug Hellmann wrote:
>>>>> Excerpts from Matthew Treinish's message of 2016-06-14 15:12:45 -0400:
>>>>>> On Tue, Jun 14, 2016 at 02:41:10PM -0400, Doug Hellmann wrote:
>>>>>>> Excerpts from Matthew Treinish's message of 2016-06-14 14:21:27 -0400:
>>>>>>>> On Tue, Jun 14, 2016 at 10:57:05AM -0700, Chris Hoge wrote:

<snip>

>>> The current active guidelines cover icehouse through mitaka. The release
>>> of 2016.08 will change that to cover juno through mitaka (with newton
>>> as an add-on to 2016.08 when it’s released). There’s overlap between
>>> the guidelines, so 2016.01 covers juno through mitaka while 2016.08
>>> will cover kilo through newton. Essentially two years of releases.
>>>
>>>>> We may also need to consider that test implementation details may
>>>>> change, and have a review process within DefCore to help expose
>>>>> those changes to make them clearer to deployers.
>>>>>
>>>>> Fixing the process issue may also mean changing the way we implement
>>>>> things in Tempest. In this case, adding a flag helps move ahead
>>>>> more smoothly. Perhaps we adopt that as a general policy in the
>>>>> future when we make underlying behavioral changes like this to
>>>>> existing tests.  Perhaps instead we have a policy that we do not
>>>>> change the behavior of existing tests in such significant ways, at
>>>>> least if they're tagged as being used by DefCore. I don't know --
>>>>> those are things we need to discuss.
>>>>
>>>> Sure I agree, this thread raises larger issues which need to be figured out.
>>>> But, that is probably an independent discussion.
>>>
>>> I’m beginning to wonder if we need to make DefCore use release
>>> branches then back-port bug-fixes and relevant features additions
>>> as necessary.

What I suggested when the TC decided on keeping the tests in tempest
was to keep them as a tempest plugin.

This would allow the tests to progress overtime, and be versioned,
so that def-core 201x.y would be equal to def-core-tempest-plugin
version 201x.y .

This allows the project developers to continue to develop tests that
match their vision, without causing unforeseen breakages in def-core.

It also allows the def-core tests to target a wider range - tests
that are run currently have no guarantee that they will run against
Kilo, but the def-core plugin could be tested against Kilo known good
clouds in its gate.

Is there any major blocker for moving them?

>> We should definitely have that conversation, to understand what
>> effect it would have both on Tempest and on DefCore.
>>
>
> While from a quick glance this would seem like it would solve some of the
> problems when you start to dig into it you'll see that it actually wouldn't,
> and would just end up causing more issues in the long run. Branchless tempest
> was originally started back at the icehouse release and was implemented to
> actually enforce the API is the same across release boundaries. We had hit many
> issues where incompatibilities inadvertently were introduced into projects' APIs
> which weren't caught because of divergence between tempest master and tempest
> stable/*. If you decide to branch you'll end up having to do the same thing or
> you'll risk the same types of regressions. Testing stable branches against
> master puts you in a weird place where upstream changes could break things for
> your branch and you'd never know until you tried to land something. From a
> tempest dev standpoint branching quite frankly doesn't make any sense.
>
> Also, the other thing to consider is that there wouldn't actually be anything to
> branch on. Tempest always supports whatever releases the community is
> supporting. Every commit is tested against all the branches of OpenStack that
> still exist. When we EOL a stable branch there is no longer anything to run
> tests against. Assuming you're primarily motivated by the fact defcore attempts
> to support branches that no longer have upstream support you wouldn't actually
> be able to do this by branching. When a branch is removed there isn't anything
> for you to test tempest changes against, and merging code without testing it
> first is a terrible idea and the opposite of how we do things in the openstack
> community.
>
>
> -Matt Treinish
>