Open Stack

>
> Does anybody remember what the mechanism for dealing with this that was
agreed at the PTG was?

The etherpad [1] shows actions that we(me and Thomas) agree to do, and
that's all for PTG discussion (the online attendee number is 0 for that
session).
Following with a recap in meeting to share the task and include that
etherpad (which Rabi's agreement on fallow). And with further implement and
research (by Rabi), we face the test separation issue, as we discussed
yesterday, Rabi reaches out for suggestion and opinions,  and I gave mine
with agreeing to move all test to the new repo and giving my +2 review to
those patches.
IMO, that's a question of whether we consider our tests as part of tempest
plugin or we want to consider them as internal Heat CI tests.

Right now what we have is following:
1. A new repo for heat-tempest-plugin with all tests in it.
2. Switch heat's functional and grenade jobs (in master) to use new heat
tempest plugin.
3. a patch to propose to remove integration tests in heat.
4. needs a plan to match branchless for heat-tempest-plugin repo

>
> Basically we need a way to:
>
> * Run a bunch of integration tests in Heat CI jobs
> * against a full OpenStack cloud
> * preferably using tempest as a testrunner, but this is optional
> * without any danger of e.g. breaking tempest test discovery just because
Heat was installed on the same machine as tempest
>
> How do we do it?

We now use tempest for all our integration tests but seems the question now
is how can we let our the CI out of this process, so all developers in heat
will have a better life while users/ops will also have a better life when
testing against their OpenStack environment.

My propose is we separate our tests into two groups:
A group of tests that should only stay in Heat CI jobs, which we still can
use tempest as test runner, but we should mark it(with a description, a
tag, or any way) as for internal CI usage, which will not follow any TC
goals since that's only for heat internal usage.

Another group of tests that make sense for us to expose to users and
operators. These stay in heat tempest plugin which users/ops can use to
test their own environment, and we will meet what the users/ops might
expect when they testing with their own heat environment. Also means we
should keep this repo branchless and fully tested with each branch. So we
stick with that goal. API tests seem to be native in this group, but we
have to keep in mind that it's not covering all cases, so we might need to
consider to add more tests in tempest plugin (in long-term).

In turns of actions:

1. Create a new tempest plugin for internal CI usage (in heat repo) and
mark it as internal(as a declare for we do not consider this as a tempest
plugin for users/ops), so others will know that they should not use it for
any other way(or use at their own risk).
2. Modify the patch which proposes to remove integration tests in heat, and
keeps tests those tests we decided as internal CI tests.
3. Allow heat gate to run tests from both tempest plugin( internal one and
`heat-tempest-plugin`) so we will still run and check both works. Of
course, we can keep the same group of tests on both sides(like a sync job)
but don't see why that's the way we like.

We should have this discussion earlier (when a simple suggestion from any
one of us will do), but since we still under developing cycle that means
it's not too late for anything.
And TBH, as a developer(not a good one but still learning), I see this goal
as pure painful plus useless for developers in heat, but I trust TCs and QA
team when they think this is so important (on how this will help users/ops
when testing their environment) that they decided to make this as a goal
for our community.

I'm fine with whatever the solution we came out from here, and I'm happy to
implement it together, but please keep in mind that we need all to help us
track the progress, so we can have more opinions or suggestions for any
issues we are/will face on these tasks.

Let's do this together and make it right at once!

Would really like to hear more suggestions on how we can deal with this or
how other teams do, so please share with us if you(heat/QA member or not)
see this mail.


[1] https://etherpad.openstack.org/p/heat-queens-ptg-tempest-plugin
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20171130/66374839/attachment.html>