Open Stack

Hi,
raising again the question about tempest running on TripleO CI as it was
discussed in the last TripleO meeting.

I'd like to get your attention that in these tests, which I ran just for
ensure it works, there were bugs discovered, and these weren't corner cases
but real failures of TripleO installation. Like this one for Sahara:
https://review.openstack.org/#/c/309042/
I'm sorry, I should have prepared these bugs for the meeting as proofs for
testing value.

The second issue that was blocker before is a wall time and now, as we can
see from jobs length, after HW upgrade of CI - is not an issue anymore. We
can run tempest without any fear to get into timeout problem, "nonha" job
for sure, as most short from all.

So I'd insist on running tempest exactly on promoting job in order not to
promote images with bugs, especially the critical like the whole service
not available at all. The pingtest is not enough for this purpose as we can
see from the bugs above, it checks very basic things and not all services
are covered. I think we aren't interested just to see the jobs green, but
sticking for the basic working functionality and quality of promoting.
Maybe it's influence of my previous QA roles, but I don't see any value to
promote something with bugs.

The point about CI stability - the last issues that CI faces now are not so
connected to tempest tests or CI code at all, it's bugs of underlying
projects and whether tempest will run or not doesn't really matters in this
case. These issues fail everything yet before any testing starts.
Indication of such issues before they leak into TripleO is different topic
and approach.

So my main point for running tempest tests on "nonha" periodic jobs is:
Quality and guaranteed basic functionality of installed overcloud services.
At least all of them are up and can accept connections. Avoid and early
discover critical bugs that are not seen in pingtest. I remind that we
going to run the only smoke tests, which takes not much time and check the
basic functionality only.

P.S. If there is interest, we can run the whole tempest set or specific
sets in experimental or third-party jobs just for indication. And I mean
not only tempest tests, but project scenario tests as well, for example
Heat integration tests. Both for undercloud and overcloud.

P.P.S Just ping me if you have any unclear points or would like to discuss
it in separate meeting, I'll give the all required info.

Thanks
-- 
Best regards
Sagi Shnaidman
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20160517/a2f2d4e2/attachment.html>