Open Stack

Excerpts from Victor Stinner's message of 2015-06-30 14:13:21 +0200:
> Hi,
> 
> Unfortunately, it looks like each regressions introduced by releases of 
> Oslo libraries are still common :-( We already have tools to detect 
> regressions, but they are run manually. It would be nice to automate 
> these tests and run them more frequently (at least one per week, or even 
> daily?).
> 
> There are tools to run unit tests of projects like neutronclient or nova 
> on the development version of oslo.* libraries. They take up to 12 hours 
> to run all tests. Example of tools:

12 hours was a guess -- I usually run the tests overnight because some
of them do tend to run several hours.

> 
> - "tox -e venv -- oslo_run_pre_release_tests" in each Oslo project
> - tools/oslo_run_cross_tests in oslotest
> 
> To prepare the latest bunch of releases, dims wrote two patches to run 
> nova unit tests and tempest:

tempest does run for every patch submitted, so we have the integration
test space covered. We don't have the unit test jobs and pep8 jobs
automated right now because of the test matrix. Today 73 projects in the
openstack/* namespace use oslo.config. Testing all of those for unit
test and pep8 regressions (yes, those happen) for one patch to
oslo.config would take 146 test nodes.

It's possible to run all of the unit tests serially on the same
server, using the tools you mention, but that takes longer than our
current timeout. We might be able to set up an experimental or periodic
job to run those tests, but given the length of time they take we
wouldn't want to run them automatically for each patch.

> 
> * https://review.openstack.org/#/c/186413/
> * https://review.openstack.org/#/c/186418/
> 
> Unfortunately, the stable version of some oslo.* projects were used 
> instead of the development version, and 2 regressions were missed :-/

Was that related to the --force-reinstall option in the tox.ini file?
Why is that there?

> It would nice to automate a job running cinder, nova and neutron unit 
> tests and tempest on the development versions (git master branch) of all 
> oslo.* projects. We can start with something simple: run 

I'm not sure what scenario that tests, though. We switched to
integration tests of released libraries because we don't want
applications to rely on unreleased features. We already run tempest
for patches to the libraries as part of the individual gate queues
for each library. Why do we need to run tempest with the master
branch of all of the libraries at once?

> tools/oslo_run_cross_tests and send the result by email every day to 
> some people interested by the result (ex: Oslo liaisons, or just me if 
> nobody cares). It would be nice to have a dedicated resource in the 
> OpenStack infra for such job.

We have a separate list for the stable-maint job reports. We could
create one for these reports, too.

> 
> I proposed to run "all" tests using Gerrit for each patch send to an 
> Oslo project, but it would use too much resources of the OpenStack infra.
> 
> Another idea would be to write a check job dedicated to Oslo releases 
> and use Gerrit to prepare a release (at least to tag a version in Git).

I'm working with Thierry and the infra team to design some tools for
reviewing release requests: https://review.openstack.org/191193

Doug