Open Stack

On 11/14/2014 10:10 PM, Martin Geisler wrote:
>> Of course, I need to run tests. That's a big part of the QA work, and I
>> > will certainly not give-up on that. You will have a hard time convincing
>> > anyone within the OpenStack community that it's OK to not run unit tests.
> That's not what I said: the OpenStack developers will continue to tests
> the software. I personally don't think it's the job of the downstream
> packagers to do this QA work. (It's of course cool to run the tests on
> the system installed by your packages -- that test run would then
> install the needed tools using npm and bower and whatnot if that's how
> the upstream has setup the test framework.)

What happens is that the environment within the distribution,
inevitably, will be different from the one ran on the gate. There's
going to be different versions of many components and so on. So it is
very important for me to also run these unit tests, to make sure that
everything continues to work.

Yes, the build-dependencies will pull the same components as pulled by
npm/bower, though they may be installed in different path, and maybe
using different versions.

On 11/14/2014 10:21 PM, Jeremy Stanley wrote:> On 2014-11-14 15:10:59
+0100 (+0100), Martin Geisler wrote:
> [...]
> Just to quibble on this particular point... distro packagers are
> also developers. They often (more often than we'd like, and we do
> try to find ways to help avoid this where possible) need to carry
> their own patches to tweak the software to fit their deployment and
> operation model. Being able to rerun tests in-place with packaged
> versions of everything including their patches helps them confirm
> that what they distribute still works as intended. Further, the
> distro users are well within their rights to modify and respin these
> packages themselves, and will potentially want to be able to run
> these tests for the same reasons.
>
> We distribute our tests as part of our software because our tests
> *are* part of our software.

Exactly! Let me give a few examples...

In Jessie, Nova carries patches so that there is support for Ceph. Until
a few days, there was still an issue with live migration over NFS. This
has just been fixed (thanks to Mehdi!), and running unit tests at build
time confirmed that.

Another example. Jessie will be released with Icehouse Horizon, but with
Django 1.7. The gate didn't test that, but I did. Most patches landed in
Juno, though never Icehouse will be tested with Django 1.7 in the gate.
Lucky, my package runs these unit tests and I can confirm that it
continues to work with Django 1.7 in Jessie.

Hoping these are giving you an insight as why it's *really* important to
run unit tests at build time for distributions,

Cheers,

Thomas Goirand (zigo)

P.S: I also run tempest tests over a Debian package installation to make
sure OpenStack is also functional. But that's another story... :)