[openstack-dev] [3rd party testing] How to setup CI? Take #2

Luke Gorrie luke at tail-f.com
Thu Mar 13 14:30:44 UTC 2014


Howdy!

I have some tech questions I'd love some pointers on from people who've
succeeded in setting up CI for Neutron based on the upstream devstack-gate.

Here are the parts where I'm blocked now:

1. I need to enable an ML2 mech driver. How can I do this? I have been
trying to create a localrc with a "Q_ML2_PLUGIN_MECHANISM_DRIVERS=..."
line, but it appears that the KEEP_LOCALRC option in devstack-gate is
broken (confirmed on #openstack-infra).

2. How do I streamline which tests are run? I tried added "export
DEVSTACK_GATE_TEMPEST_REGEX=network" in the Jenkins job configuration but I
don't see any effect. (word on #openstack-infra is this option is not used
by them so status unknown.)

3. How do I have Jenkins copy the log files into a directory on the Jenkins
master node (that I can serve up with Apache)? This is left as an exercise
to the reader in the blog tutorial but I would love a cheat, since I am
getting plenty of exercise already :-).

I also have the meta-question: How can I test changes/fixes to
devstack-gate? I've attempted many times to modify how scripts work, but I
don't have a global understanding of the whole openstack-infra setup, and
somehow my changes always end up being clobbered by a fresh checkout from
the upstream repo on Github. That's crazy frustrating when it takes 10+
minutes to fire up a test via Jenkins even when I'm only e.g. trying to add
an "echo" to a shell script somewhere to see what's in an environment
variable at a certain point in a script. I'd love a faster edit-compile-run
loop, especially one that doesn't involve needing to get changed merged
upstream into the official openstack-infra repo.

I also have an issue that worries me. I once started seeing tempest tests
failing due to a resource leak where the kernel ran out of loopback mounts
and that broke tempest. Here is what I saw:

    root at egg-slave:~# losetup -a
    /dev/loop0: [fc00]:5248399
(/opt/stack/data/swift/drives/images/swift.img)
    /dev/loop1: [fc00]:5248409 (/opt/stack/data/stack-volumes-backing-file)
    /dev/loop2: [fc00]:5248467
(/opt/stack/data/swift/drives/images/swift.img)
    /dev/loop3: [fc00]:5248496
(/opt/stack/data/swift/drives/images/swift.img)
    /dev/loop4: [fc00]:5248702
(/opt/stack/data/swift/drives/images/swift.img)
    /dev/loop5: [fc00]:5248735
(/opt/stack/data/swift/drives/images/swift.img)
    /dev/loop6: [fc00]:5248814
(/opt/stack/data/swift/drives/images/swift.img)
    /dev/loop7: [fc00]:5248825
(/opt/stack/data/swift/drives/images/swift.img)

and trying to remove this with 'losetup -d ...' had no effect. I rebooted.
(I'm on Ubuntu 13.10.)

This kind of spurious error has the potential to cause my CI to start
casting negative votes (again) and upsetting everybody's workflows, not
because my tests have actually found a problem but just because it's a
non-trivial problem for me to keep a devstack-gate continuously
operational. I hope that doesn't happen, but with this level of
infrastructure complexity it does feel a little like playing russian
roulette that the next glitch in
devstack/devstack-gate/Jenkins/Gerrit/Zuul/Gearman/... will manifest itself
in the copy that's running on my server. </vent>

Cheers,
-Luke
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20140313/3d527558/attachment.html>


More information about the OpenStack-dev mailing list