Open Stack

On Wed, Sep 23, 2015 at 4:56 PM, Emilien Macchi <emilien at redhat.com> wrote:
> Background
> ==========
>
> Current rspec tests are tested with modules mentioned in .fixtures.yaml
> file of each module.
>
> * the file is not consistent across all modules
> * it hardcodes module names & versions
> * this way does not allow to use "Depend-On" feature, that would allow
> to test cross-modules patches
>
> Proposal
> ========
>
> * Like we do in beaker & integration jobs, use zuul-cloner to clone
> modules in our CI jobs.
> * Use r10k to prepare fixtures modules.
> * Use Puppetfile hosted by openstack/puppet-openstack-integration
>
> In that way:
> * we will have modules name + versions testing consistency across all
> modules
> * the same Puppetfile would be used by unit/beaker/integration testing.
> * the patch that pass tests on your laptop would pass tests in upstream CI
> * if you don't have zuul-cloner on your laptop, don't worry it will use
> git clone. Though you won't have Depends-On feature working on your
> laptop (technically not possible).
> * Though your patch will support Depends-On in OpenStack Infra for unit
> tests. If you submit a patch in puppet-openstacklib that drop something
> wrong, you can send a patch in puppet-nova that will test it, and unit
> tests will fail.
>
> Drawbacks
> =========
> * cloning from .fixtures.yaml takes ~ 10 seconds
> * using r10k + zuul-clone takes ~50 seconds (more modules to clone).
>
> I think 40 seconds is something accept regarding the benefit.
>

As someone who consumes these modules downstream and has our own CI
setup to run the rspec items, this ties it too closely to the
openstack infrastructure. If we replace the .fixtures.yml with
zuul-cloner, it assumes I always want the openstack version of the
modules. This is not necessarily true. I like being able to replace
items within fixtures.yml when doing dev work. For example If i want
to test upgrading another module not related to openstack, like
inifile, how does that work with the proposed solution?  This is also
moving away from general puppet module conventions for testing. My
preference would be that this be a different task and we have both
.fixtures.yml (for general use/development) and the zuul method of
cloning (for CI).  You have to also think about this from a consumer
standpoint and this is adding an external dependency on the OpenStack
infrastructure for anyone trying to run rspec or trying to consume the
published versions from the forge.  Would I be able to run these tests
in an offline mode with this change? With the .fixures.yml it's a
minor edit to switch to local versions. Is the same true for the
zuul-cloner version?

>
> Next steps
> ==========
>
> * PoC in puppet-nova: https://review.openstack.org/#/c/226830/
> * Patch openstack/puppet-modulesync-config to be consistent across all
> our modules.
>
> Bonus
> =====
> we might need (asap) a canary job for puppet-openstack-integration
> repository, that would run tests on a puppet-* module (since we're using
> install_modules.sh & Puppetfile files in puppet-* modules).
> Nothing has been done yet for this work.
>
>
> Thoughts?
> --
> Emilien Macchi
>
>

I think we need this functionality, I just don't think it's a
replacement for the .fixures.yml.

Thanks,
-Alex