Open Stack

On 09/24/2015 10:14 AM, Alex Schultz wrote:
> On Wed, Sep 23, 2015 at 4:56 PM, Emilien Macchi <emilien at redhat.com> wrote:
>> Background
>> ==========
>>
>> Current rspec tests are tested with modules mentioned in .fixtures.yaml
>> file of each module.
>>
>> * the file is not consistent across all modules
>> * it hardcodes module names & versions
>> * this way does not allow to use "Depend-On" feature, that would allow
>> to test cross-modules patches
>>
>> Proposal
>> ========
>>
>> * Like we do in beaker & integration jobs, use zuul-cloner to clone
>> modules in our CI jobs.
>> * Use r10k to prepare fixtures modules.
>> * Use Puppetfile hosted by openstack/puppet-openstack-integration
>>
>> In that way:
>> * we will have modules name + versions testing consistency across all
>> modules
>> * the same Puppetfile would be used by unit/beaker/integration testing.
>> * the patch that pass tests on your laptop would pass tests in upstream CI
>> * if you don't have zuul-cloner on your laptop, don't worry it will use
>> git clone. Though you won't have Depends-On feature working on your
>> laptop (technically not possible).
>> * Though your patch will support Depends-On in OpenStack Infra for unit
>> tests. If you submit a patch in puppet-openstacklib that drop something
>> wrong, you can send a patch in puppet-nova that will test it, and unit
>> tests will fail.
>>
>> Drawbacks
>> =========
>> * cloning from .fixtures.yaml takes ~ 10 seconds
>> * using r10k + zuul-clone takes ~50 seconds (more modules to clone).
>>
>> I think 40 seconds is something accept regarding the benefit.
>>
> 
> As someone who consumes these modules downstream and has our own CI
> setup to run the rspec items, this ties it too closely to the
> openstack infrastructure. If we replace the .fixtures.yml with
> zuul-cloner, it assumes I always want the openstack version of the
> modules. This is not necessarily true. I like being able to replace
> items within fixtures.yml when doing dev work. For example If i want
> to test upgrading another module not related to openstack, like
> inifile, how does that work with the proposed solution?  This is also
> moving away from general puppet module conventions for testing. My
> preference would be that this be a different task and we have both
> .fixtures.yml (for general use/development) and the zuul method of
> cloning (for CI).  You have to also think about this from a consumer
> standpoint and this is adding an external dependency on the OpenStack
> infrastructure for anyone trying to run rspec or trying to consume the
> published versions from the forge.  Would I be able to run these tests
> in an offline mode with this change? With the .fixures.yml it's a
> minor edit to switch to local versions. Is the same true for the
> zuul-cloner version?

What you did before:
* Edit .fixtures.yaml and put the version you like.

What you would do this the current proposal:
* Edit openstack/puppet-openstack-integration/Puppetfile and put the
version you like.

What you're suggesting has a huge downside:
People will still use fixtures by default and not test what is actually
tested by our CI.
A few people will know about the specific Rake task so a few people will
test exactly what upstream does. That will cause frustration to the most
of people who will see tests failing in our CI and not on their laptop.
I'm not sure we want that.

I think more than most of people that run tests on their laptops want to
see them passing in upstream CI.
The few people that want to trick versions & modules, will have to run
Rake, trick the Puppetfile and run Rake again. It's not a big deal and
I'm sure this few people can deal with that.

>>
>> Next steps
>> ==========
>>
>> * PoC in puppet-nova: https://review.openstack.org/#/c/226830/
>> * Patch openstack/puppet-modulesync-config to be consistent across all
>> our modules.
>>
>> Bonus
>> =====
>> we might need (asap) a canary job for puppet-openstack-integration
>> repository, that would run tests on a puppet-* module (since we're using
>> install_modules.sh & Puppetfile files in puppet-* modules).
>> Nothing has been done yet for this work.
>>
>>
>> Thoughts?
>> --
>> Emilien Macchi
>>
>>
> 
> I think we need this functionality, I just don't think it's a
> replacement for the .fixures.yml.
> 
> Thanks,
> -Alex
> 
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 

-- 
Emilien Macchi

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 473 bytes
Desc: OpenPGP digital signature
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20150924/fcc6b941/attachment.pgp>