Open Stack

On Thu, Sep 24, 2015 at 1:58 PM, Emilien Macchi <emilien at redhat.com> wrote:
>
>
> On 09/24/2015 02:19 PM, Alex Schultz wrote:
>> On Thu, Sep 24, 2015 at 11:54 AM, Emilien Macchi <emilien at redhat.com> wrote:
>>>
>>>
>>> On 09/24/2015 10:14 AM, Alex Schultz wrote:
>>>> On Wed, Sep 23, 2015 at 4:56 PM, Emilien Macchi <emilien at redhat.com> wrote:
>>>>> Background
>>>>> ==========
>>>>>
>>>>> Current rspec tests are tested with modules mentioned in .fixtures.yaml
>>>>> file of each module.
>>>>>
>>>>> * the file is not consistent across all modules
>>>>> * it hardcodes module names & versions
>>>>> * this way does not allow to use "Depend-On" feature, that would allow
>>>>> to test cross-modules patches
>>>>>
>>>>> Proposal
>>>>> ========
>>>>>
>>>>> * Like we do in beaker & integration jobs, use zuul-cloner to clone
>>>>> modules in our CI jobs.
>>>>> * Use r10k to prepare fixtures modules.
>>>>> * Use Puppetfile hosted by openstack/puppet-openstack-integration
>>>>>
>>>>> In that way:
>>>>> * we will have modules name + versions testing consistency across all
>>>>> modules
>>>>> * the same Puppetfile would be used by unit/beaker/integration testing.
>>>>> * the patch that pass tests on your laptop would pass tests in upstream CI
>>>>> * if you don't have zuul-cloner on your laptop, don't worry it will use
>>>>> git clone. Though you won't have Depends-On feature working on your
>>>>> laptop (technically not possible).
>>>>> * Though your patch will support Depends-On in OpenStack Infra for unit
>>>>> tests. If you submit a patch in puppet-openstacklib that drop something
>>>>> wrong, you can send a patch in puppet-nova that will test it, and unit
>>>>> tests will fail.
>>>>>
>>>>> Drawbacks
>>>>> =========
>>>>> * cloning from .fixtures.yaml takes ~ 10 seconds
>>>>> * using r10k + zuul-clone takes ~50 seconds (more modules to clone).
>>>>>
>>>>> I think 40 seconds is something accept regarding the benefit.
>>>>>
>>>>
>>>> As someone who consumes these modules downstream and has our own CI
>>>> setup to run the rspec items, this ties it too closely to the
>>>> openstack infrastructure. If we replace the .fixtures.yml with
>>>> zuul-cloner, it assumes I always want the openstack version of the
>>>> modules. This is not necessarily true. I like being able to replace
>>>> items within fixtures.yml when doing dev work. For example If i want
>>>> to test upgrading another module not related to openstack, like
>>>> inifile, how does that work with the proposed solution?  This is also
>>>> moving away from general puppet module conventions for testing. My
>>>> preference would be that this be a different task and we have both
>>>> .fixtures.yml (for general use/development) and the zuul method of
>>>> cloning (for CI).  You have to also think about this from a consumer
>>>> standpoint and this is adding an external dependency on the OpenStack
>>>> infrastructure for anyone trying to run rspec or trying to consume the
>>>> published versions from the forge.  Would I be able to run these tests
>>>> in an offline mode with this change? With the .fixures.yml it's a
>>>> minor edit to switch to local versions. Is the same true for the
>>>> zuul-cloner version?
>>>
>>> What you did before:
>>> * Edit .fixtures.yaml and put the version you like.
>>>
>>> What you would do this the current proposal:
>>> * Edit openstack/puppet-openstack-integration/Puppetfile and put the
>>> version you like.
>>>
>>
>> So I have to edit a file in another module to test changes in
>> puppet-neutron, puppet-nova, etc? With the zuul-cloner version, for
>> local testing what does that workflow look like?
>
> If you need to test your code with cross-project dependencies, having
> current .fixtures.yaml or the proposal won't change anything regarding
> that, you'll still have to trick the YAML file that define the modules
> name/versions.
>
>>
>>> What you're suggesting has a huge downside:
>>> People will still use fixtures by default and not test what is actually
>>> tested by our CI.
>>> A few people will know about the specific Rake task so a few people will
>>> test exactly what upstream does. That will cause frustration to the most
>>> of people who will see tests failing in our CI and not on their laptop.
>>> I'm not sure we want that.
>>
>> You're right that the specific rake task may not be ideal. But that
>> was one option, another option could be use fixtures first then
>> replace with zuul-cloner provided versions but provide me the ability
>> to turn of the zuul cloner part? I'm just saying as it is today, this
>> change adds more complexity and hard ties into the OpenStack
>> infrastructure with non-trival work arounds. I would love to solve the
>> Depends-On issue, but I don't think that should include a deviation
>> from generally accepted testing practices of puppet modules.
>
> I agree it's not best practice in Puppet but I don't see that as an huge
> blocker. Our Puppet modules are Approved by Puppetlabs and respect most
> of best practices AFIK. Is that fixctures thing a big deal?
> I would like to hear from *cough*Hunner/Cody*cough* Puppetlabs about that.
> Another proposal is welcome though, please go ahead.
>

IMHO, it's more of a new developer thing. As a person who works on
other puppet modules, to have a completely different dependency method
for the OpenStack modules raises the barrier to entry for development.
It could be addressed with better documentation...

>>>
>>> I think more than most of people that run tests on their laptops want to
>>> see them passing in upstream CI.
>>> The few people that want to trick versions & modules, will have to run
>>> Rake, trick the Puppetfile and run Rake again. It's not a big deal and
>>> I'm sure this few people can deal with that.
>>>
>>
>> So for me the zuul-cloner task seems more of a CI specific job that
>> solves the Depends-On issues we currently have. Much like the beaker
>> and acceptance tests that's not something I run locally.
>
> Hum. We implemented beaker tests in our modules so you can test the
> module on your infra (laptop/cloud/whatever).
> Here, we're just talking about unit testing, but it's still testing
> after all.
>
> Beaker code is already using this proposal to clone the module.
> The proposal is re-using the same code to be consistent, and keep one
> single centralized Puppetfile.
>
>> I usually run the local rspec tests first before shipping off to CI to see how that
>> plays out but I would manage the .fixtures.yml if necessary to test
>> cross module dependancies.  I don't expect to replicate an entire CI
>> environment setup on my laptop for testing.
>
> This proposal do not "replicate an entire CI". It just clone all
> modules, on the same version, hence the 40 seconds difference.
> The rspec tests will still work for you, and you won't see any difference.
>
>> The rspec tests for me, represent a quick way to test fixes before shipping off to CI for more
>> testing.
>
> This is wrong. What you test on your laptop is not what we test in
> upstream CI: not the same modules, not the same dependencies.
> OpenStack Puppet modules are working with (upstream) dependencies that
> help to build OpenStack Clouds.
>
> OpenStack is already running the same structure with Global Requirements
> [1] (Python dependencies), where each project works with it.
>
> [1] http://git.openstack.org/cgit/openstack/requirements
>
> With this proposal, our modules will follow the same concept, where they
> would be tested (unit + functional) against the same dependencies.
>

Sorry, by this I meant for additional version testing. Rather than
running puppet 3.{4,5,6,7,8} and 4.x locally. Yes we need to test with
the same version of the modules as would be in CI but that would be
where the .fixtures.yml comes in.

>> Going back to the background item from the original email, the
>> .fixtures.yml shouldn't be identical for all modules. It should only
>> be the modules required to test the specific module. I doubt all of
>> the puppet OpenStack modules require each other, right? So that's not
>> a problem, that's an expectation. Additionally, we should be managing
>> these anyway so when we publish the modules to the forge, it has
>> proper metadata indicating the dependancies.
>
> All modules have quite often openstacklib/mysql/rabbitmq/qpid/keystone
> at least.
>
>> This change seems targeted towards solving OpenStack CI environmental
>> setup issues, and not really improving individual module development
>> from a regular puppet standpoint.
>
> Nope, if you read again the proposal, it's solving:
> "I would like to run tests on my laptop that will pass OpenStack CI with
> the right modules and the right versions".
>
> If you want to run your specific modules & version, I guess you'll have
> to do like you're doing with fixtures: running rake (stop it after
> cloning Puppetfile), editing Puppetfile, running rake again, instead of
> just editing fixtures.
>

That is not very user-friendly and that's what I was referring to with
the non-trival work arounds.  This would make this a very OpenStack
puppet specific development workflow.

I also have concerns around the downstream CI implications for this.
Being able to run tests without internet connectivity is important to
some people so I want to make sure that can continue without having to
break the process mid-cycle to try and inject a workaround. It would
better if we could have a workaround upfront. For example make a
Puppetfile location an environment variable and if not defined pull
down the puppet-openstack-integration one?  I wish there was a better
dependency resolution method that just pulling everything down from
the internets.  I just know that doesn't work everywhere.

>>>>>
>>>>> Next steps
>>>>> ==========
>>>>>
>>>>> * PoC in puppet-nova: https://review.openstack.org/#/c/226830/
>>>>> * Patch openstack/puppet-modulesync-config to be consistent across all
>>>>> our modules.
>>>>>
>>>>> Bonus
>>>>> =====
>>>>> we might need (asap) a canary job for puppet-openstack-integration
>>>>> repository, that would run tests on a puppet-* module (since we're using
>>>>> install_modules.sh & Puppetfile files in puppet-* modules).
>>>>> Nothing has been done yet for this work.
>>>>>
>>>>>
>>>>> Thoughts?
>>>>> --
>>>>> Emilien Macchi
>>>>>
>>>>>
>>>>
>>>> I think we need this functionality, I just don't think it's a
>>>> replacement for the .fixures.yml.
>>>>
>>>> Thanks,
>>>> -Alex
>>>>
>>>> __________________________________________________________________________
>>>> OpenStack Development Mailing List (not for usage questions)
>>>> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>>
>>>
>>> --
>>> Emilien Macchi
>>>
>>>
>>> __________________________________________________________________________
>>> OpenStack Development Mailing List (not for usage questions)
>>> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>
>>
>> __________________________________________________________________________
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>
> --
> Emilien Macchi
>
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>