Open Stack

On Thu, Sep 24, 2015, at 12:39 PM, Alex Schultz wrote:
> On Thu, Sep 24, 2015 at 1:58 PM, Emilien Macchi <emilien at redhat.com>
> wrote:
> >
> >
> > On 09/24/2015 02:19 PM, Alex Schultz wrote:
> >> On Thu, Sep 24, 2015 at 11:54 AM, Emilien Macchi <emilien at redhat.com> wrote:
> >>>
> >>>
> >>> On 09/24/2015 10:14 AM, Alex Schultz wrote:
> >>>> On Wed, Sep 23, 2015 at 4:56 PM, Emilien Macchi <emilien at redhat.com> wrote:
> >>>>> Background
> >>>>> ==========
> >>>>>
> >>>>> Current rspec tests are tested with modules mentioned in .fixtures.yaml
> >>>>> file of each module.
> >>>>>
> >>>>> * the file is not consistent across all modules
> >>>>> * it hardcodes module names & versions
> >>>>> * this way does not allow to use "Depend-On" feature, that would allow
> >>>>> to test cross-modules patches
> >>>>>
> >>>>> Proposal
> >>>>> ========
> >>>>>
> >>>>> * Like we do in beaker & integration jobs, use zuul-cloner to clone
> >>>>> modules in our CI jobs.
> >>>>> * Use r10k to prepare fixtures modules.
> >>>>> * Use Puppetfile hosted by openstack/puppet-openstack-integration
> >>>>>
> >>>>> In that way:
> >>>>> * we will have modules name + versions testing consistency across all
> >>>>> modules
> >>>>> * the same Puppetfile would be used by unit/beaker/integration testing.
> >>>>> * the patch that pass tests on your laptop would pass tests in upstream CI
> >>>>> * if you don't have zuul-cloner on your laptop, don't worry it will use
> >>>>> git clone. Though you won't have Depends-On feature working on your
> >>>>> laptop (technically not possible).
> >>>>> * Though your patch will support Depends-On in OpenStack Infra for unit
> >>>>> tests. If you submit a patch in puppet-openstacklib that drop something
> >>>>> wrong, you can send a patch in puppet-nova that will test it, and unit
> >>>>> tests will fail.
> >>>>>
> >>>>> Drawbacks
> >>>>> =========
> >>>>> * cloning from .fixtures.yaml takes ~ 10 seconds
> >>>>> * using r10k + zuul-clone takes ~50 seconds (more modules to clone).
> >>>>>
> >>>>> I think 40 seconds is something accept regarding the benefit.
> >>>>>
> >>>>
> >>>> As someone who consumes these modules downstream and has our own CI
> >>>> setup to run the rspec items, this ties it too closely to the
> >>>> openstack infrastructure. If we replace the .fixtures.yml with
> >>>> zuul-cloner, it assumes I always want the openstack version of the
> >>>> modules. This is not necessarily true. I like being able to replace
> >>>> items within fixtures.yml when doing dev work. For example If i want
> >>>> to test upgrading another module not related to openstack, like
> >>>> inifile, how does that work with the proposed solution?  This is also
> >>>> moving away from general puppet module conventions for testing. My
> >>>> preference would be that this be a different task and we have both
> >>>> .fixtures.yml (for general use/development) and the zuul method of
> >>>> cloning (for CI).  You have to also think about this from a consumer
> >>>> standpoint and this is adding an external dependency on the OpenStack
> >>>> infrastructure for anyone trying to run rspec or trying to consume the
> >>>> published versions from the forge.  Would I be able to run these tests
> >>>> in an offline mode with this change? With the .fixures.yml it's a
> >>>> minor edit to switch to local versions. Is the same true for the
> >>>> zuul-cloner version?
> >>>
> >>> What you did before:
> >>> * Edit .fixtures.yaml and put the version you like.
> >>>
> >>> What you would do this the current proposal:
> >>> * Edit openstack/puppet-openstack-integration/Puppetfile and put the
> >>> version you like.
> >>>
> >>
> >> So I have to edit a file in another module to test changes in
> >> puppet-neutron, puppet-nova, etc? With the zuul-cloner version, for
> >> local testing what does that workflow look like?
> >
> > If you need to test your code with cross-project dependencies, having
> > current .fixtures.yaml or the proposal won't change anything regarding
> > that, you'll still have to trick the YAML file that define the modules
> > name/versions.
> >
> >>
> >>> What you're suggesting has a huge downside:
> >>> People will still use fixtures by default and not test what is actually
> >>> tested by our CI.
> >>> A few people will know about the specific Rake task so a few people will
> >>> test exactly what upstream does. That will cause frustration to the most
> >>> of people who will see tests failing in our CI and not on their laptop.
> >>> I'm not sure we want that.
> >>
> >> You're right that the specific rake task may not be ideal. But that
> >> was one option, another option could be use fixtures first then
> >> replace with zuul-cloner provided versions but provide me the ability
> >> to turn of the zuul cloner part? I'm just saying as it is today, this
> >> change adds more complexity and hard ties into the OpenStack
> >> infrastructure with non-trival work arounds. I would love to solve the
> >> Depends-On issue, but I don't think that should include a deviation
> >> from generally accepted testing practices of puppet modules.
> >
> > I agree it's not best practice in Puppet but I don't see that as an huge
> > blocker. Our Puppet modules are Approved by Puppetlabs and respect most
> > of best practices AFIK. Is that fixctures thing a big deal?
> > I would like to hear from *cough*Hunner/Cody*cough* Puppetlabs about that.
> > Another proposal is welcome though, please go ahead.
> >
> 
> IMHO, it's more of a new developer thing. As a person who works on
> other puppet modules, to have a completely different dependency method
> for the OpenStack modules raises the barrier to entry for development.
> It could be addressed with better documentation...
> 
> >>>
> >>> I think more than most of people that run tests on their laptops want to
> >>> see them passing in upstream CI.
> >>> The few people that want to trick versions & modules, will have to run
> >>> Rake, trick the Puppetfile and run Rake again. It's not a big deal and
> >>> I'm sure this few people can deal with that.
> >>>
> >>
> >> So for me the zuul-cloner task seems more of a CI specific job that
> >> solves the Depends-On issues we currently have. Much like the beaker
> >> and acceptance tests that's not something I run locally.
> >
> > Hum. We implemented beaker tests in our modules so you can test the
> > module on your infra (laptop/cloud/whatever).
> > Here, we're just talking about unit testing, but it's still testing
> > after all.
> >
> > Beaker code is already using this proposal to clone the module.
> > The proposal is re-using the same code to be consistent, and keep one
> > single centralized Puppetfile.
> >
> >> I usually run the local rspec tests first before shipping off to CI to see how that
> >> plays out but I would manage the .fixtures.yml if necessary to test
> >> cross module dependancies.  I don't expect to replicate an entire CI
> >> environment setup on my laptop for testing.
> >
> > This proposal do not "replicate an entire CI". It just clone all
> > modules, on the same version, hence the 40 seconds difference.
> > The rspec tests will still work for you, and you won't see any difference.
> >
> >> The rspec tests for me, represent a quick way to test fixes before shipping off to CI for more
> >> testing.
> >
> > This is wrong. What you test on your laptop is not what we test in
> > upstream CI: not the same modules, not the same dependencies.
> > OpenStack Puppet modules are working with (upstream) dependencies that
> > help to build OpenStack Clouds.
> >
> > OpenStack is already running the same structure with Global Requirements
> > [1] (Python dependencies), where each project works with it.
> >
> > [1] http://git.openstack.org/cgit/openstack/requirements
> >
> > With this proposal, our modules will follow the same concept, where they
> > would be tested (unit + functional) against the same dependencies.
> >
> 
> Sorry, by this I meant for additional version testing. Rather than
> running puppet 3.{4,5,6,7,8} and 4.x locally. Yes we need to test with
> the same version of the modules as would be in CI but that would be
> where the .fixtures.yml comes in.
> 
> >> Going back to the background item from the original email, the
> >> .fixtures.yml shouldn't be identical for all modules. It should only
> >> be the modules required to test the specific module. I doubt all of
> >> the puppet OpenStack modules require each other, right? So that's not
> >> a problem, that's an expectation. Additionally, we should be managing
> >> these anyway so when we publish the modules to the forge, it has
> >> proper metadata indicating the dependancies.
> >
> > All modules have quite often openstacklib/mysql/rabbitmq/qpid/keystone
> > at least.
> >
> >> This change seems targeted towards solving OpenStack CI environmental
> >> setup issues, and not really improving individual module development
> >> from a regular puppet standpoint.
> >
> > Nope, if you read again the proposal, it's solving:
> > "I would like to run tests on my laptop that will pass OpenStack CI with
> > the right modules and the right versions".
> >
> > If you want to run your specific modules & version, I guess you'll have
> > to do like you're doing with fixtures: running rake (stop it after
> > cloning Puppetfile), editing Puppetfile, running rake again, instead of
> > just editing fixtures.
> >
> 
> That is not very user-friendly and that's what I was referring to with
> the non-trival work arounds.  This would make this a very OpenStack
> puppet specific development workflow.
> 
> I also have concerns around the downstream CI implications for this.
> Being able to run tests without internet connectivity is important to
> some people so I want to make sure that can continue without having to
> break the process mid-cycle to try and inject a workaround. It would
> better if we could have a workaround upfront. For example make a
> Puppetfile location an environment variable and if not defined pull
> down the puppet-openstack-integration one?  I wish there was a better
> dependency resolution method that just pulling everything down from
> the internets.  I just know that doesn't work everywhere.
> 
I will admit to not properly following this entire thread, but I want to
jump in and point out that zuul-cloner clones from arbitrary locations.
You can point it to your local disk (in fact we do this for cached
repos), or any other git server that can be fetched from without
interactive authentication needs. This means you can do downstream
testing just fine without an internet connection, just point zuul-cloner
at wherever your git repos are.

The idea behind zuul-cloner is that it will do all of the correct things
with zuul refs and fallback to the appropriate branches when the refs
are not set. This means that it should just work by falling back to your
dev branches if you tell it to. If it doesn't work outside of a zuul
context that would be a bug and we should fix it.

Clark